Enhanced Expert Merging for Mixture-of-Experts in Graph Foundation Models
Authors: Lei Liu
Xingyu Xia
Qianqian Xie
Ben Liu
Wenjie Xu
Min Peng
Venue: NeurIPS 2025 poster ()
Keywords: Graph Representation Learning
Graph Foundation Model
Mixture of Experts
Expert Merging
Primary area: deep_learning
TL;DR: Efficient expert merging for MoE-based graph foundation models approaching ensemble-level performance.

Abstract: Graph foundation models (GFMs) have emerged as a promising paradigm for learning transferable knowledge across diverse graph-structured data. The inherent heterogeneity in features and graph structures poses significant challenges for building scalable and generalizable GFMs. Existing research has employed mixture-of-experts (MoE) models to handle the challenges, assigning the most suitable expert to each graph. Despite this, the underlying mechanisms of MoE within the context of GFMs remain insufficiently explored. In this work, we conduct an in-depth experimental study on an MoE-based GFM and uncover an intriguing finding: the experts ranked second and third assigned by the router perform better than the top-ranked expert. This insight motivates us to investigate the potential of leveraging knowledge embedded across multiple experts. However, directly ensembling the outputs of multiple experts would incur substantial computational overhead, while applying a standard expert merging strategy risks suboptimal performance. To address these challenges, we introduce two enhanced expert merging strategies that retain the computational efficiency of expert merging, while improving performance to approach the effectiveness of expert ensembling. Specifically, we propose (i) a knowledge distillation-inspired expert merging method that aligns the behavior of parameter-fused experts with expert ensembles, and (ii) a theoretical parameter proximity approach that leverages the similarity of expert parameters to approximate ensemble outputs while preserving diversity. Extensive experiments demonstrate that our methods effectively enhance model performance.

Enhanced Expert Merging for Mixture-of-Experts in Graph Foundation Models
Lei Liul.3, Xingyu Xia', Qianqian Xie2.3, Ben Liu', Wenjie Xu', Min Peng 1,233 Schooo of Computer Science, Wuhan University, Wuhan, China Sccooo of Artificial Intelligence, Wuhan University, Wuhan, China Center for Language and Information Research, Wuhan University, Wuhan, China (liulei95, summer, xieq, liuben123, vingerxu, pengm) @whu. edu. cn
Abstract
Graph foundation models (GFMs) have emerged as promising paradigm for learning transferable knowledge across diverse graph-structured data. The inher- ent heterogeneity in features and graph structures poses significant challenges for building scalable and generalizable GFMs. Existing research has employed mixture-of-exper (MoE) models to handle the challenges, assigning the most suit- able expert to each graph. Despite this, the underlying mechanisms MoE within the context of GFMs remain insufficiently explored. In this work, conduct in-depth experimental study on an MoE-based GFM and uncover an intriguing finding: the experts ranked second and third assigned by the router perform better than the top-ranked expert. This insight motivates us to investigate the potential of leveraging knowledge embedded across multiple experts. However, directly ensembling the outputs of multiple experts would incur substantial computational overhead, while applying a standard expert merging strategy risks suboptimal performance. To address these challenges, we introduce two enhanced expert merging strategies that retain the computational efffcite of expert merging, while improving performance to approach the effectiveness of expert ensembling, Specif- ically, we propose (i) a knowledge distillation-inspired expert merging method that aligns the behavior of parameter-fused experts with expert ensembles, and (ii) a theoretical parameter proximity approach that leverages the similarity of expert parameters to approximate ensemble outputs while preserving diversity. Extensive experiments demonstrate that our methods effectively enhance model performance.
Introduction
Graph-structured data are ubiquitous and appear in diverse domains such as social networks [3], molecular biology (32). recommendation systems [12, and knowledge graphs [9] As the complexity and volume such data continue to increase, the development of graph foundation models (GFMs) becomes essential meet the growing demand for generalizable, scalable, and efficient learning systems capable of handling diverse graph-structured data from various domains. The concept of GFMs mirrors that of foundation models in natural language processing (NLP) and computer vision (CV) I: they are pre- pre-trained on extensive graph data and can be adapted to a wide range of downstream graph tasks [24] However, from technical perspective, GFMs have yet catch up with their counterparts language and vision
Inspired by the success of foundation models in NLP and CV, the graph learning community is The goal is to learn transferable knowledge
increasingly focused on developing GFMs 9 49 37 19
"Corresponding author. Email: pengm @ whu.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

from broad graph data, enabling their application across various graph domains and downstream tasks. However, building effective GFMs presents two major challenges: (1) feature heterogeneity. which arises from the diverse types, semantics, and dimensionalities of node and edge features across graphs, and (2) structure heterogeneity, which reflects differences in graph topologies and connectivity patterns, such as variations in local neighborhood density or the presence of scale-free or small-wort mooldwwolld properties. Currently, there are two main approaches to address feature heterogeneity: (i) Using language models to encode textual attributes from graphs in multiple domains into unified textual semantic space [2 [5]; (ii) Employing singular value decomposition (SVD) to align the feature dimensions across different graphs 41 40]. To address structure heterogeneity, recent work has started incorporating mixture-of-experts (MoE) mechanisms into general graph learning models [40].
The effectiveness of MoE has been widely validated in LLMs 17 and multimodal models [22]. However, its working mechanisms in GFMs remain insufficiently explored. To further investigate the role of MoF in GFMs, we conduct exploratory experiments on AnyGraph [40], a recently proposed MoF-based GFM, and discover an interesting phenomenon (shown in [i: the performance of the second and third-ranked experts assigned by the router is stronger than that of the top-ranked expert. This insight motivates us to explore the potential of leveraging knowledge from multiple experts. To this end, we increase the number of selected experts and ensemble their outputs. Experiments reveal that this approach significantly improves the model's 's performance, which is consistent with findings from previous study [30]. The underlying principle is that consulting multiple experts enables the model to effectively balance and integrate diverse knowledge.
However, directly increasing the number of selected experts significantly increases training and inference costs, particularly for large expert models. A common strategy to mitigate these costs is expert merging, which combines the parameters of the top-k selected experts via weighted av- eraging [I 28). Yet, our findings indicate that this approach often degrades model performance. Such degradation likely stems from the high specialization of learned experts, leading to significant parameter disparities that cause interference during merging [4] To overcome these challenges, we propose to integrate the efficiency of expert merging with the performance benefits of expert ensembles. Specifically, we introduce two advanced expert merging strategies:
(I) Knowledge Distillation Enhanced Expert Merging (KDEM): Since the expert ensemble significantly outperforms the fused expert, we treat the ensemble of the top-k selected experts as the teacher and the parameter-merged expert as the student to align the behavior of the merged expert with that of the ensemble. During training, the collective knowledge of the expert ensemble is distilled into the fused expert, thereby enhancing its capabilities. During inference, the only additional overhead is the weighted averaging of the selected expert parameters, and only the fused expert processes the inputs. This avoids the computational overhead of evaluating multiple experts, significantly reducing inference time. However, ensembling the outputs of multiple experts during training still incurs considerable time costs. Motivated by the observation that real-world teachers do not instruct students continuously, we propose periodically applying knowledge distillation to reduce training time.
(2) Parameter Proximity Enhanced Expert Merging (PPEM): To further enhance training effi ciency, we undertake a theoretical investigation. Since multi-layer perceptrons (MLPs) constitute the majority of experts in MoE architectures, we demonstrate that when multiple MLPs have similar pa- rameters, their parameter-merge outputs. Based on this theoretical foundation, we propose a strategy to gradually reduce parameter distances between experts. Specifically, we guide the parameters of the top-k experts toward their weighted average, thereby improving mergeability while preserving expert diversity. This method operates directly in parameter space without introducing significant computational overhead.
MLP output closely approximates the ensemble of their individual
In summary, our key contributions are as follows:
To gain deeper insights into the MoE mechanism within graph learning, we examine an MoE-based GFM, revealing the key limitation of the routing strategy: the experts ranked second and third frequently surpass the top-ranked expert. This motivates us to explore expert ensembling, which we find can enhance performance, while a simple expert merging method leads to suboptimal results.
To effectively and efficiently harness the knowledge of multiple experts, we propose two enhanced expert merging methods: (i) KDEM, which applies knowledge distillation to align the behavior of merged experts with that of expert ensembles. (ii) PPEM, which leverages the theoretical finding to gradually brings the expert parameters closer during training. These methods align the performance of expert merging with that of expert ensembling while preserving computational efficiency.

---

We validate our approaches through extensive experiments on 38 graph datasets across various domains and downstream tasks, demonstrating their effectiveness and efficiency.
2 Preliminaries
Graph-Structured Data. graph can be denoted as (V,E), where V= {m:} is the node set and & (11,, [y)} is the edge set. The topological structure of graph is often represented in the form of an adjacency matrix A | where the elements are 1, indicating whether an edge exists between the corresponding two nodes. Each node v in the graph typically contains a feature vector X; IRd, and the feature vectors of all nodes form feature matrix E RR V|xda
Graph Foundation Models (GFMs). GFMs aim to learn generalizable graph representations, graph vocabulary [2 37 comprising basic transferable units that encode invariant structural and semantic properties across diverse graph datasets. Let D { denote diverse collection of graphs used for training, where each graph 9; = (V,, E,) consists of a node set V,, an edge set &,, and associated node features X,, with varying node counts and feature dimensions across graphs. The GFM training process generally involves learning a universal function f(G;; 0), parameterized by 0, that captures key structural and feature-based patterns. Once trained, the GFM can generate embeddings (9mew; () for new graph Gnew, enabling broad application to diverse downstream tasks.
Model Merging and Expert Merging, Model merging (also called model fusion) refers to the pro- cess of combining multiple pre-trained models into unified one, typically to improve generalization and reduce computational costs in machine learning tasks (00] One common approach is to merge the parameters of multiple models by performing weighted average of their learned parameters. Formally, let [M:s (x)) represent distinct pre-trained models, with 6g being the parameters of
them, then the merged model can be obtained by M(: 0) M(: Σ 0g,0,), where o, represents the weight assigned to M, and Σ a; = ensures that the weights are normalized.
Expert merging 11 is a model merging variant applied within the MoE architecture with top-k gating, where the selected top-k expert models are combined or fused. The current expert merging strategy based on a simple weighted average of the expert parameters. Formally, let [-Mx(: [x) denote K expert models, the merged MoE layer with top-k routing can be expressed as follows:
I. g(x), y= M(x; 0) M(x: Σ,Ez a.),;;
(1)
where y is the gating mechanism (also called the router) which outputs the indices C {1, K} (IZ) = k) and weights of the selected top-k experts, M(::;) () is the fused expert model with parameters x and y represent the input and output of the MoE layer, respectively.
3 A Deep Dive into Existing GFM
The rapid advancement of foundation models in NLP and CV has highlighted the need for similar progress in the development of GFMs. Inspired by the success of MoE in language models, researchers have recently begun to apply MoE to GFMs. Nevertheless, the underlying mechanics of MoE in the context of GFMs remain insufficiently explored. To address this gap, we conducted in-depth experimental investigation of an existing MoE-based GFM to deepen our understanding of MoE in the graph domain. This section begins with brief introduction the recently proposed GFM, AnyGraph 40). followed by a detailed description of the experimental explorations and observations.
3.1 AnyGraph
AnyGraph is a GFM which uses SVD to address the feature dimension misalignment issue and employs MoE architecture to handle structure heterogeneity, i.e., assigning specialized expert to each graph. Specifically, for each graph G E D, first applies SVD to both aaplacian-normalln adjacency matrix A and feature matrix X, and then combines the dimension-reduced representations:
Eg Combine SVD((x))) E RR Vixd
(2)
3

---

where d is the reduced dimensionality of the features. Subsequently, it injects higher-order connectiv- ity information into Eg via non narmmaraettiic message passing mechanism (38, producing:
E
E = A E 1-1
E
En,
where L is the number of paasaag-passing layers. After this, a non pan-aaamettic routing mechanism evaluates all the K experts on Ej, and selects the one with the highest competence score:
E Mr (Miiii 0x), E (1,2, K,,,
(4)
k- arg max VR, du
èT sigmoid (ek,a, Ck.. ekaa, e ex,n,),
(5)
where M₁ denotes the k-th expert, K- is the index of the chosen expert, the (0,1) denotes the competence score of Mr. (Vo,: Up,, Up, Uny) are sampled anchor-posie cettoor-postiive-eeattie triplets, S is the number of sampled triplets, and êk Eg is the node embedding of vy from expert k. Finally, the selected expert Mg- is applicd to obtain the final embedding matrix:
Ma (E1; 0x-),
(6)
which can be used for downstream tasks. Note that Any uses the most competent expert, i.e.. the top-1 expert. More technical details are provided in Appendix B.1
3.2
Experimental Explorations and Observations
Recall
0.30
0.45
NDCG
Performance
0.40
Perforaanc
0.28
0.35
0.30
0.24
Likk Tapl
Lavd Top!
Linkt Nap2-M
Linch Rop
0.25
Luid
0.22
Tip2
Topo
0.20
Linic Teph--
0.20
1st.
2nd
3rd
4th
5th
6th
7th
8th
1000 1250 1500 1750 2000
2500
Rank of Experts
Truining Time un Epoch
Figure Average test performance of experts ranked by competence score across 18 datasets.
Figure 2: Training time vs. performance of top-1 routing, expert ensemble (-E) and merging (-M).
To gain a deeper understanding of the MoE mechanism in AnyGraph, we conducted a series of empir- ical studies. Our initial investigation focused on the performance of individual experts, specifically those with varying competence scores thg. We directly evaluated the pre-trained expert models- ranked by their competence scores the test datasets and observed the following phenomenon:
Observation As illustrated in Figure besides the top-ra top-ranked expert, higher- higher-ranked ranked experts also demonstrate substantial performance. For example, the second- and third-ranked experts outperform the top-ranked expert.
This suggests that the top-k experts collectively possess valuable predictive knowledge. However, AnyGraph exclusively utilizes only the single most competent expert, potentially leaving useful expertise from other high-performing experts untapped. Motivated by this insight, we further explored strategies to incorporate the knowledge embedded in multiple experts. Specifically, we modified the MoE layer to select the top-k experts and aggregate their outputs using weighted averaging (see Appendix B.2 for details), This led the following observation:
ensembling the outputs of the top-k experts significantly boosts performance. However, this improvement
Observation 2: As shown in Figure 2
the cost of increased computational overhead.
To mitigate this overhead while still leveraging multi-expert knowledge, we turned to recent work on expert merging [1] 28 51,, which suggests that averaging the parameters of the top-k experts can effectively reduce training and inference costs. This is because only single forward pass through the fused expert is required. Adopting this approach, we made the following observation:

---

Observation 3: As depicted in Figure while parameter merging reduces time overhead, it may lead degraded performance certain cases, such as on the Linkl group. (Linkl and Link2 refer to two datasel groups comprising 155 and 18 datasets, respectively, which will be described in Section (.7)
This trade-off raises key question: Is it possible to develop method that combines the efficiency of expert merging with the performance benefits of expert ensembling, thereby enabling both effective and efficient utilization of knowledge from multiple experts?
4
Methods
This section provides an aftirmative response to the question. Observation 2 indicates that ensembling the outputs of the top-k experts often yields favorable results. Our key insight is to maintain the efficiency of the parameter-fuse expert while aligning its performance more closely with that of the ensemble. To accomplish this, we propose two methods inspired by knowledge distillation and theoretical principles, which are discussed in Sections 4.1 and 4.2 respectively.
4.1 Knowledge Distillation Enhanced Expert Merging (KDEM)
Knowledge distillation 14 is technique that involves transferring knowledge from large, powerful model (called the teacher model) smaller, weaker model (called the student model). The goal is to enable the stu- dent model to achieve similar performance to the teacher while being more efficient in terms of computational re- sources, such as memory and inference time. This aligns perfectly with our goal. Since the performance of the top- experts' ensemble is excellent, we treat it as a teacher model to teach the parameter- fused expert. This approach allows the knowledge from the top-k experts' ensemble to be transferred to the parameter-fused expert, thereby enhancing its performance. However, traditional knowl- edge distillation methods have two potential issues: (1)
Merged Expert
Expert
Expert 2 ...
Expart
Expert K
Gate
Graph
Figure Architecture of KDEM.
The two-stage training paradigm imposes substantial time overhead, since the teacher model must be fully trained before the student can begin training. (2) The top-k experts selected by the router in the pre-trained teacher model may not be consistent with those selected during the training of the student model, leading potential interference in knowledge transfer.
As demonstrated in Figure we introduce a unified training paradigm that jointly optimizes the teacher and student models to address these issues. Specifically, for given graph G, after obtaining the initial embeddings E, via Eq. the top-k router assigns experts (Mi(- 0,) to it based on the competence scores { pb: == yielding expert indices I and corresponding weights a:
I Topk ((v)),
softmax ((z))
The parameters these & experts are combined through weighted averaging to form a fused expert. This fused expert generates the outputs of the MoE layer, which are subsequently used to compute superviee loss for training the model:
E M(E:;))) M(E;; Σ Ea.0.).
(8)
VEI
Loc
Σ
log
exp(Das.Ds
Dmnx)
B
exp(Uan.ms Umax
(Pug "9g Mng JEB EE
where E, is the embedding matrix from Eq. B is the positive edge, UU,); is the randomly sampled negative edge, Har.p. T êp. is the inner product, é; E É is the embedding of node e,, and 9max is the maximum prediction score in the batch, which is used to avoid numerical instability.
of triplets. in which (Vas Up,) is
To encourage the output of the merged expert to closely resemble the ensemble of the top-k experts while minimizing time overhead, we compute the knowledge distillation loss every few training steps
5

---

and incorporate it into the self-supervised loss to form the final loss function:
Lkd = MSE(E, E).
E = Σ (B,; IEI
(10)
L L- + 7Lkds
(11)
where MSE denotes the mean squared error, and is tunable parameter that controls the influence of knowledge distillation loss. Notably, we do not explicitly train a separate teacher model. Instead, during training, the ensemble of the top-k experts, E, serves the teacher, distilling its collective knowledge into the fused expert. This distilled knowledge 'stored' within individual experts through gradient backpropagation and parameter updates. During the forward pass, the knowledge is recovered through expert ensembling. As the expert parameters are updated, the teacher's 's knowledge is concurrently refined, facilitating subsequent rounds of distillation. Furthermore, the teacher model (E) and the student model (E) share input samples and gating decisions, ensuring they attend the same top-k experts for each graph, thereby eliminating any potential inconsistency in expert selection.
4.2 Parameter Proximity Enhanced Expert Merging (PPEM)
Although KDEM improves the expressivity of expert merging, it also introduces nontrivial com- putational overhead. To minimize this cost, we carried out a thorough investigation. Noting that most experts in MoE architectures are instantiated as MLPs 30 40), we analyzed the theoretical relationship between the output of a terrameter-mergee MLP and the ensemble of its constituent MLP experts. Our analysis culminates in the following theorem (see Appendix [A for a formal proof):
Theorem Given k MLPs (f((;O))== with identical architectures and Lipschitz continuous
activation function, if their parameters converge pairwise, the merged MLP f(; Ex= a;0,) through parameter average approximates the convex combination of their outputs. Formally, let A8; = 01 02
measure the parameter deviation from reference parameter 02. and Σ}= 02 = 1. Then:
lim A0, ,3)
f(x: Σ 0,0,)
> Easf(xx:) =0,
(12)
where || denotes the Euclidean norm on the parameter and output spaces.
Theorem indicates that when the parameters of multiple MLP experts are close each other, the output of the expert obtained through parameter fusion is also close to the ensemble of those experts. Motivated by this, we aim to make the parameters of the top-k MLP experts selected by the router converge to one another, so that the performance of expert fusion approaches that of expert ensemble. However, if the parameters of all the experts become too close, they will lose their diversity and specialization, which would negatively impact the performance of the MoE architecture.
To address this, we propose method that brings the top-k experts sufficiently close to leverage the conclusion of Theorem I while concurrently maintaining diversity across all experts to preserve the advantages of the MoE architecture. Coneretely, we employ an exponential moving average (EMA) to gradually guide the parameters of the top-k experts toward their average:
0, + 30; + (1 B)0, E
(13)
where 0= - Og0; and 3 E (0, 1) controls how rapidly the experts converge: the closer 3 is to 1, the more weight is placed on the expert's previous parameters, resulting in slower updates and more stable behavior. To further regulate this process, we apply the EMA periodically.
This method allows the parameters of the top-k experts selected by the router to gradually move in the same direction, as training progresses, these experts become more similar while retaining their individual specialization, striking balance between expert similarity and diversity preservation, For instance, over 100 epochs with 6000 iterations each, setting 3 = 0.999 and the period of 100, after training, each expert retains only about 0.9996000 as 0.002 its original parameters. By applying the EMA exclusively to the router's current top-k experts while leaving the parameters of the remaining experts unchanged, we preserve the overall diversity and the specialized capabilities of the MoE layer. The training pipeline and time complexity analysis of the two methods presented in Appendix

---

Experiments
5.1 Experimental Settings
Datasets We follow the setup of previous work [40] and use 38 datasets spanning diverse domains. To assess cross-domain generalization ability, we group the datasets in two ways. First, they are split into two major groups, Linkl and Link2, balanced in total and domain-specttn they are categorized into three comain-specifer groups: E-commerce, Academic, and Others. Five datasets are used for the node classification task. Full dataset details are provided in Appendix
edge counts, Second,
Baselines and Evaluation We compare against eleven baselines spanning four categories; details are provided in Appendix E.I We adopt a zero-shot setting where models trained on Link/ group are tested on Link2 group, and vice versa. Results for each dataset group are averaged based on the number of test samples. For GFM baselines, we use their released pre-trained models; other baselines follow few-shot setup. Full evaluation procedures are described Appendix F.2 and implementation details and hyperparameters are documented in Appendix E.3
5.2 Overall Performance
To evaluate the zero-shot prediction capability of our methods, we conduct extensive experiments across various datasets. The evaluation results are shown in Tables and We observe that:
Table Results under zero-shot (AnyGraph, ours) and few-shot (others, 10% train data) settings. For link prediction, we report Recall @ 20 and NDCG@ @ 20; for node classification, we use accuracy and macro-FI score. The best and second-bess results are highlighted in bold and underlined, respectively.
Link1
Link2
E- E-commerce
Academic
Others
Node cls.
Method
Roc
NDCG
Rec
NDCG
Rec
NDCG
Rec
NDCG
Rec
NDCG
Acc
MacFl
GIN
11,80
5.45
21.62
13.41
13.41
8.06
20.61
9.04
18.43
11.85
36.04
30.60
GAT
13.45
6.78
15.30
8.84
9.64
5.78
11.17
4.67
16.17
20.88
54.83
41.61
GPF
6.80
3.27
16.58
9.84
18:72
10.94
14.83
6.41
4.51
3.44
16.29
16.00
GruphPrompt
5.42
3.11
6.10
3.62
6.06
3.36
7.72
3.40
3.42
2.72
23.15
22.89
GraphCL.
20.55
10.76
31.42
19.91
26.05
14.59
28.69
24.62
15.90
48.75
AnyGraph
23.94
12.68
46.42
27.21
26.92
15.05
32.74
15.31
46.83
28.97
64.31
43.24
KDEM
24.11
12.80
51.69
32,60
35.05
22.02
32.15
15.21
46.86
30,00
63.56
45.03
PPEM
24.33
12.93
50.77
31.93
33.99
21.17
32.47
15.47
46.52
29.73
62.65
44.16
Table 2: Comparison with other existing GFMs in zero-shot scenario. GR denotes the Goodreads dataset. Since UniGraph uses Arxiv for instruction fine-tuning in the zero-shot setting, we report its few-shot result instead. The GFT result is also under few-shot setting. As GOFA does not support ranking-baa based link prediction metrics, we report accuracy to align with its evaluation protocol. The GOFA result on PubMed- link under supervised learning setup and is obtained from its paper.
Method
GraphGPT
OpenGraph
UniGraph
GFT
GOFA
Data
Cora
PubMed
Ecom. GR
Arxiv
Cora-link
PubMed-link
Metric
Acc
MacF1
Acc
MacFl
Recall
NDCG
Acc
Acc
Acc
Bascline
18.13
12.72
70.11
64.91
14.44
10.99
31.35
36.29
86.31
93.97
KDEM
62.45
56.32
70.36
68.83
28.30
20.02
61.86
93.03
93.30
PPEM
60.10
55.93
69.94
67.73
22.76
19.76
61.20
93.14
94.10
(I) MoE-based methods substantially outperform other models. Our methods, alongside AnyGraph, employ the MoE mechanism to assign the most suitable experts to each graph, achieving higher zero-shot capabilities. The MoE's effective handling of heterogeneity across different graphs demon- strates its potential to become a powerful backbone for GFMs. Other methods, such as classical GNNs, require supervised training on specific datasets and exhibit weak generalization capabilities. Pretraining then fine-tuning or prompting paradigms also fail to achieve good performance due to substantial distribution disparities among graph datasets. These methods struggle learn transferable knowledge under distribution shifts and can even lead to negative transfer issues (3]
7

---

(2) Compared to AnyGraph, our proposed methods achieve superior performance. For example, KDEM and PPEM achieve 5.27% and 4.35% improvement in recall, respectively, on the Link2 dataset group. The superior performance stems from our model's s ability to effectively combine the knowledge of multiple experts, whereas AnyGraph can only utilize a single expert. Incorporating multiple experts offers the following advantages: (i) More accurate and robust predictions by combining diverse expertise; (ii) Reduced risk of overfitting on single, overly specialized expert. Moreover, our methods yield significantly larger improvements on E-commerce datasets compared to those from Academic or Others domains, suggesting that the E-commerce domain benefits more substantially from the integration of multiple experts. The fine- fnn-grainedd experimental results and analyses for each dataset are provided in Appendix
5.3 Ablation Study
This section demonstrates the effectiveness of the submodules in model by compar- ing with ablated variants. As a reference, we present the results of expert ensembling. From 3 we can observe that:
Table 3: Effectiveness of the key components.
Linkl
Link2
Method
Recall
NDCG
Recall
NDCG
Expert Ensemble
25.69
13.50
48.89
30.39
(1) Knowledge distillation significantly en- hances the performance of expert fusion. The -kd variant, which directly performs a weighted average of the parameters of the selected top-k experts without any addi- tional enhancements, shows degraded per- formance compared to KDEM. This may be attributed to the simple fusion method causing interfer- ence 44 among the expert parameters, leading to the loss of some knowledge during expert fusion. In contrast, KDEM improves performance by distilling knowledge from the expert ensembling into the fused expert, guiding it to retain the complementary strengths of each specialist expert.
KDEM-kd KDEM
22.42
11.66
48.34
29.72
24.11
12.80
51.69
32.60
PPEM-EMA
22.42
11.66
48.36
29.62
PPEM
24.33
12.93
50.77
31.93
(2) Encouraging parameter proximity among the top-k experts substantially improves the performance of expert fusion. Grounded in theoretical derivation, PPEM gradually brings the parameters of the selected experts closer together, aligning the output of the fused experts with that of expert ensembling, thereby boosting the overall performance.
(3) Expert merging is a promising method for leveraging the knowledge of multiple experts. Since ensembling multiple experts incurs significant training overhead, we evaluated only the top-2 and top-3 expert ensembles, and the results shown are the best among these. As demonstrated in the results, KDEM and PPEM achieve performance comparable expert ensembling on the Linkl group and even surpass it on Link2, highlighting the potential of expert fusion strategies.
5.4
Computational Efficiency Analysis
(s) Except Ensemble
1200
Except Ensemble
230
PPEM KDEM Top-1
Expert Merging
2000
PPEM KDEM
Expert Merging Expert Ensemble
Expert Ensemble
Ensemble
220
Ensemble
1100
Top-1
1800
210
225
1000
200
1600
260
Time (
190
Time (s)
Time
1400
245
1200
170
230
Top2
Top3
Top4
Top5
Top6
Top?
Top8
Top2
Top3 Top4 Top5 Topó Top? Top8
(a) Training time per epoch on Link1
(b) Inference time on the entire Link2
Figure 4: Computational efficiency compared with top-1 routing of AnyGraph, standard expert merging, and expert ensemble (expert ensemble: right y-axis; all other methods: left y-axis).
This section evaluates the computational efficiency of KDEM and PPEM compared to top-1 routing, standard expert merging, and expert ensemble. As depicted in Figure 4 observe: (1) Relative to

---

the other methods, the expert ensemble imposes substantial training and inference time overhead. (2) Compared to top-1 routing, using top-k routing inherently increases training overhead, with training time gradually growing as k increases. (3) KDEM incurs marginal increase in training time over standard expert merging (averaging 4.8%), whereas PPEM exhibits almost no increase. Note that relatively small & value often suffices for excellent performance (refer to Appendix G.2), so the additional training overhead is limited. (4) The inference times for expert merging methods exhibit only minor fluctuations around that of top-1 routing. These fluctuations are likely due to device measurement variability, indicating that these expert merging methods introduce negligible increases in test time. These findings align with our time complexity analysis in Appendix C.2
5.5 Further Discussion
5.5.1 Discussion of Two-Stage Knowledge Distillation and KDEM
pretraining a teacher model before performing knowledge distillation only incurs substantial training overhead, but also risks inconsistency between the experts selected during teacher training and those used in the distillation phase, due to the dynamic nature of expert assignment during training. In contrast, KDEM ensures that the teacher and student attend the same set of experts on the same input, thereby maintaining consistency in the information flow, Moreover, this distillation strategy makes the knowledge transfer process more stable: the teacher's outputs are continuously updated throughout training, allowing the student model to optimize along convergence path aligned with the teacher. As shown in Table 4 the performance of the two-stage approach- first training the teacher and then distilling is clearly inferior to our proposed KDEM.
As mentioned Section 4.1
Table 4: Performance comparison between KDEM and the two-stage approach of training teacher model followed by knowledge distillation.
Linkl
Link2
Method
Recall
NDCG
Recall
NDCG
Two-stage KD
23.19 24.11
12.03
47.13
28.61
KDEM
12.80
51.69
32.60
5.5.2 A Deeper Analysis of PPEM Effectiveness
We posit that expert merging only yields benefits when the top-k experts are both sufficiently similar-so so as to minimize parameter interference-at and yet retain enough diversity to maximize overall knowledge capacity. To test this, we performed two complementary experiments:
Cora
Products-Home
Proteins-1
Exper2
Exser
Spert
Expent
40
Exper
Expent
20
Faseer
Expert
Expert
Taser:
Expert
-0-
-20
400
-40
-00
-50
-25
25
50
-00
25
50
-50
-60
20
20
40
Figure 5: T-SNE visualization of node embeddings generated by all 8 experts trained with PPEM on the Cora (academic), Products-Hooee (e-commerce), and Proteins- (biological) datasets. random sample of 2.500 nodes is selected from each graph for visualization.
(1) Expert Similarity Analysis:
We trained two models, one using PPEM and one using the vanilla expert merging strategy, and then measured the cosine similarity between the outputs of the top-2 selected experts each dataset. Under PPEM, the mean similarity across all test sets was

---

0.953, compared to just 0.828 for the vanilla approach. The higher similarity under PPEM reduces interference between expert parameters, facilitating a more effective merge of the top-k experts.
(2) Expert Diversity Verification: Figure 5 presents a t-SNE visualization of the output repre- sentations from all 8 expert models trained with PPEM. Despite the high similarity between the top-2 experts, the experts remain well separated in the embedding space, confirming that diversity preserved. This maintained heterogeneity allows the merged experts cover broader range of knowledge, thereby improving the model's generalization performance.
5.5.3 Visualization of Routing Mechanism
We visualize expert routing to intuitively demonstrate its working mechanism. Figure [ depicts the competence scores of a PPEM model trained on Link2 and tested on Linkl. From this visualization, we observe: (1) Shared expert competence. Many datasets receive similarly high scores from multiple top-ranked experts. For example, experts () and achieve nearly identical scores on the Prod- tech dataset, suggesting that combining knowledge from several experts is advantageous. (2) Domain-consisten assignments. Related datasets are routed to the same ex- perts. Notably, both citation datasets assign their highest scores experts O, and 6, indicating that the router cap- tures underlying domain similarities. A further discussion of the routing mechanism can be found in Appendix H
Competence Scores
Yelp.t
.59 .722
7022
914
919
780
036
946
Steam-t-
572
216
030
860
0.9
Amazon-t
537
363
973
969
372
934
PPA
#42 .37
352
926
532
802
933
932
Prod-tech
589
911
989
915
580
972
Cit-2019
.98
Citatlon-20C
299 .399
260
933
088
968
398
392
0.6
Email
575 947
315
960
905
850
576
972
2
Expert
Figure 6: Competence scores of different experts in a top-2 merged PPEM model.
6 Related Work
Graph Foundation Models GFMs are still in their infancy: most are tailored to specific tasks or domains, with only handful demonstrating preliminary cross-domain and cross-task transferability. For instance. ULTRA [9] and GraphAny (49] excel at knowledge graph completion and node classification, respectively, while DiG [50] and MiniMol [18) focus on the molecular domain. More versatile methods leverage large language models: OFA (23) generates textual node and edge descriptions and embeds them in a unified space: GraphGPT 33 and LLaGA 4 align graph and
text embeddings via projection layers before prediction; UniGraph [3] and GOFA [9] design self- supervised pretraining objectives learn graph-langueg language encoder that jointly captures structural and textual information. However, these methods all rely on text features and cannot directly exploit the original graph attributes. AnyGraph (40) addresses structural heterogeneity by dispatching specialized expert per graph via an MoF layer, enabling improved generalization, but it still underutilizes the complementary knowledge encoded in its multiple experts.
Additional literature relevant to this study is discussed in Appendix
7 Conclusion
This paper presents an in-depth investigation of an existing MoE-based GFM, revealing an interesting finding: the performance of the second and third-ranked experts assigned by the router exceeds that of the top-ranked expert. This inspires us to explore the use of multiple experts in MoE architectures. Additionally, we find that integrating the outputs of multiple experts incurs significant computational overhead, and directly applying expert merging leads to suboptimal performance. To address these challenges, propose two enhanced expert merging strategies: (1) using knowledge distillation to align the performance of the merged expert with that of the expert ensemble; (2) based thcoretical principles. using parameter proximity to bring the selected expert parameters closer together. Extensive experiments demonstrate that our methods achieve excellent performance without introducing significant time overhead. This work provides new insights into the role of MoE mechanisms in GFMs and establishes a foundation for efficient expert utilization within MoE architectures. We envision applying our methods to MoE-basee large language models, alleviating their computational and inference-time overhead while preserving strong performance.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by Key Project of the National Natural Science Foundation of China (U23A20316) and CCF-Tencent Rhino-Bird Open Research Fund (CCF-Tencent RAGR20250115),
References
[1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021.
[2] Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint urXiv:2407.062000 2024.
[3] Yuxuan Cao, Jiarong Xu, Carl Yang, Jiaan Wang, Yunchao Zhang, Chunping Wang, Lei Chen, and Yang Yang. When to pre-train graph neural networks? from data generation perspective! In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD, pages 142-153, 2023.
[4] Runjin Chen, Tong Zhao, Ajay Kumar Jaiswal, Neil Shah, and Zhangyang Wang. Llaga: Large language and graph assistant. In The Forty-first International Conference Machine Learning, ICML, 2024.
[5] Damai Dai, Chengqi Deng, Chenggang Zhao, X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL, pages 1280-1297, 2024.
[6] Taoran Fang, Yunchao Zhang, Yang Yang, Chunping Wang, and Lei Chen, Universal prompt tuning for graph neural networks. In The Seirth- Conference on Neural Information Processing Systems, NeurlPS, pages 52464 52464-52489, 2023.
[7] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
[8] Michael Galkin, Michael Bronstein, Jianan Zhao, Haitao Mao, and Zhaocheng Zhu. Foundation models in graph & geometric deep learning. Medium (Towards Data Science), 2024.
[9] Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. Towards foundation models for knowledge graph reasoning. The Twelfth International Conference on Learning Representations, ICLR, 2024.
[10] Haoyu Han, Juanhui Li, Wei Huang, Xianfeng Tang, Hanqing Lu, Chen Luo, Hui Liu, and Jiliang Tang. Node-wise filtering in graph neural networks: A mixture of experts approach. arXiv preprint arXiv:2406.088: 03464, 2024.
[11] Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Merging experts into one: Improving computational efficiency of mixture of experts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 14685-14691, 2023.
[12] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. Lightgnn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR, pages 639-648, 2020.
[13] Yufei He, Yuan Sui, Xiaoxin He, and Bryan Hooi. Unigraph: Learning unified cross- domain foundation model for text-attributed graphs. Proceedings of the 31 ACM SIGKDD Conference Knowledge Discovery and Data Mining V.1, KDD, page 449,45,, 2025.
11

---

[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 3.XI::5503.0253. 2015.
[15] Zhenyu Hou, Haozhan Li, Yukuo Cen, Jie Tang, and Yuxiao Dong. Graphalign: Pretrain- ing one graph neural network on multiple graphs via feature alignment. arXiv preprint 2:Xiv:2466.029533 2024.
[16] Fenyu Hu, Liping Wang. Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. Graphdive: Graph classification by mixture of diverse experts. In Proceedings of the Thirty-Fine First International Joint Conference on Artificial Intelligence, IJCAI, pages 2080-2086, 2022.
[17] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401. 2400000800000088,, 2024.
[18] Kerstin Klaser, Blazej Banaszewski, Samuel Maddrell- Callum McLean, Luis Müller, Ali Parviz, Shenyang Huang, and Andrew W Fitzgibbon. Minimol: A parameter-efficien foundation model for molecular learning. In ICML 2024 Workshop on Efficient and Accessible Foundation Models for Biological Discovery, 2024.
[19] Lecheng Kong, Jiarui Feng, Hao Liu, Chengsong Huang. Jiaxin Huang, Yixin Chen, and Muhan Zhang. Gofa: A generative one-for-all model for joint graph language modeling. In The Thirteenth International Conference on Learning Representations, ICLR, 2025.
[20] Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey, arXiv preprint arXiv:2309. 15698, 2023.
[21] Yuhan Li, Peisong Wang. Zhixun Li, Jeffrey Xu Yu, and Jia Li. Zerog: Investigating cross- dataset zero-shot transferability in graphs. In Proceedings the on Knowledge Discovery and Data Mining, KDD, pages 1725-1735, 2024.
the 30th ACM SIGKDD Conference
[22] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-Ilava: Mixture of experts for large vision-language models. arXiv preprint 2rXiv:2401.15944 15947, 2024.
[23] Hao Liu, Jiarui Feng. Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. The Twelfth International Conference on Learning Representations, ICLR, 2024.
[24] Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang, Lichao Sun, Philip S Yu, et al. Graph foundation models: Concepts, opportunities and challenges. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[25] Zemin Liu, Xingtong Yu, Yuan Fang, and Ximming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference, WWW, pages 417-428, 2023.
[26] Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of- experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI., pages 6159-6172 2024.
[27] Haitao Mao, Zhikai Chen, Wenzhuo Tang, Jianan Zhao, Yao Ma, Tong Zhao, Neil Shah, Mikhail Galkin, and Jiliang Tang. Position: Graph foundation models are already here. In The Forty-first International Conference on Machine Learning, ICML, 2024.
[28] Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing. Transactions on Machine Learning Research, 2024.
[29] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
12

---

[30] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. In The Fifth International Conference on Learning Representations, ICLR, 2017.
[31] Shashank Sheshar Singh, Samya Muhuri, Shivansh Mishra, Divya Srivastava, Harish Kumar Shakya, and Neeraj Kumar. Social network analysis: survey on process, tools, and application. ACM Computing Surveys, 56(8):1-39, 2024.
[32] Maciej Sypetkowski, Frederik Wenkel, Farimah Poursafaei, Nia Dickson, Karush Suri, Philip Fradkin, and Dominique Beaini. On the scalability of gnns for molecular graphs. In The Thirty-eigtth Conference on Neural Information Processing Systems, NeurIPS, volume 37, pages 19870-19906, 2024.
[33] Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR, pages 491-500, 2024.
[34] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In The Sixth International Conference on Learning Representations, ICLR, 2018.
[35] Haotao Wang, Ziyu Jiang, Yuning You, Yan Han, Gaowen Liu, Jayanth Srinivasa, Ramana Rao Kompella, and Zhangyang Wang. Graph mixture of experts: Learning on large-scale graphs with explicit diversity modeling. In The Thirty-seventh Conference on Neural Information Processing Systems, NeurIPS, 2023.
[36] Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary- loss-freee load balancing strategy for mixture-of-expeee arXiv preprint arXiv:2408. 1566445644 2024.
[37] Zehong Wang. Zheyuan Zhang, Nitesh V Chawla, Chuxu Zhang, and Yanfang Ye. Gft: Graph foundation model with transferable tree vocabulary. In The Thirty-eighth Conference on Neural Information Processing Systems, NeurIPS, 2024.
[38] Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Wein- berger. Simplifying graph convolutional networks. In Proceedings of the 36th International Conference on Machine Learning, ICML, pages 6861-6871, 2019.
[39] Shirley Wu, Kaidi Cao, Bruno Ribeiro, James Zou, and Jure Leskovec. Graphmetro: Mitigating complex graph distribution shifts via mixture of aligned experts. In The Thirty-eighth Conference on Neural Information Processing Systems, NeurlPS, 2024.
[40] Lianghao Xia and Chao Huang. Anygraph: Graph foundation model in the wild. arXiv preprint arXiv:2408. 10700, 2024.
[41] Lianghao Xia, Ben Kao, and Chao Huang. Opengraph: Towards open graph foundation models. In Findings of the Association for Computational Linguistics: EMNLP, pages 2365-2379, 2024.
[42] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In The Seventh International Conference on Learning Representations, ICLR, 2019.
[43] Zhengzhuo Xu, Bowen Qu, Yiyan Qi, SiNan Du, Chengjin Xu, Chun Yuan, and Jian Guo. Chartmoe: Mixture of diversely aligned expert connector for chart understanding. In The Thirteenth International Conference on Learning Representations, ICLR, 2025.
[44] Prateek Yaday, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In The Thirth- Conference on Neural Information Processing Systems, NeurlPS, pages 7093-7115, 2023.
[45] Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in lim,, milms, and beyond: Methods, theories, applications and opportu- nities. arXiv preprint 240800240,,,7666, 2024.
13

---

[46] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen. Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In The Thirty-fourth Conference on Neural Information Processing Systems, NeurIPS, volume 33, pages 3812-5823, 2020.
[47] Hanqing Zeng, Hanjia Lyu, Diyi Hu, Yinglong Xia, and Jiebo Luo. Mixture of weak and strong experts on graphs. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
[48] Guibin Zhang, Xiangguo Sun, Yanwei Yue, Chonghe Jiang, Kun Wang, Tianlong Chen, and Shirui Pan. Graph sparsification via mixture of graphs. In The Thirteenth International Conference Leurning Representations, ICLR, 2025.
[49] Jianan Zhao, Zhaocheng Zhu, Mikhail Galkin, Hesham Mostafa, Michael M. Bronstein, and Jian Tang,
node classification on arbitrary graphs. In The Thirteenth International Conference on Learning Representations, ICLR, 2025.
[50] Shuxin Zheng, Jiyan He, Chang Liu, Yu Shi, Ziheng Lu, Weitao Feng, Fusong Ju, Jiaxi Wang, Jianwei Zhu, Yaosen Min, et al. Predicting equilibrium distributions for molecular systems with deep learning. Nature Machine Intelligence, pages 1-10, 2024.
[51] Zexuan Zhong, Mengzhou Xia, Danqi Chen, and Mike Lewis. Lory: Fully differentiable mixture-of-expprts for autoregressive language model pre-training. In The First Conference Language Modeling, COLM, 2024.
[52] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixtere-of-experts with expert choice routing. In The siitt Conference on Neural Information Processing Systems, NeurIPS, volume 35, pages 7103-7114, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We include the paper's contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

We provide clear assumptions and proofs for each theorem and lemma. Please refer to Appendix A
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed experimental setup in Appendix which can be used to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes)
Justification: We use only publicly available datasets (such as those released by AnyGraph co/ so/datasets/ http///ud/////// htte//Anyyraaaaaaaaaaae datasets), and we include an anonymized zip file with all our codes.
htttps://bugg::
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ccm cc/
Snuiides/CodeSubmisss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
www.c//wwgeeeeeeeee Sawelee//oooSSbbmiiiea SoroodeeumisssiooPPP.. wwiiiisssiiipcyooioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed experimental setup in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow existing studies in setting up the experiments and currently report only the mean performance over 5 (or 10) random seeds, capturing variability due to random initialization, but without including error bars. We can provide the standard deviation if needed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Section 5.4 for the training and testing time and Appendix E.3 for the hardware device.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms to the ethical guidelines in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Appendix K
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This research poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We only publicly available datasets and open-source codes, and explicitly cite each asset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed documentation for the submitted code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bettts://neurre /0tps//eec.ooooeeeeee////2////// for what should or should be described.
21

---

A Proof of Theorem I
To prove Theorem |
we require the following lemma.
Lemma 2. Given the same conditions as in Theorem [ if the parameters of k. ML.Ps are sufficiently close, the output of the merged MLP through parameter averaging will converge to the output of each individual MLP. Specifically, for any j E 21, ,k).
lim Oi (2." R))
f(x: Σ a,,0,) 1x(x0;)) 0.
(14)
|
Proof of Lemma 2 case follows similarly.
Without loss of generality, we prove the case for h: = 2 and j = 1; the general
Let 01 = (w b ,.. W( , 1 and 02 = (
w ,
bb4) denote the
parameters of two MIPss with L, layers, where ( b(9) are the weight matrix and bias of the 1-th layer, respectively. We proceed by induction on the layer index I.
Base Case (1 (1=1) 1): Let z()) and hj" be the outputs of the merged MIP ʃ and the first MLP fi at layer 1, respectively. Then:
(1)) h1 101 = 0 (al w a.g x x
+ azbbg") a (wi"x x + xxb("))))))))))")()) b(")|
I C ( ((1))))) + +az x a1b, (2) + tabb (2) - (W("'x x + bltt)]
(W!") - w x + (b² 11) -b,i
Ca2
(15)
[[5(" w ||x| + |b - -b0 1
S Co2
+ 0
as || = 1|0:
821l - 0.
where o-)) is the activation function with Lipschitz constant C, and the first inequality arises from the Lipschitz continuity of the activation function.
Inductive Step (1 2 2): Assume the claim holds for layer - (i.e., ||ū1-1) h1 (1-11) + 0). For layer I, the outputs of f and fi are denoted as follows:
こ a ((03 W/ (0) + az w.!" g0-1) +arb
(2) + taabb")
(16)
h = σ (w 11 ((-1) + bb()).
(17)
then we have
(2(00
h (0)
= 0 (( [1 w (1) az w!!)) ( + tab (1) + aagb)) - (w(" ((-1)) + bSy)
sc 81 (( (1) + az w z(1-1)
+ aa
( taab 10) (w(0n (- + bbo))
(18)
C ( (2((1) - h1 ((-111) + 02 (w - w(O) راق + a2 (b.)) b,n)l
5 C (Iw -| - h [1' +  w (1) w( |لااسا||| a2 1by0 b (11)
0 as
| + 0,
where the last step follows from the inductive hypothesis and || being bounded (since x and activation functions are bounded). By induction, the lemma holds for all layers.
Proof of Theorem
7 The expression inside the norm of Eq. 12 can be rewritten as:
aj
f(x:
a.0,)
(x((x())
(19)
i=1
22

---

Therefore, Theorem ! can be proven as follows:
lim
f(::
a,O,)
Σ
||00 (2,
1=1
j=1
k
lim
Σ
0
[x:
Σ
a,0i)) - 15((x; 0j)
(by Eq. (9)
-o, (2, .8]
==1
i=1
k
lim
Σ aj
f(x; Σ
ag0:) - Ox(x;0))
(triangle inequality)
(20)
||0011- O..VE
==1
(jj
lim
f(x: Σ a,0,) - 15(x;0,))
(linearity of limits)
|A001I-+0,viiE
3=1
= 0. (by Lemma
Most common activation functions used in neural networks are Lipschitz continuous, with varying Lipschitz constants: ReLU (Lipschitz constant 1), Leaky ReLU (Lipschitz constant max(1, a)) where is the slope parameter), ELU (Lipschitz constant max(1, a)), Sigmoid (Lipschitz constant 0.25), Tanh (Lipschitz constant 1), Softplus (Lipschitz continuous with constant 1), GELU (Lipschitz continuous but no simple closed-form constant). Thus, our theorem broadly applicable.
Remark:
B Details of AnyGraph and Observations
B.1 AnyGraph
For each graph 9 E D, AnyGraph begins by applying SVD to both its Laplacian normaliize adjacency matrix À E [R|V and its node feature matrix X E [R|Vxd,, yiiliii,yielding yielding
U1, V1 = SVD(A), U2, V2 = SVD(X).
(21)
Here, U1, V1 E V and U2 E R Vixd V2 Rdaxd The initial node embeddings then formed by linearly combining these factors and normalizing:
Eg = LayerNorm (U.VA
VIVA + ViVA, (Hip(U.AA))
(22)
where Flip(-) denotes reversing the order of the d dimensions in each row. Next, it inject multi-hop connectivity information into Eg via a simplified GCN module to obtain E, (3q.33). The enriched embeddings E, are then passed through an MoE. layer, where each graph is dispatched to its most competent MLP expert according to the routing rule in 55.
To alleviate the expert training imbalance issue, AnyGraph employs a training frequency regulariza- tion method to adjust the competence score follows:
(
3/4 1/k 1 1
mE
P
(23)
Σ m.
where my is the current training steps of k-th expert, P is hyperparameter controlling the strength of recalibration, and vig is the resulting adjusted score for the k-th expert. If an expert is trained too frequently, its competence score is multiplied by penalty factor.
Finally, the entire model is optimized end-to-end using a self-supervised loss:
exp(jan.pb 9mux)
Lcs
Σ
log Σ Eeny exp( 1max)
(24)
B
(Ua,
JEB
where (Va,, Uno) is anchor cittoo--oosiiiveeeattii triplet, the term Øc,j = _,e is the inner product of node embeddings, each É the output of the chosen expert (Fq. 6)), and Jmmx = maxi, 0i.3 is used for numerical stability.
23

---

B.2 Experimental Details of Observations
This section provides detailed explanation of the experiments presented in Section 3.2
Observation We evaluate the performance of experts with different rankings on the test set using the publicly available code and pretrained models released by AnyGraph The authors provide two pretrained models, trained separately on the Link/ and Link2 dataset groups. To assess zero-shot generalization, we test the model trained
dataset group against the datasets in the other group. Figure [ presents the results of the model trained on Link] and evaluated on the Link2 dataset group. The results for the reverse selting-traines on Link2 and testing on Linkl-are shown in Figure 7 As we can see, although the second- and third-ranked experts do not perform as dramatically as in Figure T they remain competitively strong. Moreover, we observe sudden performance drop as expert rank increases for instance, from the fourth-ranked to the fifth-ranked expert in Figure 7 The same trend is apparent in Figure [ to the prediction, while the lowest-ranked few experts may be harmful.
This suggests that the top-k experts may all contribute positively
0.25
Recall
Performance
NDCG
0.20
0.15
0.10
0.05
1st
2nd
3rd
4th
5th
6th
7th
8th
Rank of Experts
Figure Average test performance of experts ranked by competence score across datasets (Linkl).
Meryed Expert
Expert 1
Expert 2
Expert k
Expert K
Expert
Expert 2
Expert
Expert K
Gate
Gate
Graph
Graph
Figure 8: Top-k expert ensembling (expert out- puts are averaged).
Standard expert merging (expert pa- rameters are averaged without enhancement).
Figure 9:
Observation 2:
As depicted in Figure we modify the MoE layer of AnyGraph by selecting the top-k experts and aggregating their outputs using weighted average. Specifically, we replace the top-1 gating in Eq. with top-k gating mechanism:
= Topk ((w))
(25)
Here, I denotes the index set of the top-k experts with the highest competence scores. And the final node embedding matrix in ( becomes:
e
E
(E)
05
Eyer eb,
(26)
OOfficia repository: httt::///gttub..com//UUSUSSS

---

where :M(:;0;) 8,) denotes the i-th expert, and is its weight, obtained by applying the softmax function the competence scores of the selected experts.
Observation 3: As demonstrated in Figure / to reduce the training and inference costs, we further modify the MoE layer adopt expert fusion approach. Concretely, fuse the parameters of the lop-k selected experts into a single expert by replacing Eq. with:
et
= M(E Σ
a,0:),
04 ΣjEz evay
(27)
IEI
where M is the parameter-merg merged expert.
C
Training Pipeline and Complexity Analysis
C.1 Training Pipeline
The training procedures of KDEM and PPEM are presented in Algorithm and Algorithm 2 respectively. In KDEM, the merged expert is used to compute self-supervised loss, to which a knowledge-distilaatoo term is periodically added, and the parameters are updated based on the combined objective. In PPEM, expert parameters first updated using the self-supervised loss, and then the top-k experts are periodically updated via an exponential moving average.
Algorithm 1 Training Pipeline of KDEM
Input: Training graphs DD preprocessed embeddings E1, expert models [MMx(;0a)]k period Ti of applying knowledge distillation.
1: for every epoch do for i-th batch of triplets B {(V Up, Ung) } do
2:
3:
Get the indices I and weights a of the top-k experts via Eq.
4:
Merge the parameters of the top-k experts
5: Get Ej and compute É using the merged expert (Eq, 8
6:
Compute self supervised loss Lee using È (Eq. 9)
7:
if (i+ 1) is divisible by Tj then
8:
Compute Cid (Eq.
9:
end if
Compute the final loss (Eq. 11)
11:
Update parameters {0c her via backpropagation 12: end for
13:
end for
14: return Expert parameters {0x [K
C.2 Complexity Analysis
The initial embedding computation and expert routing be preprocessed, with per-graph time complexities of - - L) and O(V] dd L/ S d. K), respectively, where I. and T/' denote the number of message passing and MLP layers. Next, analyze the per-batch time complexity of the MoE module. Both our method and AnyGraph process data using one expert (merged expert vs. top-1), with time complexity of O( d² L'). ). In addition, our methods incorporate expert merging, which involves averaging of the expert parameters with cost of O d² L'), where < [B]. KDEM periodically applies expert ensembling, and when averaged over each batch, the 0
time complexity is 0( & | d² L'). where Ti the period. In our experiments, Ti is set to 100. PPEM periodically applies an exponential moving average, introducing a time complexity of 0 de L'). Note that during inference, methods require expert merging only once per dataset, making the associated computational cost practically negligible.
25

---

Algorithm 2 Training Pipeline of PPEM
Input:
Training graphs D. preprocessed embeddings E1, expert models (MMx(;Ox))Fe) period 72 of exponential moving average (EMA).
for every epoch do for i-th batch of triplets B = {(Vas + op,, Un,)} do
2:
3:
Get the indices I and weights a of the top-k experts via Eq.
4:
Merge the parameters of the top-k experts
5:
Get E and compute É using the merged expert (Eq. 8
Compute self supervised loss Lee using E (Eq 9 Update expert parameters {0: via backpropagation if (i + 1) is divisible by T2 then Update expert parameters {0:JEz via EMA (Eq. 10: end if
6:
7:
8:
9:
11:
end for
12:
end for
13:
return Expert parameters {0
D
Datasets
Our study leverages extensive collection of graph datasets drawn from diverse domains, comprising a total of 14,437,372 nodes and 199,265,66 6,,888 edges. All datasets in this study are obtained from prior presented in Table These datasets are organized into multiple categories and groups to address specific research questions and evaluation needs. All datasets are used for link prediction, five additional datasets are also used for node classification.
research 333 21 40 The dataset statisties
D.1 Dataset Categories
1. E-commerce Datasets This category includes 13 datasets extracted from various e-commerce contexts such as online retail services and user-rating platforms. While these datasets display variability in node feature availability and generation, some share common methods. For example, Amazon-text, Steam-text, and Yelp-text employ one feature generation strategy, whereas Fitness, Photo, and Goodreads consistently apply an alternative method.
2. Academic Network Datasets This category encompasses 9 datasets that model scholarly interactions such as paper citations and author collaborations across diverse research disciplines. The datasets vary in feature construction methods, employing techniques like bag-of-words models, NLP-based embeddings, and features derived from large language models. Datasets in this category include Cora, PubMed, Citeseer, Arxiv, Arxiv-t (which uses a unique feature derivation method), CS, Citation-2019. Citation-20Centin and OGB-Collat
3. Biological Information Networks This set comprises six datasets centered biological entities such as proteins, drugs, and discases. Specifically, this group includes the OGB-DDI and OGB-PPA networks, along with four species-specife protein relation datasets designated Proteins-O, Proteins-1, Proteins-2, and Proteins-3.
4. Miscellaneous Datasets In addition to the primary domains, we incorporate datasets originating from distinct and less thematically consistent domains. These datasets provide additional diversity and include: the Email-Enron network (emails), Web-Stanford (website connectivity), RoadNet-PA (road networks), P2P-Gnutella06 (peer-to-peer web), and Soc-Epinions (trust networks).
D.2 Dataset Grouping for Performance Evaluation
To streamline performance comparisons and effectively prevent information leakage, particularly in zero-shot settings, the datasets are further organized into groups. This grouping is based on their respective sources, feature generation methods, and specific domains, as detailed below:
26

---

Table 5: Statistics of the datasets. All datasets used for link prediction, and five additional datasets are also used for node classification (those marked with check the last column).
Datasets
Nodes
# Edges
Feats
Groups
Node Cls.
DDI
4,267
1,334,889
0
Link2 / Others
Collab
235,868
1,285,465
128
Link2 Academic
ML
9,746
920,193
0
Link2 / E-commerce
MI.10m
80,555
9,200,050
Link2 E-commerce
Amazon-book
144,242
2,984,100
0
Link / E-commerce
PPA
576,289
45,495,642
58
Link1 Others
Yelp2018
69,716
1,561,406
0
Linkl / E-commerce
Gowalla
70,839
1,027,370
Link2 E-commerce
Cora
2,708
10,556
1433
Link2 / Academic
PubMed
19,717
88,648
500
Link I Academic
Citeseer
3,327
9,104
3703
Link / Academic
Proteins-O
25,449
11,660,646
Link2 Others
Proteins-1
6,568
1,845,900
0
Link2 / Others
Proteins-2
18, 108
7,418,688
0
Link2 Others
Proteins-3
13,015
3,962,930
0
Link2 Others
Products-home
9,790
131,843
100
Link E-commerce
Products-tech
47,428
2,077,241
100
Link E-commerce
Yelp-t
22,101
277,535
1536
Link / E-commerce
Amazon-t
20,332
200,860
1536
Link / E-commerce
Steam-t
28,547
525,922
1536
Link / E- E-commerce
Goodreads
676,084
8,582,306
768
Link2 E-commerce
Fitness
173,055
1,773,500
768
Link2 / E-commerce
Soc-Epinions
75,879
508.837
Linkl Others
Email-Enron
36,692
183,831
0
Link / Others
Web-Stanford
281,903
2,312,497
Link2 Others
RoadNet-PA
1,088,092
1,541,898
0
Link2 / Others
P2P-Gnutella
8,717
31,525
128
Link Others
Citation-2019
765,658
1,917,381
128
Link 1 / Academic
Citation 20Century
1,016,241
5,565,798
128
Link Academic
Arxiv
169,343
1,166.243
128
Link2 / Academic
Arxiv-t
169,343
1,166,243
768
Link2 Academic
Photo
48,362
500,939
768
Link2 / E-commerce
CS
18,333
163.788
6805
Link2 Academic
1. Balanced Groups: Two primary groups, Linkl and Link2, are formed exclusively from link prediction datasets. The total number of edges each group approximately equal, and within each domain, the edge counts of the individual datasets are likewise comparable. Datasets collected from the same source (e.g., Movielens- IM and Movielens- 10MM) or those sharing identical feature generation methods (e.g., Fitness and Photo) are kept together to ensure evaluation integrity.
Linkl Group: Contains 15 datasets such Products-tech, Yelp2018, Yelp-textfeat, Products- home. Steam-text, Amazon-text, Amazon-booke Citation-2019. Citation-2UCenturyy PubMed, Citeseer, OGB-PPA, 22P-Gnutella06, Opi-iopni, and Email-Enron.
Comprises 18 datasets, i.c.. Photo, Goodreads, Fitness, Moviclens- IM, Movielens- 10M, Gowalla, Arxiv, Arxiv-t, Cora, CS, OGB-Collab, Proteins-O, Proteins-l, Proteins-2, Proteins- 3, OGB-DDI, Web-Stanford, and RoadNet-PA.
Link2 Group:
Domain-Specific Groups: To enable a more focused analysis of model performance and groups by
generalization capabilities across distinct thematic areas, define consolidating datasets based on their primary domain origin or application. This categorization allows for targeted assessment of how models perform, adapt, exhibit specific behaviors within these coherent operational contexts, revealing domain-specific strengths or limitations:
E-commerce Group: Containing all datasets from the e-commerce categories.
27

---

Academic Group: Containing all datasets from the academic categories.
This comprises all biological datasets along with the other miscellaneous datasets such as Email-Enron, Web-Stanford, RoadNet-PA, P2P-Gnutella06, and Soc-Epinionsl.
Others Group:
E
Experimental details
E.1 Baselines
We benchmark our approaches against eleven representative baselines spanning four distinct cate- gories: recent graph foundation models, graph prompt learning methods, self-supervised pre-training methods, and classic graph neural networks.
Graph Foundation Models
We adopt six recently proposed GFMs, which described follows:
designed to generalize across diverse and unseen graph data by integrating a unified graph tokenizer, a scalable graph transformer, and LLMeenhanced data augmentation. This architecture enables strong zero-shot learning performance by capturing global topological patterns and adapting to varying graph properties without requiring retraining.
OpenGraph [4]
GraphGPT [33] integrates LLMs with graph structural knowledge through a graph instruction tuning paradigm. It employs a dual-stage instruction tuning starrtnng with superviud graph matching to align graph structures with natural language, followed by task-specific fine- tuning-to enhance the L.M'' ability to understand and reason graph data, achieving strong performance both supervised and zero-shot graph learning tasks.
versatile GFM designed to address the challenges of structure and feature heterogeneity across diverse graph datasets. Built upon an MoE architecture, it incorporates dynamic expert routing mechanism that enables efficient adaptation new graph domains and exhibits scaling law behavior, where performance improves with increased data and model size.
AnyGraph 40
a unified foundation model for text-attributed graphs, which employs novel cascaded architecture of language models and GNNs alongside a self-supervised pre-training algorithm based on masked graph modeling and instruction tuning.
UniGraph [3]
GFT 37 treats computation
transferable patterns and learns graph vocabulary by performing several computation tree reconstruction tasks, encoding general graph knowledge into the vocabulary, which can then be adapted downstream tasks through fine-tuning.
GOFA [19 presents novel generative foundation model for graph-language tasks by interleaving trainable GNN layers into frozen pre-trained L.M compressor, thereby marrying structural graph understanding with free-form text generation.
Graph Prompt Learning Methods
This category includes two prompt-based tuning approaches:
GraphPrompt 25
unifies pre-training and downstream tasks into common task template by em- ploying learnable prompt. This assists downstream tasks in locating the most relevant knowledge from the pre-trained model in a task-specific manner.
GPF [6] is universal prompt-based tuning method for pre-trained GNN models under any pre- training strategy. operates on the input graph's ' feature space and can theoretically achieve equivalent effect to any form of prompting function, eliminating the need to design specific prompting functions for each pre-training strategy.
Graph Self-supervised Pre-training Methods We employ GraphCL [46] as representative method. GraphCL maximizes agreement between differently augmented views of the same graph, enabling effective pretraining without labels. It introduces set of graph-specific augmentations, such as node dropping and edge perturbation, to generate diverse yet semantically consistent views for contrastive learning.
Graph Neural Networks This category includes two classic GNNs: Graph Attention Network (GAT 34 and Graph Isomorphism Network (GIN 42 GAT leverages attention mechanisms to assign different importances to neighboring nodes, while GIN is designed to maximally preserve graph structure by mimicking the power of the Weisfeiler-Lchman test for graph isomorphism.
28

---

E.2 Evaluation Protocols
We follow the same dataset splits as prior works [3 21 40 Our method adopts the same zero-shot setting as AnyGraph. We train two separate models Linkl and Link2 respectively, then evaluate the zero-shot performance of the Link/-trained model on Link2 and vice versa. Models are cross-applied between Link/ and Link2's domain-specific data. For example, we apply the model trained on Link to the Academic graphs from Link2, and conversely apply the Link2-trained model to Link/'s 's Academic data. The results are then aggregated through a weighted average based on the number of edges in each dataset. For the six graph foundation models, we either report the results from their original papers or evaluate them using their officially released code. GAT and GIN are trained from scratch under few-shot setting (with 10% training samples). GraphCL, GraphPrompt, and GPF undergo pretraining followed by tinettuning on the evaluation datasets.
Zero-shot Setting for Node Classification In our zero-shot node classification framework, we represent label classes distinct nodes and connect nodes with training labels to their corresponding class nodes. This method eliminates the need for learning separate parameters for each class, simplifying the zero-shot learning process. By integrating this method into baseline models, we enhance their ability to efficiently handle unseen node labels.
Evaluation Metries For link prediction, we follow previous work 40) and use Recall @ 20 and NDCG@2 20 as evaluation metrics. For node classification, we use accuracy and macro Fl as metrics, The results for each group of datasets are averaged based on the number of test samples.
E.3 Implementation Details and Hyperparameters
All experiments are conducted on single NVIDIA GeForce RTX 3090 GPU (24GB VRAM). Our methods are implemented using PyTorch, where SVD is computed via the built-in svd_ wrr_l_owrnk function for enhanced computational efficiency. When the number of training edges in a large dataset exceeds 500x the batch size, we randomly sample 500 batches from it in each epoch. models are trained for 100 epochs using the Adam optimizer with batch size of 4096 and learning rate of either 1 x 10 (LinkI) or 2 x 10 (Link2). The number of message passing layers L is set to 3. We use expert models, each consisting of an 8-layer MLP with 512 units per layer. The training step interval Tj for knowledge distillation is set to 100, with a loss weight ~=0.01. The training step interval T2 for the exponential moving average (Eq. (3) is selected from 100,20,50, 1,0,,,000000 100,200), with the decay factor 3 chosen from 099.9.0999.0099955..99999 (Eq. 23 is set to 0.2.
The recalibration strength parameter p
F Fine-grained Results
We present the fine-grained link prediction results for each dataset in Table We evaluate AnyGraph, the standard expert fusion method (Figure P9). and our enhanced expert fusion method under both full-shot (train on Linkl, test on Link/) and zero-shot (train on Linkl, test on Link2) settings. As shown, AnyGraph performs well under the full-shot setting but struggles in the more common real-world zero-shot scenario. This may be due its reliance on a single expert, which makes susceptible to overfitting the training set. In contrast, all expert merging methods leverage the knowledge of multiple experts, outperforming AnyGraph in the zero-shot setting. Moreover, our methods achieve the best overall performance, surpassing both AnyGraph and vanilla expert merging strategies, thereby demonstrating the superior generalization ability of our approaches.
The fine-grained node classification results compared to AnyGraph are shown in Table 7 Overall, KDEM and PPEM achieve higher macro-Fl scores on 4 out of datasets. Specifically, KDEM improves the average macro-F1 by 1.79% (from 43.24 to 45.03), while PPEM increases it by 0.92% (to 44.16), although both methods show slight decreases in overall accuracy relative to the baseline. Notably, on the Arxiv dataset, KDEM's macro-F1 rises substantially from 36.50 to 41.76, underscoring the significant advantage of multi-expert fusion strategies in addressing the class imbalance issue. Given the large variance in class sizes in the Arxiv test set (ranging from dozens to over ten thousand), the expert fusion strategy effectively mitigates the challenges posed by skewed class distributions by enhancing discrimination of minority classes. This is because the multi-expert
29

---

Table 6: Fine-grained link prediction results for each dataset, compared with AnyGraph and stan- dard/vanilla expert merging (Figure Vanilla EM refers to the vanilla expert merging method, which directly computes weighted average of the parameters from the top-k experts without any additional enhancements. All models are trained on Link/ and then tested on Link/ (full-shot) and Link2 (zero-shot), respectively. The results for each dataset averaged over 5 runs. The averages over dataset groups are computed by weighting each dataset according to the number of test edges.
AnyGraph
Vanilla EM
KDEM
PPEM
Groups
Datasets
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
Amazon-book
4.39
3.53
4.41
3.51
4.38
3.50
4.47
3,56
Amazon-t
13.74
9.12
12.25
8.18
12.58
8.32
13.04
8.55
Citation-2019
13.76
6.05
14.49
6.65
15.40
7.10
14.04
6.29
Citation-20C
35.01
14.74
30.09
12.28
31.66
12.76
34.59
14.71
Citeseer
79.34
64.89
79.24
62.06
79.31
65.85
79.47
Email-Fnron
64.43
44,50
64.93
44.83
64.15
44.70
64.52
44.86
P2P-Gnuiella
4.72
1.98
3.50
1.22
3.41
1.21
3.61
1.33
Link
PPA
25.64
14.49
24.22
13.56
25.65
14.43
23.96
13,46
full-shot
Products-home
70.63
42.09
69.87
40.82
69.39
41.31
69.29
41.84
Products-tech
52.42
34.09
34.14
50.49
33.33
52.32
34.52
PubMed
76.15
68.88
74.99
62.92
74.86
64.11
76.27
68.70
Soc-Epinionsl
24.86
15.19
24.57
14.96
24.71
15.26
24.68
15,26
Steam-t
10.63
6.63
8.79
5.44
9.79
6.13
9,21
5,70
Yelp2018
5.08
4.17
4.94
4.05
5.03
4.13
5.00
4.12
Yelp-t
11.01
6.92
7.17
4.46
8.06
5.09
8.13
5.07
Average
27.34
14.09
25.03
12.85
26.20
13.40
26.45
13.65
CS
79.85
63.54
79.14
62.06
79.59
63.78
79.18
63.88
Fitness
45.59
24.95
52.73
29.96
55.74
33.47
55.50
33.14
Goodreads
27.61
14.88
32.38
17.34
40.23
23.51
38.72
22.43
Photo
47.14
25.36
54.93
30.33
$7.79
34.07
56.53
33.45
Arxiv
36.91
16.96
34.75
15.77
35.63
16.59
34.88
15.85
Arxiv-t
38.67
18.58
35.19
16.47
39.71
20.37
39.21
20.32
Collab
3.66
1.76
3.80
1.81
3.46
1.68
3.60
1.74
Cora
82.19
65.87
81.83
67.74
81.73
66.16
81.13
65.27
Link2
DDI
9.38
19.92
8.45
16.32
19.67
zero-shot
Gowalla
11.18
8.06
9.66
6.92
12.57
9.26
10.48
7.56
ML.10m
6.51
8.56
24.11
28.00
23.40
27.18
19.52
23.66
ML
10.65
16.43
13.35
20.15
11.41
17.78
10.50
16.32
Proteins-O
16.93
23.23
17.30
23.73
17.57
23.99
18.81
25.82
Protcins-1
19.68
24.05
23.64
27.78
22.04
25.93
25.77
29.46
Proteins-2
14.02
20.07
17.80
24.95
17.84
25.27
13.66
20.22
Proteins-3
23.94
26.37
23.24
26.02
21.92
25.26
23.05
26.18
RoadNet-PP PA
89.47
50.42
89.15
52.62
88.83
52.93
88.91
52,83
Web-Stanfe Stanfotaooor
72.39
52.02
71.67
51.79
72.06
52.28
71.83
52.78
Average
45.431
27.01
48.34
29.72
51.69
32.60
50.77
31.93
Since AnyGraph does not report results for each individual dataset, we directly evaluate the released model and present the results. Therefore, the overall zero-shol average we report may differ slightly from that the original paper (45.43 vs. 46.42 for recall, and 27.01 vs. 27.21 NDCG).
mechanism integrates diverse perspectives from different experts, leveraging their diversity to reduce the risk of errors arising from a single expert's 's bias.
Additional node classification results compared with other GFMs are displayed in Table These models tap into the robust generalization abilities of large language models (LLMs), pretrained on massive text corpora, to elevate their performance in graph learning. For example, OpenGraph employs LLMs for data augmentation, while other methods leverage them to encode textual node attributes or predict node classes directly. The results reveal a striking trend: LLM-based GFMs deliver superior accuracy on small-scale datasets with fewer classes, such as Cora (7 classes), yet their performance in zero-shot and few-shot scenarios falters larger, class-rich datasets like Arxiv (40 classes). This drop-off likely stems from the heightened complexity of processing lengthy contexts and intricate graph structures, which taxes the models' reasoning and generalization strengths.
30

---

Table 7: Fine-grained node classification results, compared with AnyGraph. The results for each dataset are averaged over runs. The overall average (last row) is computed by weighting each dataset according to the number of test nodes.
AnyGraph
KDEM
PPEM
Datasets
Acc
Macro-F1
Acc
Macro-FI
Acc
Macro-FF
Arxiv
62.05
36.50
61.86
41.76
61.20
39.09
Cora
62.27
56.00
62.45
56.32
62.10
55.93
Products-home
66.36
40.31
65.44
41.37
67.30
48.03
PubMed
69.80
67.49
70.36
68.83
69.94
67.73
Products-tech 74.45
64.89
69.42
48.56
66.31
55.30
Average
64.38
43.24
63.56
45.03
62.65
44.16
Table 8: Fine-grand Fine-grained node classification results, compared with other existing GFMs (accuracy). The results of GFT are under few-shot setting. The Arxiv results of GraphGPT and UniGraph are under supervised learning and few-shot settings, respectively.
GraphGPT
OpenGraph
UniGraph
GFT
GOFA
KDEM
PPEM
Cora 18.13
75.04
69.53
67.36
70.81
62.45
62.10
PubMed
70.1
68.69
72,48
70.36
69.94
Arxiv
62.58
31.35
36.29
61.86
61.20
G Additional Experimental Results
G.1
Comparison with the Best Single Expert
We demonstrate in Table 9 that both of our enhanced expert merging strategies, KDEM and PPEM, outperform the best individual expert. This improvement arises because methods effectively integrate knowledge from multiple experts, thereby reducing the potential bias or overlitting of any single expert. Notably, although Figure T shows that the second-ranked expert (as assigned by the router) performs worse than the top-ranked expert in isolation, merging their parameters still leads to superior overall performance. This is because even lower-ranked experts capture complementary patterns or task-relevant nuances that the top expert might miss. By combining their knowledge, the model cover broader set of features relationships, improving robustness and generalization. Furthermore, the ensemble effect helps smooth out individual expert errors, allowing the fused model achieve better aggregate predictions than any expert alone.
Table 9: Performance comparison with the single top-performed expert. On the Link/ group, the top-ranked expert achieves the best performance, while on the Link2 group, the third-ranked expert performs best.
Linkl
Link2
Method
Recall
NDCG
Recall
NDCG
Tor-performed expert
23.94
12.68
48.22
30.56
KDEM
24.11
12.80
51.69
32.60
PPEM
24.33
12.93
50.77
31.93
G.2 Parameter Sensitivity Analysis
This section studies the impact of key hyperparameters in Figure IO, our observations are as follows:
models. Based the results presented
(1) Selecting multiple experts is essential. As shown in the first subfigure, integrating multiple experts significantly enhances model performance. This suggests that the fused experts can leverage complementary expertise, thereby improving generalization ability. However, using too many experts
31

---

can lead to suboptimal performance, power-ranked experts may not be well-suited for the given graph data, and merging their parameters can cause knowledge conflicts.
(2) While knowledge distillation applied at each training step (71 = 1) leads to satisfactory perfor- mance, it will incur significant computational achieve comparable even better performance, similar to real-world scenarios where teachers do not need teach students every day.
Periodic application of knowledge distillation
(3) The experimental results for KDEM are relatively insensitive to the weight of the knowledge distillation loss, 7, long is not too small. Empirically, a value of 0.01 works well.
(4) The period T22 and decay factor of the EMA need to be appropriately chosen. Smaller values for 72 and 33 lead to faster changes in expert parameters, which may cause the experts to become too similar, preventing the merged expert from effectively utilizing the diversity of knowledge. This confirms that experts should retain degree of diversity while maintaining some level of similarity, so the fused expert can benefit from the expertise of contributors.
Recall (bar)
NDCG (bar)
Rocall (linc)
NDCG (line)
Performance
40
Performance
40
Performance
40
30
20
20-
20
10
Topl
Top3
Top5
Top7
100
200
300
1.0
le-1
le-2
le-3
Number of Selected Experts (KDEM)
Knowledge Distillation Frequency (T₂)
KD Loss Coefficient (y)
Perforaance
Performance
Performance
15
10-
5
Top2
Top4
Top6
Top8
10
20
50
100
200
0.99
0.9995 0.9999 Decay Factor of EMA (B)
0.999
Number of Selected Experts (PPEM)
Period of Applying EMA (T2)
Figure 10: Parameter Sensitivity Analysis. The first row shows the results of the KDEM model trained on the Linkl group and evaluated on the Link2 group, while the second row shows the test results of the PPEM model trained on the Link2 group and evaluated on the Linkl group.
G.3 Efffec of the Number of Experts
This section investigates the impact the number of experts on model performance. As presented in Table TO, the results show that increasing the total number of experts from 2 to 8 yields consistent gains in both recall and NDCG, with the most pronounced improvements on Link2.
Table 10: Performance of KDEM with varying numbers of experts and activated experts.
Link
Link2
Experts
Activated Experts
Recall
NDCG
Recall
NDCG
2
22.85
12.02
45.68
27.64
22.23
11.72
48.02
29.31
2
21.53
11.13
48.64
29.43
4
3
22,66
11.97
48.02
29.31
8
24.11
12.80
49.15
30.63
8
3
23.35
12.33
51.69
32.60
32

---

H Further Discussion of the Expert Assignment Mechanism
To gain comprehensive understanding of the routing mechanism, record the expert assignment behavior of a trained PPEM model (trained on the Link2 group and evaluated on the Link/ group), and plot the competence scores of all experts across all datasets, as shown in Figure From this, we derive the following observations and insights.
Competence Scores Across All Datasets
542
B61
703
8:1
442
722
448
974
76
4:7
SAT
2310
BFA
781
840
816
1:00
SET
412
9:17 4X7
9811
(987
401
0014
700
572
971
0.9
3.R
Expert
m 541
240
245
245
478
245
441
400
7.90
457
340
6:1
810
805
902
857
1:00
475
912
977
992
4115
921, 980
509 94T SSY 180
952
914
5 -1-
5578-7800
.33
572
903
500
945
9:35
557
1102
522
520
672
538
532
576
911
953
463
590
(OSS
(963)
D82 R33 (912 931 (002 -21
090
701
6.5
272 269 1870
249
.278 ast
.983
987
213
1910
972
457
815
320
a29
0.00
923
952
924
924
992
992
1922
985 5922 .775 .972 .08 .063 (660
000
344
C5
Fitness
Goodreads
Photo
anxiv
anciv-
collab
cora
ddl
gowallo
ml10m
ml1m
proteins
proteins
proteins 2
proteins
readNet-PA
web Stanfordd
amazon- moazonnbookk book
amazon,
citation-200
citation-20C
citeseer
email-Enroo Enrnnn
pZp-Gnutella
ppa
prod home
prod tech pubmed
Soc-Epin1
steam_.
velp2018
yelp
amazoon
Dataset
Figure 11: Competence scores of 8 experts across all datasets. These expert scores obtained from
a PPEM model trained on the Link2 set and evaluated on the LinkI group. The left half of the figure corresponds the training datasets (from CS to Web-Stanford), while the right half represents the test datasets (from Amazon-book to Yelp-t).
H.1 Transferability the Routing Mechanism
We observe that the routing mechanism exhibits strong transferability from the pretraining data to previously unseen test data. For example, the training set (Link2), Experts 0, 3. and consistently specialize in e-commerce datasets such Photo, Goodreads, and Fitness. Similarly, in the test set (Linkl), the e-commerce dataset Products-Tedd Tech is assigned the highest routing scores to these same experts, indicating that the router effectively generalizes expert assignment patterns across domains.
H.2 Additional Insights
We also observe that Expert 2 and Expert 5 are rarely selected. To understand the root cause, we analyze the model's training dynamics and find that these two experts receive minimal updates, leading to a training imbalance. A similar phenomenon is observed in AnyGraph, which employs the same routing mechanism, suggesting that this issue is not unique to our model. We hypothesize that the limited diversity in the training set-comprising only 18 graphs-results in a small number of structural clusters, reducing the opportunity for all experts to be engaged.
Table 11: Results of changing the calibration parameter Increasing / enforces stricter expert balancing but leads to drop in performance.
0
0.1
0.2
0.3
0.4
Balanced
X
X
Recall 48.20
48.35
50.77
49.11
11
49.84
NDCG
30.07
29.83
31.93
30.08
30.47
To encourage more balanced expert utilization, we adjust the calibration range parameter p (Eq. 23 Although increasing this parameter leads to a more uniform expert assignment, as shown in Table results in a drop in overall model performance. This trade-off consistent with findings from prior work [36] 43 which suggest that enforcing balanced routing may result in suboptimal expert selection. Notably, in LLMSs based on the MoE, paradigm, load balancing is typically adopted not to improve performance, but to accelerate training and inference through parallelization strategies.
33

---

The load balancing strategies are not the main focus of this work and fall outside the scope of our study. Potential directions for future work include exploring moallarry-oss--reee load balancing methods 36 or incorporating expert pruning strategies (26]
H.3
Change the Routing Mechanism
The routing mechanism we adopt is non-parametric; to more comprehensively investigate the impact of routing strategies, we replace it with a trainable, parameterized router commonly used in LLMS (7]. Specifically, we average-pool -ool the representations E₁ (Eq() and feed them into a linear router to compute an affinity score for each expert, then select the top-k experts with the highest scores for fusion. In addition, we also evaluate the expert choice router [52] as an alternative routing strategy. As reported in T2ble using these trainable, parameterized routers yields inferior results-like because the heuristic non- parametric router more accurately assigns structurally similar graphs to the same experts, whereas the learnable router can introduce instability during training [52 29].
Table 12: Effect of changing the router to commonly used trainable ones.
Linkl
Link2
Method
Recall
NDCG
Recall
NDCG
PPEM w/ standard trainable router PPEM w/ expert choice routing PPEM
22.19
11.82
50.25
31.38
50.06
30.91
24.33
12.93
50.77
31.93
I Deeper Discussions of the Two Approaches
The two enhanced expert merging strategies, KDEM (Section 4.1 and PPEM (Section 4.2 though distinct in their mechanics- KDEM operating in the output space and PPEM in the parameter space-share common characteristics and offer insightful perspectives on optimizing the MoE layers.
Commonalities of KDEM and PPEM (1) Alignment with Expert Ensemble Performance. Both methodologies address the observed performance degradation that can occur with standard expert merging techniques and fundamentally strive to elevate the performance of expert merging to levels comparable with more computationally intensive yet effective expert ensembling. While KDEM leverages knowledge distillation to directly align the merged expert's outputs with those of the ensemble, PPEM exploits a theoretical insight to approximate ensemble behavior by bringing expert parameters closer via EMA. (2) Computational Efficiency. Efficiency remains a core tenet for both, as they aim to sidestep the substantial computational demands of direct expert ensembling.
KDEM introduces a modest additional computational step through periodic ensemble computation, whereas PPEM is engineered directly within the parameter space. (3) Regularization Effect. KDEM explicitly introduces regularization effect via its distillation mechanism, guiding the merged expert toward the "soft target" distribution of the ensemble. This encourages the merged expert to adopt broader, more generalizable representation. PPEM, by fostering parameter similarity among selected experts, may also confer an implicit regularization effect. This can prevent individual experts from diverging too dramatically, potentially improving model robustness and generalization by reducing interference during the merge.
deliver comparable outcomes with negligible overhead by operating
Insights Gained from the Two Methods (1) Flexibility in Design. The existence of two viable strategies highlights the flexibility in designing expert merging techniques. KDEM's output-space approach leverages well-established technique (knowledge distillation), making it intuitive and adaptable to various MoE setups. PPEM's 's parameter-space method, rooted in theoretical principles, offers computationally lean alternative that requires minimal overhead. This versatility suggests that future methods could explore hybrid approaches or context-specific adaptations based on dataset characteristics or resource constraints. (2) Trade-Off's and Practical Considerations. KDEM offers robust performance improvements but requires periodic ensemble computations, adding moderate training cost. It may be preferable when computational resources allow for this overhead and high
34

---

accuracy is paramount. PPEM achieves comparable results with negligible cost, making it ideal for cessurce-constrained scenarios. The choice between KDEM and PPEM could be tailored to specific applications, such as prioritizing efficiency in large-scale e-commerce systems or accuracy in academic graph analysis. (3) Potential for Hybridization. The success of two different approaches indicates that they may possess complementary advantages and potentially offers synergistic benefits combined. Future work could explore this synergy to fully release potential.
J Extended Related Work
Model Merging and Expert Merging Model merging integrates parameters from multiple net- works to harness their complementary strengths (see surveys (0)). Expert merging its counter- part in MoE architectures combines only specialized experts rather than entire models, yielding two key benefits: reduced compute costs and fully differentiable routing via soft fusion. For exam- ple, MEO [1] averages the parameters of the top-k experts to cut overhead, SMEAR 28 fuses all experts to avoid discrete router optimization (and outperforms gradient-estimation baselines), and Lory [51] generalizes SMEAR's principles to large autoregressive LMs. Nonetheless, existing merging strategies remain relatively rudimentary and do not fully realize the potential of combining multiple experts' knowledge.
Mixture-of-Experts (MoE) on Graphs With the widespread application of MoE in the NLP domain [2] they have recently been adopted in graph learning as well. For example, GraphDIVE [66] utilizes MoE to integrate multi-view graph representations, addressing class imbalance issues in graph classification. GMoE [35] treats multiple independent message- paasiing functions experts and uses gating mechanism to assign aggregation experts for each node. MoG 488 dynamically selects sparsification strategies for nodes with tailored sparsifier experts. GraphMETRO 39 employs an MoE architecture to address complex graph distribution shifts, enhancing the generalization ability of graph neural networks. Mowst [47 utilizes a weak MLP expert for features and a strong GNN expert for structure and designs novel MoE mechanism to integrate their expert knowledge. Node-MoE [00 employs an MoE framework to adaptively select appropriate filter experts for different nodes. GlaphAlign assigns different feature transformation experts to each node, aligning the features from different data sources. Despite these advancements, research on MoE in graphs still requires further exploration to fully maximize its potential.
K Broader Impact
Our work advances MoE-based GFMs by enabling efficient fusion of multiple experts, leading to improved accuracy and robustness across variety of graph-related tasks, such as academic network analysis, molecular interaction prediction, and recommendation systems. These improvements have the potential accelerate discovery in domains like drug development by more effectively modeling molecular interaction graphs and to enhance personalization in e-commerce platforms.
In addition, since MoE layers in large language models typically composed of MLPs, proposed expert fusion strategies are readily applicable to this broader class models. By reducing both training and inference overhead while maintaining performance competitive with top-k expert ensembles, our methods contribute to more sustainable machine learning practices by lowering computational costs and associated carbon emissions. Furthermore, this efficiency paves the way for more practical deployment of LLMMs on resource-constrainee edge devices.
L Limitations
To ensure a fair comparison with AnyGraph and highlight the advantages of our proposed expert merging strategy, we follow its setup by excluding graph-level tasks (e.g., graph classification). We leave a comprehensive investigation of these tasks to future work. Since our approach does not integrate LLMs, it cannot generate free-text responses to broad graph queries as GOFA [19 does; nonetheless, our model demonstrates strong graph understanding and can serve as an effective graph encoder, which, when coupled with an LLM-based decoder and instruction fine-tuning could achieve comparable capabilities. We leave these extensions to future work.
35


Reviews:
[{"review_id": "1eunV4L1fD", "reviewer": "Reviewer_aTL7", "summary": "This paper addresses a key limitation of the current mixture-of-experts (MoE)-based graph foundation model, which relies solely on the top-ranked expert, by showing that lower-ranked experts (e.g., second- and third-ranked) can often perform better. To more effectively harness the collective knowledge of multiple experts, this paper proposes two novel expert merging techniques: (1) KDEM, which distills knowledge from an ensemble of top-k experts into a parameter-merged expert, and (2) PPEM, which encourages parameter alignment among the selected experts. The resulting model achieves strong performance on several downstream tasks in a near-consistent manner and improves training and inference efficiency over naive expert ensembling.", "questions": "Please refer to the main review for detailed comments and suggestions.\n\nOne additional question: The paper applies KDEM and PPEM only individually. Was combining the two techniques considered? If so, why was this not pursued? If it was not considered, do the authors believe the two techniques could be complementary if used together?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n\nS1. The paper is well-motivated and easy to follow. It builds on an interesting observation: in mixture-of-experts (MoE) models, the second- and third-ranked experts often outperform the top-ranked one. This highlights (1) a limitation of the recent MoE-based graph foundation model, AnyGraph, which uses only the top-ranked (i.e., most competent) expert, and (2) the need to incorporate the knowledge embedded in multiple experts.\n\nS2. The paper introduces two interesting and novel expert merging techniques: Knowledge Distillation Enhanced Expert Merging (KDEM) and Parameter Proximity Enhanced Expert Merging (PPEM). KDEM leverages knowledge distillation, using an ensemble of the top-k experts as the teacher and the parameter-merged expert as the student. PPEM encourages the parameters of the top-k experts selected by the router to move gradually in the same direction.\n\nS3. The proposed method near-consistently outperforms several existing graph foundation models and graph learning methods on downstream tasks. Ablation studies demonstrate the effectiveness of the expert merging techniques, as well as internal components such as the use of EMA. Compared to naive expert ensembling, the method also achieves improved training and inference efficiency, as supported by experimental results.\n\n\n[Weaknesses]\n\nW1. The paper currently lacks a comparison between the proposed method and existing expert merging strategies. The experiments focus on comparing the proposed method against graph foundation models and graph learning methods for graph tasks such as link prediction and node classification. However, the core technical contributions of this work lie in the development of expert merging techniques, which are not specific to graph data or graph learning methods. Given the existence of prior expert merging approaches such as SMEAR [28] and Lory [51], a comparison with those methods is necessary to establish the proposed technique as a new state of the art.\n\nW2. The number of total and selected experts (denoted as K and k, respectively) has a significant impact on performance, as demonstrated in the experiments. However, these values must be manually specified prior to training, which may be non-trivial in the context of graph foundation models spanning diverse datasets and tasks. It would strengthen the paper to provide guidance or at least discussion on how to choose these values effectively and efficiently.\n\nW3. The generalizability of the proposed method to new domains and tasks appears limited or remains unverified. A core requirement for a graph foundation model (GFM) is the ability to generalize to unseen domains and tasks. While the proposed GFM builds on AnyGraph and improves it by leveraging multiple experts, it remains unclear whether the selected experts can perform well under significant domain or task shifts. The current experiments use two dataset groups (Link 1 and Link 2), but these are relatively homogeneous in both domain and task. It would strengthen the paper to include evaluations on more diverse domains and tasks to better assess the model’s robustness in generalization scenarios, which are central to the purpose of GFMs.", "was_revised": true, "final_justification": "The author response addressed my concerns about the comparison with existing expert merging strategies and the choice of total and selected number of experts. In light of these improvements, I have increased my rating.", "rebuttal": "We thank the reviewer for this detailed and constructive feedback. We are encouraged that the reviewer found the paper well-motivated and easy to follow (S1), the proposed techniques (KDEM and PPEM) to be interesting and novel (S2), and the empirical results and efficiency gains to be strong (S3). We appreciate the critical suggestions for improvement, which we address below.\n> *W1: The paper currently lacks a comparison between the proposed method and existing expert merging strategies ... Given the existence of prior expert merging approaches such as SMEAR [28] and Lory [51], a comparison with those methods is necessary to establish the proposed technique as a new state of the art.*\n\nWe thank the reviewer for raising this important point and for suggesting these relevant baselines. We would like to clarify how our work is positioned relative to these methods.\n\nThe approaches of SMEAR and Lory are indeed foundational to recent work in expert merging. Both propose a soft merging of experts where the parameters of all experts are fused via a weighted average based on routing weights. Their primary motivation is to create a fully differentiable training process that avoids the challenges of discrete router optimization. We note that **the expert merging strategy of Lory is the same as SMEAR**, with the difference of application scenario (encoder-decoder v.s. decoder-only architecture). \n\nOur \"vanilla expert merging\" baseline, shown in Figure 8 and Table 5, is inspired by this principle but adapted to the top-$k$ routing framework. Specifically, it performs a parameter merge of the top-$k$ experts selected by the router's gating mechanism. Therefore, the expert merging strategy used by SMEAR and Lory (i.e., $k=K$) is a special case of our adapted vanilla expert merging approach.\n\nTo provide a more direct and comprehensive comparison as requested, we report the top-8 (in our case, $K$=8) expert merging results following the SMEAR/Lory methodology on the whole Link2 group datasets:\n\n| Method             | Recall | NDCG  |\n|-------------------|--------|-------|\n| SMEAR/Lory-style Merge | 48.52  | 29.46 |\n| KDEM              | 51.69  | 32.60 |\n| PPEM              | 50.77  | 31.93 |\n\nAs the results clearly show, our proposed KDEM and PPEM methods significantly outperform the SMEAR/Lory-style baseline. We hypothesize this is because fusing all experts—including those with very low competence for a given graph—can introduce noise and parameter interference from irrelevant specialists. Our approach is more targeted: it first identifies the most relevant experts via top-$k$ routing and then applies our enhanced merging strategies to effectively combine their specialized knowledge. This focused enhancement proves to be more effective than a simple fusion of all available experts.\n\n> *W2: The number of total and selected experts (denoted as $K$ and $k$, respectively) has a significant impact on performance ... It would strengthen the paper to provide guidance or at least discussion on how to choose these values effectively and efficiently.*\n\nWe thank the reviewer for this insightful suggestion. Providing clear guidance on selecting these key hyperparameters is crucial for practitioners, and we will add a dedicated section in the appendix to address this based on our findings.\n\nOur recommendations are as follows:\n\n1. Guidance on Choosing the Total Number of Experts ($K$)\n\nThe choice of $K$ involves a trade-off between model capacity and efficiency. A larger $K$ allows the model to learn a more diverse set of specializations, which is beneficial when training on a wider range of domains. However, it also increases the model's parameter count and can exacerbate the expert imbalance issue. A smaller $K$ is more parameter-efficient but may not fully capture the data's heterogeneity.\n\n**Recommendation:** Based on our experiments across 38 diverse datasets (Table 10), we find that $K=8$ provides a good balance, effectively covering multiple domains without being excessively large.\n\n2. Guidance on Choosing the Number of Selected Experts ($k$)\n\nWe propose two strategies for selecting $k$, catering to different experimental constraints:\n\n**Empirical Guideline (For Optimal Performance):** We recommend a preliminary analysis based on the performance of individual experts, similar to our study in Figure 1. After an initial training run, one can evaluate the performance of each expert ranked by competence score. The value of $k$ should be chosen to include the top-performing experts while excluding those that cause a sharp performance drop. For instance, in Figure 1, performance drops after the 5th expert, suggesting an optimal $k\\leq5$. This avoids merging poorly performing experts whose parameters could introduce noise and interference.\n\n**Efficient Heuristic (As a Default):** In scenarios where extensive tuning is not feasible, our results in Figure 10 (the first column) show that using a small value like $k=2$ or $k=3$ serves as a robust and effective default. This consistently provides a significant performance lift over using a single expert ($k=1$) without incurring the high computational cost or risk of noise associated with larger values of $k$.\n\n> *W3: The generalizability of the proposed method to new domains and tasks appears limited or remains unverified ... it remains unclear whether the selected experts can perform well under significant domain or task shifts. The current experiments use two dataset groups (Link 1 and Link 2), but these are relatively homogeneous in both domain and task. It would strengthen the paper to include evaluations on more diverse domains and tasks ...*\n\nWe thank the reviewer for this crucial comment. We completely agree that rigorous evaluation of generalizability across diverse domains and tasks is paramount for advancing GFMs.\n\n**1. On the Diversity of Domains in Our Current Evaluation:**\n\nWe would like to respectfully highlight the significant domain diversity already present in our experimental setup. Our evaluation, which involves training on one group of datasets and zero-shot testing on another, was designed to explicitly test this cross-domain generalization. For example:\n\n- Domains unique to Link1 group: Email Networks (Email-Enron), Trust Networks (Soc-Epinions1), and Internet peer-to-peer network (P2P-Gnutella06).\n- Domains unique to Link2 group: Web Graphs (Web-Stanford) and Road Networks (RoadNet-PA).\n\nOur model's strong performance when transferred between these groups demonstrates a robust ability to generalize to entirely unseen graph domains. The inclusion of common domains like \"Academic\" and \"E-commerce\" in both groups is a deliberate choice since they are widespread, analogous to how LLMs are pre-trained on broad web data that contains domains also seen in downstream evaluations (e.g., math, code). This ensures the model learns both domain-specific and widely generalizable patterns.\n\n**2. On the Diversity of Tasks:**\n\nRegarding task diversity, our work focuses on node-level (classification) and edge-level (link prediction) tasks. This aligns with the evaluation protocol of our primary baseline, AnyGraph, and much of the current GFM literature. We agree that a truly universal GFM should also excel at graph-level tasks (e.g., graph classification, regression). We consider the extension and evaluation of our methods on these tasks an important direction for future work, which we have noted in our limitations section (Appendix L).\n\n**3. On the State of GFM Benchmarking:**\n\nWe also acknowledge the reviewer's sentiment about the need for even broader evaluations. A significant challenge in the current GFM landscape is the lack of a standardized, large-scale benchmark that spans a vast number of disparate domains and tasks. In fact, our collection of 38 datasets is one of the more comprehensive evaluation suites used in GFM research to date. We see our work not only as a methodological contribution but also as a step towards highlighting the need for more extensive, community-driven GFM benchmarks, and we believe our findings provide a strong baseline for such future evaluations.\n\n> *Question: The paper applies KDEM and PPEM only individually. Was combining the two techniques considered? If so, why was this not pursued? If it was not considered, do the authors believe the two techniques could be complementary if used together?*\n\nWe thank the reviewer for this forward-looking idea. The potential for combining KDEM and PPEM is indeed an exciting prospect.\n\nIn the current work, we presented KDEM and PPEM individually to clearly isolate and analyze their distinct mechanisms. **As discussed in Appendix I**, the two methods achieve the same goal—elevating the performance of a merged expert to that of an expert ensemble—through different means: KDEM operates in the output space while PPEM operates in the parameter space. This indicates that they may possess complementary advantages and potentially offers synergistic benefits if combined. A potential hybrid approach could work as follows:\n\n- **PPEM as a Regularizer:** The PPEM method, by applying an EMA to the selected experts' parameters, can be seen as a strong regularizer that encourages the \"student\" model (the merged expert) to be inherently more similar to the \"teacher\" model (the expert ensemble) even before distillation occurs.\n\n- **KDEM as a Fine-Tuning Mechanism:** With PPEM creating a more stable and better-aligned student, the knowledge distillation from KDEM could then act as a more focused fine-tuning step. The distillation loss might be smaller and converge faster, as PPEM has already done much of the work to close the gap between the student and teacher.\n\nGiven the scope of this work, we chose to present them as two distinct and effective strategies. However, we consider the investigation of a hybrid KDEM-PPEM model to be a promising direction for future research."}, {"review_id": "OVaPpw0Bs1", "reviewer": "Reviewer_bgSg", "summary": "Authors propose two enhanced expert merging strategies for Graph Foundation Models (GFMs) to improve performance while maintaining computational efficiency. The authors aim to address the challenges of the two extreme versions -- (1) a single top-ranked expert underutilizing knowledge from multiple experts and (2) full ensemble strategies with huge computational overhead.\n\nIn particular, the authors perform an in-depth analysis of an MoE-based GFM and discover that the second- and third-ranked experts often outperform the top-ranked expert selected by the router, revealing valuable but untapped expertise. Motivated by this insight, authors propose two methods: (i) Knowledge Distillation Enhanced Expert Merging (KDEM), which aligns a fused expert’s behavior with that of an expert ensemble via periodic distillation, and (ii) Parameter Proximity Enhanced Expert Merging (PPEM), which reduces parameter distances among top-k experts so that merging approximates ensemble outputs while preserving diversity. These approaches are designed to harness the strengths of multiple experts efficiently.\n\nAuthors perform extensive experiments on 38 graph datasets, evaluating their methods under zero-shot and few-shot settings against baseline GFMs, classical GNNs, and standard expert merging. Results show that KDEM and PPEM consistently improve performance over single-expert routing and standard merging while maintaining low computational costs. They further provide ablation studies, theoretical analysis, and routing visualizations to validate the effectiveness and efficiency of the proposed methods.", "questions": "- What exactly does  `-EMA` in Table 3 indicate?\n- It would be great if Figure 4 can be readable along with performance as well (i.e. clear inteprataion over time / top K / performance)\n- It would be great to see how different experts are trained between KDEM and PPEM, given that PPEM adds the restriction of not diverging too much.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* Strengths\n- Authors carefully analyze the behavior of existing MoE-based GFMs and identify the important challenge that second- and third-ranked experts often outperform the top expert chosen by routing. This empirical observation is clearly presented and motivates the necessity for improved multi-expert utilization, which grounds the rest of the work in a real and measurable problem.\n- Authors provide a clean and rigorous rationale of the proposed solutions, combining empirical findings and theoretical insights—for example, proving that parameter proximity enables parameter merging to approximate expert ensembles. This thoroughness makes the proposed approaches well justified and not merely heuristic.\n- The introduction of knowledge distillation into expert merging (KDEM) is novel to tackle the challenges in the ensemble learning of GFMs. The method intuitively bridges the gap between the efficiency of expert merging and the performance of expert ensembles, and the authors provide practical implementation details that make it feasible.\n- The authors perform experiments on 38 diverse graph datasets across domains, under both zero-shot and few-shot settings. This broad scope of evaluation, along with detailed ablation studies, strongly supports the validity and robustness of the proposed methods.\n- The paper is well-written and organized, making a technically complex topic accessible. Figures, tables, and theoretical explanations are presented in a way that is easy to follow, which enhances the reader’s understanding.\n\n\nWeakness\n- The core contributions of the paper lie in improving expert merging strategies within an existing MoE-based GFM framework rather than proposing a fundamentally new architecture or a completely novel GFM paradigm. While the enhancements are well-justified and practically useful, some readers might view the work as an incremental improvement rather than a groundbreaking innovation.\n- Despite recognizing the limitations and risks of parameter averaging in expert merging (e.g., parameter interference due to expert specialization), the authors focus solely on improving parameter averaging rather than exploring fundamentally different fusion mechanisms. \n- The PPEM method intentionally brings expert parameters closer together to enable more effective merging, while the main purpose of MOE is to cover multi-modal diversities. While proper regularization might work well in practice to balance between these two conflicting goals, the authors need to address this concern.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for this detailed and positive assessment of our work. We are delighted that the reviewer found our paper to be well-motivated (S1), technically solid and novel (S2、S3), comprehensively and robustly evaluated (S4), and clearly written (S5). We appreciate the insightful questions and address each point below.\n\n> *W1: The core contributions of the paper lie in improving expert merging strategies within an existing MoE-based GFM framework rather than proposing a fundamentally new architecture or a completely novel GFM paradigm. While the enhancements are well-justified and practically useful, some readers might view the work as an incremental improvement rather than a groundbreaking innovation.*\n\nWe thank the reviewer for this thoughtful perspective on the positioning of our work. We agree that our work focuses on a specific, critical component of MoE-based models rather than proposing a new GFM architecture from scratch. We respectfully clarify that this targeted focus is a significant strength and a crucial contribution to the field.\n \nMoE is becoming a cornerstone for large-scale models across many domains, such as LLMs (DeepSeek, Qwen3), vision models (V-MoE), and multi-modal models (Gemini 2.5 Pro). While models like AnyGraph have pioneered the use of MoE in the graph space, the fundamental question of how to efficiently utilize the collective knowledge of multiple experts remains a key bottleneck. Our paper tackles this precise challenge head-on. We are, to our knowledge, the first to move beyond simple averaging and propose enhanced expert merging strategies specifically for this context. Our methods provide principled, effective, and efficient solutions to this critical bottleneck.\n\nTherefore, **while our work builds upon an existing framework, it solves a fundamental problem that will be faced by any future MoE-based GFM.** By providing a blueprint for efficient multi-expert utilization, our research offers a foundational piece of the puzzle that will enable the development of more powerful and scalable graph foundation models. We believe this contribution is both practical and of significant long-term value to the community. \n\n\n> *W2: Despite recognizing the limitations and risks of parameter averaging in expert merging (e.g., parameter interference due to expert specialization), the authors focus solely on improving parameter averaging rather than exploring fundamentally different fusion mechanisms.*\n\nWe thank the reviewer for raising an interesting point about exploring different fusion mechanisms. We chose to focus on enhancing parameter averaging because it is the cornerstone of efficient expert merging in MoE models. While other fusion mechanisms exist [1], they often come with increased computational complexity, defeating the primary purpose of merging. \n\nOur contribution lies in making the most computationally efficient paradigm, simple parameter averaging, perform on par with expensive expert ensembling. By demonstrating that this simple mechanism can be elevated to match the performance of complex ensembling through our proposed methods, we provide a practical and powerful toolkit for a common scenario.\n\n[1]. Prateek Yadav et al. TIES-Merging: Resolving Interference When Merging Models. *NeurIPS 2023*.\n\n> *W3: The PPEM method intentionally brings expert parameters closer together to enable more effective merging, while the main purpose of MOE is to cover multi-modal diversities. While proper regularization might work well in practice to balance between these two conflicting goals, the authors need to address this concern.*\n\nWe thank the reviewer for raising this critical point. We were fully aware of this potential conflict, which is why we designed PPEM to strike a careful balance between expert similarity and diversity. As detailed in Appendix G.3, our approach has two key features:\n\n1. Selective Similarity: The parameter-proximity constraint is applied only to the top-$k$ experts selected for a given input. The parameters of the remaining experts are left unchanged, preserving the overall diversity of the expert pool.\n\n2. Verified Diversity: Our t-SNE visualization in Figure 9 empirically confirms that even after training with PPEM, the full set of 8 experts remains well-separated and specialized in the embedding space, demonstrating that global diversity is maintained.\n\nTherefore, PPEM successfully encourages local similarity for effective merging while preserving the global diversity essential for the MoE architecture.\n\n> *Q1: What exactly does `-EMA` in Table 3 indicate?*\n\nIn Table 3, PPEM-EMA refers to an ablated version of our PPEM model where the core Exponential Moving Average (EMA) update mechanism (described in Eq. 13) has been removed. This variant is equivalent to the standard expert merging baseline, where the parameters of the top-$k$ selected experts are simply averaged without any of our proposed enhancements. The performance drop from PPEM to PPEM-EMA thus directly demonstrates the effectiveness of our parameter proximity strategy. We are sorry for any confusion and will clarify this in the final version.\n\n> *Q2: It would be great if Figure 4 can be readable along with performance as well (i.e. clear inteprataion over time / top K / performance)*\n\nThis is a great suggestion. For the sake of clarity, we separated the analysis of computational efficiency (Figure 4) from the parameter sensitivity analysis that shows performance vs. $k$ (Figure 10). To make the demonstration clearer, we will add a new plot to the appendix in the final version that directly visualizes performance against training/inference time for different values of $k$.\n\n> *Q3: It would be great to see how different experts are trained between KDEM and PPEM, given that PPEM adds the restriction of not diverging too much.*\n\nThank you for this insightful feedback. We will generate a t-SNE visualization of node embeddings from all 8 experts, similar to the existing Figure 9, but this time for a model trained with KDEM. We expect to see that the expert clusters for PPEM are tighter and show less separation than for KDEM."}, {"review_id": "Z6V1M2MRSf", "reviewer": "Reviewer_WjEh", "summary": "Based on the current Graph MoE models like Anygraph, this works provide detailed discussitions on Expert Selection and Merging. After analyzation on this, this work propose two improved expert\nmerging strategies, KDEM and PPEM. Further Experiments show that they can achieve better performance than naive experts merging mechanism.", "questions": "1. L183 : \"Since the performance of the top-k experts’ ensemble is excellent, we treat it as a teacher model to teach the parameter-fused expert.\" How is the \"top-k\" determined in a MoE model?\n\n2. KD-based methods for merging graph-based models have been widely discussed, such as in ParetoGNN[1](during training) or WAS[2](during fine-tuning). Could you please provide more details about your design? Why your approach is unique or different?\n\n---\n\n[1]ICLR'23 https://arxiv.org/abs/2210.02016\n\n[2]ICLR'24 https://arxiv.org/abs/2403.01400", "limitations": "N.A.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The authors address an important and previously under-explored problem — the selection and merging of experts in Graph MoE models \n\n2. Preliminary analysis is both reasonable and insightful.\nThe method proposed by the authors is effective and powerful.\n\nWeaknesses:\n\nWill these processes (KDEM or PPEM) be harmful to the training time? It seems quite difficult to converge, especially when KD is introduced in the middle of the training process.", "was_revised": true, "final_justification": "Great work. Good Paper. So I will keep my score.", "rebuttal": "We are grateful to the reviewer for this positive evaluation. We are encouraged that the reviewer found the problem we addressed to be important (S1), our analysis insightful (S2), and our proposed methods effective (S2). We address the specific weakness and questions below.\n\n> *Weakness: Will these processes (KDEM or PPEM) be harmful to the training time? It seems quite difficult to converge, especially when KD is introduced in the middle of the training process.*\n\nWe understand the reviewer's concern regarding training time and convergence. We are pleased to clarify that both KDEM and PPEM were designed for efficiency and converge effectively.\n\n- **On Training Time:** Our computational efficiency analysis in Section 5.4 shows that the additional overhead is minimal. As stated, \"KDEM incurs a marginal increase in training time over standard expert merging (averaging 4.8%), whereas PPEM exhibits almost no increase\". This is visually demonstrated in Figure 4, where the training times for our methods are very close to the Top-1 and standard merging baselines, and significantly faster than expert ensembling.\n\n- **On Convergence:** The strong and consistent performance improvements reported in our main results (Tables 1 and 2) serve as direct evidence that our methods converge successfully to a high-performing state. Furthermore, applying knowledge distillation periodically (rather than at every step) in KDEM actually promotes stable convergence. This is because the student model is not forced to chase a constantly changing teacher target, allowing for more consistent gradient updates, a principle we highlight in our discussion of KDEM's design.\n\n> *Q1: L183 : \"Since the performance of the top-$k$ experts’ ensemble is excellent, we treat it as a teacher model to teach the parameter-fused expert.\" How is the \"top-$k$\" determined in a MoE model?*\n\nThank you for the question. In our framework, $k$—the number of experts to select for merging—is a hyperparameter. The process is as follows:\n\nFor a given input graph, the routing mechanism calculates a competence score for each of the $K$ experts in the pool. We then simply select the $k$ experts that have the highest competence scores. \n\nWe provide an empirical analysis of this hyperparameter in Figure 10, which shows that a small value of $k$ (e.g., 2 or 3) is a robust choice that consistently yields strong performance. This provides a practical guideline for its selection.\n\n> *Q2: KD-based methods for merging graph-based models have been widely discussed, such as in ParetoGNN[1](during training) or WAS[2] (during fine-tuning). Could you please provide more details about your design? Why your approach is unique or different?*\n\nThank you for your insightful feedback and for highlighting these important related works. We appreciate the opportunity to provide more details about our design and clarify the unique contributions of our proposed method, KDEM.\n\nWhile all three works involve integrating knowledge from multiple sources, they operate on different principles and architectural levels.\n\nWe respectfully clarify that ParetoGNN's methodology is distinct from knowledge distillation. ParetoGNN's core contribution is a multi-task learning framework that reconciles conflicting signals from different self-supervised pretext tasks. It operates in the **gradient space**, using a Multiple Gradient Descent Algorithm (MGDA) to find a Pareto-optimal descent direction that balances the objectives of all tasks simultaneously. In contrast, our KDEM operates in the **output space**. The \"teacher\" is the soft prediction distribution from an expert ensemble, and the \"student\" (the merged expert) is trained to mimic this distribution. Therefore, ParetoGNN focuses on balancing training objectives at the gradient level, whereas our work focuses on compressing the predictive knowledge of an expert ensemble into a single, efficient expert.\n\nWhile both our work and WAS employ knowledge distillation (KD) during a fine-tuning or unified training phase, we address fundamentally different problems with distinct goals and methodologies. Our work focuses on **efficient expert merging within a single Mixture-of-Experts (MoE) model**, whereas WAS focuses on **integrating and selecting from a pool of separate, independently pre-trained models.**\n\nOur contributions are unique in two key ways:\n\n**1. Different Problem & Architecture:** We optimize a single MoE-based GFM. Our goal is to merge the parameters of the top-$k$ experts to efficiently approximate the performance of a computationally expensive expert ensemble. WAS integrates multiple, separate pre-trained models by combining their output distributions during a fine-tuning stage.\n\n**2. Novel KD Application (in KDEM):** The teacher-student dynamic is fundamentally different.\n\n- In our KDEM, the \"teacher\" is the on-the-fly ensemble of top-$k$ experts, and the \"student\" is the parameter-merged expert. KD's role is to align the behavior of the efficient merged expert with its powerful ensemble counterpart within a unified training loop.\n\n- In WAS, the \"teacher\" is a weighted combination of outputs from selected pre-trained models, and the \"student\" is a separate model being fine-tuned.\n\nIn summary, our work introduces new techniques for efficient expert utilization inside an MoE model, while WAS tackles the integration of external models. Our focus on MoE mechanics is a distinct and complementary contribution."}, {"review_id": "VzrQwFpRhH", "reviewer": "Reviewer_Etth", "summary": "This paper investigates mixture-of-experts (MoE) mechanisms in graph foundation models (GFMs) and addresses the challenge that while GFMs use MoE to handle graph heterogeneity, current approaches only utilize the single top-ranked expert, potentially leaving valuable knowledge from other high-performing experts untapped. Through experimental analysis of AnyGraph, the authors discover that experts ranked second and third by the router often outperform the top-ranked expert, motivating them to explore multi-expert knowledge integration. They propose two enhanced expert merging strategies: (1) Knowledge Distillation Enhanced Expert Merging (KDEM), which uses knowledge distillation to align the behavior of parameter-fused experts with expert ensembles, and (2) Parameter Proximity Enhanced Expert Merging (PPEM), which leverages theoretical analysis showing that when MLP experts have similar parameters, their merged output approximates the ensemble output. The proposed approach bridges the efficiency of expert merging with the performance benefits of expert ensembling through these theoretically-grounded and practically-efficient approaches. Extensive experiments demonstrate that KDEM and PPEM achieve significant performance improvements over the AnyGraph baseline while maintaining computational efficiency close to single-expert inference, which effectively solve the trade-off between performance and efficiency in MoE-based GFMs.", "questions": "•\tHave the authors investigated whether the non-parametric routing mechanism is actually optimal for scenarios where multiple experts are merged?\n•\tIn Appendix H.3, the authors briefly show that trainable routers perform worse, could the authors explore other routing strategies specifically designed for multi-expert selection?\n•\tBeyond adjusting the calibration parameter $rho$, what principled approaches could ensure more balanced expert utilization while maintaining performance?\n•\tHow does expert imbalance affect the theoretical guarantees of Theorem 1, especially when some experts receive minimal training?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n•\tThe motivation is sound, which is based on straightforward empirical observation. This points out the deficiency of current approaches and poses a natural goal to solve the explicitly existing problem.\n•\tThe paper presents two complementary solutions addressing the same problem from different angles, where KDEM operates in output space using knowledge distillation, while PPEM works in parameter space using theoretical proximity principles.\n•\tThe theoretical validation for PPEM is sound and solid. Theorem 1 provides a solid mathematical basis for PPEM, proving that when MLP expert parameters converge pairwise, the merged MLP output approximates the convex combination of individual expert outputs. This insight directly translates into PPEM that maintains expert diversity while reducing parameter interference.\n•\tThis work demonstrates substantial empirical contributions. Evaluation on 38 datasets spanning diverse domains with rigorous zero-shot evaluation protocols ensures robustness and generalizability. The proposed approach achieves significant improvements (5.27% and 4.35% recall improvement on Link2) while maintaining computational efficiency nearly identical to single-expert inference. As for efficiency, both proposed expert merging strategies introduce minimal computational overhead - KDEM adds only 4.8% training time, while PPEM has negligible increase, making them practically deployable.\nWeaknesses:\n•\tThe paper inherits AnyGraph's routing mechanism without questioning its optimality for multi-expert scenarios.\n•\tThe expert imbalance problem (some experts rarely selected) is acknowledged but not systematically addressed.\n•\tInsufficient analysis of when and why the methods work well across different graph types. A more principled analysis of what graph characteristics, e.g., size, feature dimensionality, domain, make certain datasets more amenable to the multi-expert approach would be good.", "was_revised": true, "final_justification": "I have read the rebuttal and also looked at the other reviews. I don't need further clarifications at this point. I am satisfied with the rating I have already provided but increased the scores slightly.", "rebuttal": "We sincerely thank the reviewer for this thorough evaluation and positive feedback. We appreciate your recognition of our paper's strengths, particularly the sound motivation, complementary solution approaches, solid theoretical foundation, and substantial empirical validation. Below we address the concerns and questions in detail.\n> *W1: The paper inherits AnyGraph's routing mechanism without questioning its optimality for multi-expert scenarios.*\n>\n> *Q1: Have the authors investigated whether the non-parametric routing mechanism is actually optimal for scenarios where multiple experts are merged? In Appendix H.3, the authors briefly show that trainable routers perform worse, could the authors explore other routing strategies specifically designed for multi-expert selection?*\n\nWe thank the reviewer for this insightful concern. Here, we handle W1 and Q1 together, and our responses are as follows:\n- **Rationale for Our Initial Choice:** Our paper's primary contribution is the development of novel expert merging strategies (KDEM and PPEM). To isolate and clearly demonstrate their effectiveness, we deliberately inherited the routing mechanism from our main baseline, AnyGraph. We believe this non-parametric router provides a strong and generalizable foundation; its competence score calculation is independent for each expert, allowing the selection to be naturally extended from a top-1 to a top-$k$ setting. This provides a fair and controlled basis for evaluating our core contributions.\n- **Investigation of Alternative Routers:** However, we agree with the reviewer that investigating alternatives is crucial for a complete analysis. Prompted by this valuable feedback, we have conducted new experiments to supplement our initial investigation of a trainable router (Appendix H.3). We selected another well-established strategy, Expert Choice Routing [1], adapted it for our framework, and evaluated its performance with our PPEM method. The results, averaged on the Link2 group, are as follows:\n\n| Method                        | Recall | NDCG  |\n|-------------------------------|--------|-------|\n| PPEM w/ Expert Choice Routing | 50.06  | 30.91 |\n| PPEM                          | 50.77  | 31.93 |\n\n- **Discussion of Results:** The results show that Expert Choice Routing, while a powerful technique, underperforms compared to the original non-parametric router in our setting. This finding reinforces our hypothesis (lines 870–872) that the heuristic non-parametric router may be particularly effective at assigning structurally similar graphs to the same experts, making it a surprisingly strong baseline for graph-based MoE.\nThese new results, combined with our initial findings in Appendix H.3 that a standard trainable router also underperforms, further validate our choice of routing mechanism as a solid foundation for our study. We are grateful for the reviewer's suggestion, as it has allowed us to strengthen the paper by demonstrating the robustness of our findings across different routing paradigms. We will add this new analysis to the appendix.\n\n\n[1]. Yanqi Zhou et al. Mixture-of-Experts with Expert Choice Routing. *NeurIPS 2022.*\n\n> *W2: The expert imbalance problem (some experts rarely selected) is acknowledged but not systematically addressed.*\n>\n> *Q2: Beyond adjusting the calibration parameter $\\rho$, what principled approaches could ensure more balanced expert utilization while maintaining performance?*\n\nWe thank the reviewer for this valuable question. We respond to W2 and Q2 together because they are fully correlated. Addressing the expert imbalance problem without compromising performance is indeed a critical and challenging research direction in MoE models, especially in the LLM community, and we appreciate the opportunity to elaborate on this.\n\nAs the reviewer noted, our investigation in Appendix H.2 showed that naively enforcing balance by adjusting the calibration parameter $\\rho$ can lead to suboptimal expert assignments and degrade performance. This highlights a fundamental tension between forcing uniform routing and allowing the model to select the most specialized expert for optimal performance. Our intention in exposing this challenge of MoE-based GFMs is to encourage the graph learning community to actively seek solutions. Drawing on the successful experiences of the LLM community, there are currently several promising approaches worth exploring:\n- Auxiliary Load Balancing Loss: This is the most common strategy, used in seminal works like [2][3]. An auxiliary loss term is added to the main training objective to penalize router imbalance.\n- Auxiliary-Loss-Free Balancing Strategies: More recent work has focused on achieving balance without auxiliary losses, for instance, the work by Wang et al. [4], which we cite, proposes a redesign of the routing algorithm to deterministically balance the load across experts while prioritizing high-confidence assignments.\n- Expert Pruning Techniques: An alternative perspective is to treat chronic underutilization as a signal of expert redundancy.  The method proposed by Lu et al. [5] could be used to identify and remove less useful experts. This would not enforce balance but would improve computational efficiency by removing the \"dead weight\" of unused parameters.\n\n[2]. Noam Shazeer et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. *ICLR 2017.*\n\n[3]. William Fedus et al. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. *JMLR 2022.*\n\n[4]. Lean Wang et al. Auxiliary-loss-free load balancing strategy for mixture-of-experts. *arXiv 2024.*\n\n[5]. Xudong Lu et al. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models. *ACL 2024.*\n\n> *W3: Insufficient analysis of when and why the methods work well across different graph types. A more principled analysis of what graph characteristics, e.g., size, feature dimensionality, domain, make certain datasets more amenable to the multi-expert approach would be good.*\n\nThis is an excellent suggestion. We agree that a more principled analysis of what graph characteristics lead to performance gains would strengthen the paper.\n- **Initial Analysis:** We have provided an initial analysis in Section 5.2, where we note that our methods give a particularly strong boost to datasets in the E-commerce domain. Additionally, in Appendix F, we discuss how our multi-expert approach helps mitigate the class imbalance issue in the Arxiv dataset by leveraging diverse expert perspectives.\n- **Action Taken:** To provide a deeper analysis as requested, we correlate the performance gains from Table 5 with the graph properties listed in Table 4. Comparing the results of Arxiv and Arxiv-t (the raw text features are provided and are encoded by a language model to get the node feature matrix), where our methods achieve a performance boost on Arxiv-t against AnyGraph but degrade on Arxiv, we guess that, the feature quality is essential for the benefits of expert merging. Following this insight, we can explain why the E-commerce datasets such as Fitness, Goodreads and Photo, benefit more from expert merging since they all use the semantic-rich node features. This may be due to the high-quality features leading to more accurate expert assignment and making it easier to merge features. We will add this analysis to Appendix F.\n\n> *Q3: How does expert imbalance affect the theoretical guarantees of Theorem 1, especially when some experts receive minimal training?*\n\nWe appreciate the reviewer's careful consideration of this theoretical question. \nThe premise of Theorem 1 is conditional; it applies specifically to the set of experts selected for merging at a given time. The theorem requires that the parameters of the selected top-$k$ experts are close to each other. It does not impose any requirements on the state of the unselected experts.\n\nOur method, PPEM, works in synergy with the upstream router in a way that self-selects for the theorem's conditions:\n\nThe router's primary function is to identify the most competent experts for a given graph. As our analysis in Appendix H shows, the router consistently assigns high competence scores to a small subset of specialized experts. Chronically under-trained experts receive low scores and are therefore very rarely included in the selected top-$k$ set for each graph.\nConsequently, the EMA update in PPEM (Eq. 13) is almost always applied to a set of well-trained, task-relevant experts, making them amenable to the convergence condition required by Theorem 1."}]

Author rebuttal:

Meta-review: This manuscript considers the problem of mixture-of-experts in building graph foundation models. The authors begin with an intriguing empirical finding: that using the top-ranked expert for each graph does not necessarily give the best performance. The authors proceed to develop two nuanced expert merging strategies that appear to provide improved performance in a series of empirical studies.

This work was received with significant enthusiasm from the reviewers, who found the problem interesting, the proposed methods well-motivated and technically sound, and the experiments convincing. I would go further to say that the reviewers expressed the feeling of having come away after reading the paper with new found _insight_ that they did not have before.

Although the reviewers expressed some minor concerns during the review period, these were adequately addressed during the discussion period.

In short, this appears to be high-quality work that should be presented at the conference without question.
Decision: Accept (poster)