Title: Enhanced Expert Merging for Mixture-of-Experts in Graph Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Graph Representation Learning

Graph Foundation Model

Mixture of Experts

Expert Merging
Abstract: Graph foundation models (GFMs) have emerged as a promising paradigm for learning transferable knowledge across diverse graph-structured data. The inherent heterogeneity in features and graph structures poses significant challenges for building scalable and generalizable GFMs. Existing research has employed mixture-of-experts (MoE) models to handle the challenges, assigning the most suitable expert to each graph. Despite this, the underlying mechanisms of MoE within the context of GFMs remain insufficiently explored. In this work, we conduct an in-depth experimental study on an MoE-based GFM and uncover an intriguing finding: the experts ranked second and third assigned by the router perform better than the top-ranked expert. This insight motivates us to investigate the potential of leveraging knowledge embedded across multiple experts. However, directly ensembling the outputs of multiple experts would incur substantial computational overhead, while applying a standard expert merging strategy risks suboptimal performance. To address these challenges, we introduce two enhanced expert merging strategies that retain the computational efficiency of expert merging, while improving performance to approach the effectiveness of expert ensembling. Specifically, we propose (i) a knowledge distillation-inspired expert merging method that aligns the behavior of parameter-fused experts with expert ensembles, and (ii) a theoretical parameter proximity approach that leverages the similarity of expert parameters to approximate ensemble outputs while preserving diversity. Extensive experiments demonstrate that our methods effectively enhance model performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates mixture-of-experts (MoE) mechanisms in graph foundation models (GFMs) and addresses the challenge that while GFMs use MoE to handle graph heterogeneity, current approaches only utilize the single top-ranked expert, potentially leaving valuable knowledge from other high-performing experts untapped. Through experimental analysis of AnyGraph, the authors discover that experts ranked second and third by the router often outperform the top-ranked expert, motivating them to explore multi-expert knowledge integration. They propose two enhanced expert merging strategies: (1) Knowledge Distillation Enhanced Expert Merging (KDEM), which uses knowledge distillation to align the behavior of parameter-fused experts with expert ensembles, and (2) Parameter Proximity Enhanced Expert Merging (PPEM), which leverages theoretical analysis showing that when MLP experts have similar parameters, their merged output approximates the ensemble output. The proposed approach bridges the efficiency of expert merging with the performance benefits of expert ensembling through these theoretically-grounded and practically-efficient approaches. Extensive experiments demonstrate that KDEM and PPEM achieve significant performance improvements over the AnyGraph baseline while maintaining computational efficiency close to single-expert inference, which effectively solve the trade-off between performance and efficiency in MoE-based GFMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n•\tThe motivation is sound, which is based on straightforward empirical observation. This points out the deficiency of current approaches and poses a natural goal to solve the explicitly existing problem.\n•\tThe paper presents two complementary solutions addressing the same problem from different angles, where KDEM operates in output space using knowledge distillation, while PPEM works in parameter space using theoretical proximity principles.\n•\tThe theoretical validation for PPEM is sound and solid. Theorem 1 provides a solid mathematical basis for PPEM, proving that when MLP expert parameters converge pairwise, the merged MLP output approximates the convex combination of individual expert outputs. This insight directly translates into PPEM that maintains expert diversity while reducing parameter interference.\n•\tThis work demonstrates substantial empirical contributions. Evaluation on 38 datasets spanning diverse domains with rigorous zero-shot evaluation protocols ensures robustness and generalizability. The proposed approach achieves significant improvements (5.27% and 4.35% recall improvement on Link2) while maintaining computational efficiency nearly identical to single-expert inference. As for efficiency, both proposed expert merging strategies introduce minimal computational overhead - KDEM adds only 4.8% training time, while PPEM has negligible increase, making them practically deployable.\nWeaknesses:\n•\tThe paper inherits AnyGraph's routing mechanism without questioning its optimality for multi-expert scenarios.\n•\tThe expert imbalance problem (some experts rarely selected) is acknowledged but not systematically addressed.\n•\tInsufficient analysis of when and why the methods work well across different graph types. A more principled analysis of what graph characteristics, e.g., size, feature dimensionality, domain, make certain datasets more amenable to the multi-expert approach would be good.\n\nQUESTIONS:\n•\tHave the authors investigated whether the non-parametric routing mechanism is actually optimal for scenarios where multiple experts are merged?\n•\tIn Appendix H.3, the authors briefly show that trainable routers perform worse, could the authors explore other routing strategies specifically designed for multi-expert selection?\n•\tBeyond adjusting the calibration parameter $rho$, what principled approaches could ensure more balanced expert utilization while maintaining performance?\n•\tHow does expert imbalance affect the theoretical guarantees of Theorem 1, especially when some experts receive minimal training?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
Based on the current Graph MoE models like Anygraph, this works provide detailed discussitions on Expert Selection and Merging. After analyzation on this, this work propose two improved expert
merging strategies, KDEM and PPEM. Further Experiments show that they can achieve better performance than naive experts merging mechanism.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The authors address an important and previously under-explored problem — the selection and merging of experts in Graph MoE models 

2. Preliminary analysis is both reasonable and insightful.
The method proposed by the authors is effective and powerful.

Weaknesses:

Will these processes (KDEM or PPEM) be harmful to the training time? It seems quite difficult to converge, especially when KD is introduced in the middle of the training process.

QUESTIONS:
1. L183 : "Since the performance of the top-k experts’ ensemble is excellent, we treat it as a teacher model to teach the parameter-fused expert." How is the "top-k" determined in a MoE model?

2. KD-based methods for merging graph-based models have been widely discussed, such as in ParetoGNN[1](during training) or WAS[2](during fine-tuning). Could you please provide more details about your design? Why your approach is unique or different?

---

[1]ICLR'23 https://arxiv.org/abs/2210.02016

[2]ICLR'24 https://arxiv.org/abs/2403.01400

LIMITATIONS:
N.A.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose two enhanced expert merging strategies for Graph Foundation Models (GFMs) to improve performance while maintaining computational efficiency. The authors aim to address the challenges of the two extreme versions -- (1) a single top-ranked expert underutilizing knowledge from multiple experts and (2) full ensemble strategies with huge computational overhead.\n\nIn particular, the authors perform an in-depth analysis of an MoE-based GFM and discover that the second- and third-ranked experts often outperform the top-ranked expert selected by the router, revealing valuable but untapped expertise. Motivated by this insight, authors propose two methods: (i) Knowledge Distillation Enhanced Expert Merging (KDEM), which aligns a fused expert’s behavior with that of an expert ensemble via periodic distillation, and (ii) Parameter Proximity Enhanced Expert Merging (PPEM), which reduces parameter distances among top-k experts so that merging approximates ensemble outputs while preserving diversity. These approaches are designed to harness the strengths of multiple experts efficiently.\n\nAuthors perform extensive experiments on 38 graph datasets, evaluating their methods under zero-shot and few-shot settings against baseline GFMs, classical GNNs, and standard expert merging. Results show that KDEM and PPEM consistently improve performance over single-expert routing and standard merging while maintaining low computational costs. They further provide ablation studies, theoretical analysis, and routing visualizations to validate the effectiveness and efficiency of the proposed methods.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths\n- Authors carefully analyze the behavior of existing MoE-based GFMs and identify the important challenge that second- and third-ranked experts often outperform the top expert chosen by routing. This empirical observation is clearly presented and motivates the necessity for improved multi-expert utilization, which grounds the rest of the work in a real and measurable problem.\n- Authors provide a clean and rigorous rationale of the proposed solutions, combining empirical findings and theoretical insights—for example, proving that parameter proximity enables parameter merging to approximate expert ensembles. This thoroughness makes the proposed approaches well justified and not merely heuristic.\n- The introduction of knowledge distillation into expert merging (KDEM) is novel to tackle the challenges in the ensemble learning of GFMs. The method intuitively bridges the gap between the efficiency of expert merging and the performance of expert ensembles, and the authors provide practical implementation details that make it feasible.\n- The authors perform experiments on 38 diverse graph datasets across domains, under both zero-shot and few-shot settings. This broad scope of evaluation, along with detailed ablation studies, strongly supports the validity and robustness of the proposed methods.\n- The paper is well-written and organized, making a technically complex topic accessible. Figures, tables, and theoretical explanations are presented in a way that is easy to follow, which enhances the reader’s understanding.\n\n\nWeakness\n- The core contributions of the paper lie in improving expert merging strategies within an existing MoE-based GFM framework rather than proposing a fundamentally new architecture or a completely novel GFM paradigm. While the enhancements are well-justified and practically useful, some readers might view the work as an incremental improvement rather than a groundbreaking innovation.\n- Despite recognizing the limitations and risks of parameter averaging in expert merging (e.g., parameter interference due to expert specialization), the authors focus solely on improving parameter averaging rather than exploring fundamentally different fusion mechanisms. \n- The PPEM method intentionally brings expert parameters closer together to enable more effective merging, while the main purpose of MOE is to cover multi-modal diversities. While proper regularization might work well in practice to balance between these two conflicting goals, the authors need to address this concern.\n\nQUESTIONS:\n- What exactly does  `-EMA` in Table 3 indicate?\n- It would be great if Figure 4 can be readable along with performance as well (i.e. clear inteprataion over time / top K / performance)\n- It would be great to see how different experts are trained between KDEM and PPEM, given that PPEM adds the restriction of not diverging too much.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a key limitation of the current mixture-of-experts (MoE)-based graph foundation model, which relies solely on the top-ranked expert, by showing that lower-ranked experts (e.g., second- and third-ranked) can often perform better. To more effectively harness the collective knowledge of multiple experts, this paper proposes two novel expert merging techniques: (1) KDEM, which distills knowledge from an ensemble of top-k experts into a parameter-merged expert, and (2) PPEM, which encourages parameter alignment among the selected experts. The resulting model achieves strong performance on several downstream tasks in a near-consistent manner and improves training and inference efficiency over naive expert ensembling.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\nS1. The paper is well-motivated and easy to follow. It builds on an interesting observation: in mixture-of-experts (MoE) models, the second- and third-ranked experts often outperform the top-ranked one. This highlights (1) a limitation of the recent MoE-based graph foundation model, AnyGraph, which uses only the top-ranked (i.e., most competent) expert, and (2) the need to incorporate the knowledge embedded in multiple experts.\n\nS2. The paper introduces two interesting and novel expert merging techniques: Knowledge Distillation Enhanced Expert Merging (KDEM) and Parameter Proximity Enhanced Expert Merging (PPEM). KDEM leverages knowledge distillation, using an ensemble of the top-k experts as the teacher and the parameter-merged expert as the student. PPEM encourages the parameters of the top-k experts selected by the router to move gradually in the same direction.\n\nS3. The proposed method near-consistently outperforms several existing graph foundation models and graph learning methods on downstream tasks. Ablation studies demonstrate the effectiveness of the expert merging techniques, as well as internal components such as the use of EMA. Compared to naive expert ensembling, the method also achieves improved training and inference efficiency, as supported by experimental results.\n\n\n[Weaknesses]\n\nW1. The paper currently lacks a comparison between the proposed method and existing expert merging strategies. The experiments focus on comparing the proposed method against graph foundation models and graph learning methods for graph tasks such as link prediction and node classification. However, the core technical contributions of this work lie in the development of expert merging techniques, which are not specific to graph data or graph learning methods. Given the existence of prior expert merging approaches such as SMEAR [28] and Lory [51], a comparison with those methods is necessary to establish the proposed technique as a new state of the art.\n\nW2. The number of total and selected experts (denoted as K and k, respectively) has a significant impact on performance, as demonstrated in the experiments. However, these values must be manually specified prior to training, which may be non-trivial in the context of graph foundation models spanning diverse datasets and tasks. It would strengthen the paper to provide guidance or at least discussion on how to choose these values effectively and efficiently.\n\nW3. The generalizability of the proposed method to new domains and tasks appears limited or remains unverified. A core requirement for a graph foundation model (GFM) is the ability to generalize to unseen domains and tasks. While the proposed GFM builds on AnyGraph and improves it by leveraging multiple experts, it remains unclear whether the selected experts can perform well under significant domain or task shifts. The current experiments use two dataset groups (Link 1 and Link 2), but these are relatively homogeneous in both domain and task. It would strengthen the paper to include evaluations on more diverse domains and tasks to better assess the model’s robustness in generalization scenarios, which are central to the purpose of GFMs.\n\nQUESTIONS:\nPlease refer to the main review for detailed comments and suggestions.\n\nOne additional question: The paper applies KDEM and PPEM only individually. Was combining the two techniques considered? If so, why was this not pursued? If it was not considered, do the authors believe the two techniques could be complementary if used together?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}