Title: Mixture of Scope Experts at Test: Generalizing Deeper Graph Neural Networks with Shallow Variants
Venue: NeurIPS 2025
Accepted: yes
Keywords: heterophily

deeper GNNs

mixture of experts
Abstract: Heterophilous graphs, where dissimilar nodes tend to connect, pose a challenge for graph neural networks (GNNs). Increasing the GNN depth can expand the scope (i.e., receptive field), potentially finding homophily from the higher-order neighborhoods. However, GNNs suffer from performance degradation as depth increases. Despite having better expressivity, state-of-the-art deeper GNNs achieve only marginal improvements compared to their shallow variants. Through theoretical and empirical analysis, we systematically demonstrate a shift in GNN generalization preferences across nodes with different homophily levels as depth increases. This creates a disparity in generalization patterns between GNN models with varying depth. Based on these findings, we propose to improve deeper GNN generalization while maintaining high expressivity by Mixture of scope experts at test (Moscat). Experimental results show that Moscat works flexibly with various GNN architectures across a wide range of datasets while significantly improving accuracy.


Peer reviews:
comments: SUMMARY:
This paper investigates the problem of deeper GNNs theoretically and empirically, from which tries to propose a post-processing method of mixture of experts to improve the generalization capability while maintaining the high expressiveness. The proposed method is verified with extensive experiments that have been reported in Appendix.

STRENGTHS AND WEAKNESSES:
* Strengths
- Sound baseline of theoretical investigation on the problem of deeper GNNs.
- Reasonable proposal of the proposed method.
- Extensive evaluation with large variations of datasets and metrics.

* Weaknesses
- Optimal number of experts and deep and shallow models should be presented in detail.
- The main text needs more refined presentation. Especially, the figures need to be replaced with readable size and format.

QUESTIONS:
- How can you decide the optimal number of experts automatically?
- Is the implementation of comparing methods fully conducted to the original proposals?
- What is the limitation of the proposed method? It should be discussed at the conclusion for the future work.

LIMITATIONS:
The same as above.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a key phenomenon in Graph Neural Networks (GNNs): with increasing depth, they successfully capture homophily from multi-hop neighborhoods, yet exhibit a generalization bias across nodes with different degrees of homophily. Building on this finding, we propose a novel GNN architecture inspired by the Mixture-of-Experts (MoE) framework. Extensive experimental results validate the efficacy of our proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper's innovative use of a Mixture-of-Experts (MoE) is interesting. This allows the GNN to learn adaptive depths for distinct \"subgroups\" of nodes (e.g., those with varying homophily levels), which is a clever solution to handle non-uniform graph structures.\n- The proposed method exhibits superior flexibility and performance. These claims are well-supported by extensive and rigorous experiments that validate the architecture's effectiveness across various benchmarks.\n\nWeakness\n\n- For the motivation of the paper, particularly the first three paragraphs,\n    - Figure 1 shows how different levels of homophily affect the performance of GNNs. When we intentionally change the graph structure to make it more or less homophilous, most standard GNNs respond in similar ways because their message-passing relies on homophily. However, I don’t think this experiment here motivates the proposed method well.\n    - The authors claim that deeper GNNs can capture homophily over a larger scope, or in common terms, a larger receptive field. However, it is not clear why a larger scope would help GNNs on heterophilous graphs. This point is not sufficiently discussed in the paper, nor are any related works cited. Heterophily is a well-studied area with many existing approaches, such as heterophily-oriented GNNs and Graph Transformers. Most of these methods do not treat the receptive field as a primary tool for addressing heterophily.\n\nQUESTIONS:\nWith many mathematical symbols and formulas, the paper ultimately concludes that some parts of the network should be shallower while others should be deeper. However, the transition from the theoretical discussion to the Mixture-of-Experts framework is not sufficiently motivated. From my perspective, it currently feels somewhat like the we wanted a theory and then developed one to fit. While this new perspective—linking network depth and heterophily—is interesting and novel, I would like to see a stronger connection established between homophily and network depth.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the generalization of deep layer GNNs and find deeper GNNs achieve better generalization than their shallow variants in either homophily or heterophily region, they sacrifice generalization performance in the other region. Then, the authors propose a mixture of experts model to select GNNs with different layers for different nodes. Experimental results demonstrate the proposed method can be applied in to different backbones.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The problem of generalization disparity of deep GNNs is worth to explore.\n2. The paper is well-written and easy to follow.\n\nWeaknesses:\n1. The novelty is limited. The generalization disparity of deep gnns has been studied in several papers, for example, GMoE [1] also argue that different nodes require different hop of information. The theoretical analysis majorly follows [2], and I did not find something new here. Also, the MoE in GNNs is also utilized in several papers. For example, Link-MoE [3] also first train different GNN experts and then train the gating model to select different experts for different samples.\n\n2. The authors filter out heterophilous nodes when training the experts. However, the heterophily of nodes are unknown due to the lack of ground-truth. How do the authors select the heterophilous nodes?\n\n3. The authors leverage a high labeling rate in the experiments, which is impractical in the real-world applications. Can the proposed method work well when the labeling rate is low? \n\n[1] Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling\n\n[2] Demystifying structural disparity in graph neural networks: Can one size fit all? \n\n[3] Mixture of Link Predictors on Graphs\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nThe authors should illustrate how do they select the heterophilous nodes and the performance on the low labeling rate.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Moscat, a framework for training a mixture of GNNs that addresses existing limitations of over-smoothing and generalization. Different from previous work relying on mixtures of experts (MoE) where experts are trained jointly together with an agreggation module, Moscat proposes to train experts separately (and possibly simultaneously), and then train a gating module aimed at combining their strengths and improving generalization. The authors present extensive theoretical and empirical findings motivating Moscat, as well as experimental results that demonstrate the effectiveness of their proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The authors provide a comprehensive motivation for their approach, rooted in issues like over-smoothing, vanishing gradients, and overfitting. Theorem 3 is insightful as it points to an explanation for the low performance of GNNs in some cases, involving disparity of generalization between homophilous and heterophilous groups of nodes at different layers of a GNN.\n2. The previous result is accompanied by experiments that support the low overlap between correctly classified nodes across different layers of a GNN.\n3. In spite of several heuristics included in Moscat, the overall approach is simple and intuitive: instead of designing specific GNNs, or jointly training experts, train separately L experts and then train a gating model. This is useful because the framework can be applied to a broad class of GNNs, as evidenced in the experiments.\n4. The approach incorporates heuristics that are well motivated and effective, such as heterophily-biased sample filtering and scope-aware logit augmentation.\n5. The improvements brough by Moscat are clear and significant, both when examining the improvements over the same architecture, and in comparison with strong baselines (including MoE approaches).\n6. The authors provide ablation experiments that justify the different parts of Moscat (Table 4) and interesting interpretation experiments in the appendix.\n\n**Weaknesses**\n\n1. The fundamental message behind Moscat is that in general, a single deep GNN is almost guaranteed to perform poorly on node classification. Therefore, the solution is to train several GNNs of different depths, while learning how to combine them properly. This means that the cost of training Moscat is always larger than the cost of a single GNN.  Runtime analysis is deferred to the appendix, though I believe it should play a more central role; but more importantly it should also include a **space complexity** analysis, together with experimental results, that answers the question: how much more expensive is it to run Moscat? This would help contextualize if the observed gains are worth the cost.\n2. The hold-out set seems important for training the gating model, but it seems that the authors adopt a single static way to generate this split. Whether this is a source of variation in performance is not clear.\n3. The approach is composed by several moving parts (underlying GNN architecture, number of layers, hold-out set, heterophily-biased sampling, scope-aware logit augmentation, plus tunable hyperparameters), rather than a more principled solution to the issues of GNNs discussed and found by the authors. I would expect that this would limit the practical impact of Moscat due to increased systems complexity and optimization cost, in contrast with a more principled solution.\n4. Even though the paper is focused on the case of heterophilous graphs, it would have been interesting to see if Moscat also works well on homophilous graphs. The closest case is Pubmed, but extending these experiments would further increase our understanding of how performant and broadly applicable Moscat is.\n\nQUESTIONS:\n1. How much more expensive is it to run Moscat in terms of memory usage, in comparison with a single GNN?\n2. Is Moscat robust to the way the data is split for generating the hold-out set used to train the gating model?\n3. When faced with new datasets in real world problems, how would you advice practitioners starting with Moscat to make the large number of options (underlying GNN architecture, number of layers, hold-out set, heterophily-biased sampling, scope-aware logit augmentation, plus tunable hyperparameters) more manageable?\n4. Could you please clarify which parts of Moscat adress each of the challenges listed at the beginning of Section 4 (L217-226)?\n\nLIMITATIONS:\nThe authors do not discuss the limitations of Moscat (apart from a short sentence in the Checklist), which justifies my Quality score. I would advice the authors to reflect on limitations such as cost, and the lack of a more principled approach, which could help understand the method as well as making future directions of research more clear.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-06"}