Title: Unifying and Enhancing Graph Transformers via a Hierarchical Mask Framework
Venue: NeurIPS 2025
Accepted: yes
Keywords: Graph Neural Networks

Graph Transformers

Graph Representation Learning
Abstract: Graph Transformers (GTs) have emerged as a powerful paradigm for graph representation learning due to their ability to model diverse node interactions.
However, existing GTs often rely on intricate architectural designs tailored to specific interactions, limiting their flexibly.
To address this, we propose a unified hierarchical mask framework that reveals an underlying equivalence between model architecture and attention mask construction.
This framework enables a consistent modeling paradigm by capturing diverse interactions through carefully designed attention masks.
Theoretical analysis under this framework demonstrates that the probability of correct classification positively correlates with the receptive field size and label consistency, leading to a fundamental design principle:
An effective attention mask should ensure both a sufficiently large receptive field and a high level of label consistency.
While no single existing mask satisfies this principle across all scenarios, our analysis reveals that hierarchical masks offer complementary strengths—motivating their effective integration.
Then, we introduce M$^3$Dphormer, a Mixture-of-Experts based Graph Transformer with Multi-Level Masking and Dual Attention Computation.
M$^3$Dphormer incorporates three theoretically grounded hierarchical masks and employs a bi-level expert routing mechanism to adaptively integrate multi-level interaction information.
To ensure scalability, we further introduce a dual attention computation scheme that dynamically switches between dense and sparse modes based on local mask sparsity.
Extensive experiments across multiple benchmarks demonstrate that M$^3$Dphormer achieves state-of-the-art performance, 
validating the effectiveness of our unified framework and model design.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits Graph Transformers through the lens of hierarchical interactions and proposes a unified hierarchical mask framework, which models different interaction levels via attention masks rather than architectural modifications. This design simplifies the model space and promotes structural consistency. Theoretical analysis under this framework identifies two key factors affecting node classification — receptive field size and label consistency — and argues that no single interaction level suffices across all scenarios. This motivates the integration of multiple levels of interaction. To address this, the authors propose M$^3$DPhormer, which uses three hierarchical masks, a bi-level expert routing mechanism for adaptive fusion, and a dual attention computation scheme for scalability. Experimental results across benchmarks demonstrate the effectiveness of both the framework and the proposed model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The paper is well-organized. It starts with a solid review of existing Graph Transformers and introduces a unified hierarchical mask framework. Both the theory and experiments are well-aligned, clearly identifying core challenges in modeling hierarchical interactions and proposing targeted solutions through M$^3$DPhormer. \n2. The framework is valuable in that it shifts focus away from complex architectures, showing that diverse interactions can be captured more cleanly via attention mask design. This abstraction also makes the theoretical analysis more tractable and informative for future model design.\n 3. M$^3$DPhormer is a thoughtful and well-motivated model. The bi-level expert routing supports adaptive multi-level fusion, while the dual attention scheme improves scalability. Experimental results are strong and provide convincing support for the proposed method.\n\nWeaknesses: \n1. More details about the proposed method should be clarified. In particular, it would be helpful to explain how the number of clusters is determined when constructing the cluster-level mask, and how the appropriate computation scheme is selected for different regions within the masks.\n\nQUESTIONS:\n1.\tGiven that the main difference between experts lies in their attention masks, have the authors considered sharing parameters across the MHA modules to improve parameter efficiency?\n2.\tThe authors state that node classification performance is positively correlated with label consistency. Could the authors clarify how label consistency differs from the homophily rate?\n\nLIMITATIONS:\nYES", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a unified framework based on hierarchical masks, demonstrating that existing Graph Transformers are implicitly designed to model hierarchical interactions, and that various interaction types can be effectively captured through the design of appropriate attention masks. Under this framework, they conduct a theoretical analysis of the node classification task and identify a core design principle for Graph Transformers. Guided by this principle, the authors develop M$^3$DPhormer, which integrates theoretically grounded mask designs, a bi-level attention Mixture-of-Experts mechanism, and a dual attention computation scheme. Extensive experiments are conducted to validate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The authors propose a unified framework from the perspective of hierarchical masks, which reduces the design space to the construction of attention masks, avoiding reliance on intricate architectural designs. \n2. The authors develop a novel theoretical framework to analyze the distinct roles of various types of interactions and establish a theoretically grounded design principle that can guide future research in the field. The theoretical analysis is comprehensive, and the proofs are rigorous and well-founded.\n3. The proposed method M$^3$DPhormer is novel. Based on theoretical insights, the authors design three hierarchical attention masks and introduce a bi-level expert attention routing mechanism, where each MHA module with a distinct attention mask is treated as an expert. This allows the model to adaptively select the most informative interaction for each node. Additionally, a novel dual attention computation scheme is presented to improve computational efficiency.\n4. Comprehensive experiments are conducted to validate the effectiveness of the proposed approach, demonstrating its advantages over existing methods.\nWeaknesses:\nOverall, the paper is strong. One minor drawback is the lack of discussion on graph-level tasks, which could further demonstrate the generality of the proposed framework.\n\nQUESTIONS:\nIn the standard Transformer architecture, the FFN module is essential for enhancing representational capacity. However, the proposed method seems to omit this component. What is the motivation of this modification?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces M3Dphormer, a graph Transformer architecture, and a framework for modeling graph interactions at multiple levels (local, cluster, global) through hierarchical attention masks. The central premise is that various Graph Transformer (GT) designs can be interpreted as applying different attention mask patterns. The authors unify these under a single masking framework and provide a theoretical analysis suggesting that classification accuracy correlates with a node’s receptive field size and the label consistency of its neighborhood. This leads to the design principle that an effective attention mask should cover a large, label-homogeneous neighborhood. Arguing that no single mask type is optimal across all nodes and graphs, the authors propose combining hierarchical masks.\n \nThe paper then presents M3Dphormer, a Mixture-of-Experts (MoE) based graph Transformer that integrates three mask types (local, cluster, global). The model uses a bi-level gating mechanism to weight expert modules, each associated with a specific mask type. To manage computational costs on large graphs, it employs a dual-mode attention computation that switches between dense and sparse implementations based on mask sparsity. The authors report experimental results on nine benchmark datasets, comparing M3Dphormer against 15 baselines. The model is shown to achieve strong performance across these datasets.\n \nThe main contributions are: first, a unified hierarchical mask framework that categorizes existing GT architectures; second, a theoretical analysis linking receptive field size and neighborhood label consistency to classification accuracy; third, the M3Dphormer model, which features multi-level masks, a bi-level MoE routing mechanism, and a dual-mode attention for scalability; and fourth, an empirical evaluation demonstrating performance gains over baseline models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nModel Design (M3Dphormer): The model architecture directly addresses the issues identified in the analysis. By using separate attention \"experts\" for local, cluster, and global interactions, combined with a bi-level gating mechanism to weight them on a per-node basis, M3Dphormer can adaptively combine information from different scales. The hierarchical nature of the gating—first deciding between local and non-local, then between cluster and global—is a reasonable approach to prioritize local signals while incorporating broader context.\n \nScalability via Dual-Mode Attention: The introduction of a dual-mode attention computation is a practical component for scalability. Since graph attention masks often have irregular sparsity, standard dense attention can be inefficient. The model switches between sparse and dense attention based on a derived criterion (Proposition 4.2) to optimize computation. This results in a significant reduction in memory usage.\n \nEmpirical Performance: M3Dphormer achieves strong empirical results on a range of benchmarks, including both homophilic and heterophilic graphs. The performance improvements are most notable on heterophilic datasets, where multi-scale information is often critical. The model is benchmarked against a comprehensive set of baselines, including recent GTs and tuned GNNs, which strengthens the empirical claims. Ablation studies in Table 3 demonstrate the contribution of each component of the proposed model.\n \nClarity and Organization: The paper is generally well-structured and clearly written. The hierarchical mask framework is introduced with helpful examples (Figure 1) and a clear mapping to prior work (Table 4), which contextualizes the contributions.\n\nWeaknesses\n  \nTheoretical Assumptions and Generality: The analysis is based on a simplified generative model. The class-conditional Gaussian assumption, with orthogonal mean vectors and isotropic variance for each class, is a strong simplification of real-world feature distributions. Furthermore, the model design, particularly the global mask, injects label information from the training set by creating global virtual nodes for each class. This tailors the model to semi-supervised node classification, and its applicability to other tasks (e.g., graph classification, link prediction, or settings with unseen classes) is not demonstrated.\n\nQUESTIONS:\nAblation on Global Nodes: An ablation study comparing the label-informed global nodes to generic, unlabeled global tokens (as in Exphormer) would be valuable. This would quantify the benefit of injecting supervised signals into the global attention mechanism.\n \nRuntime Performance: Including training time or throughput metrics would provide a more complete picture of the model's practical efficiency alongside pre existing memory usage analysis.\n\nLIMITATIONS:\nScope of Evaluation: The evaluation is confined to node classification. While the results are strong, this narrow focus leaves open questions about the framework's generalizability. The performance gains on standard homophilic benchmarks (e.g., Pubmed, Ogbn-Arxiv) are modest, suggesting the model's complexity may not be justified in all scenarios. It would also be informative to compare against a simpler integration of the masks, such as concatenating them and allowing a standard attention mechanism to learn the weights, to better isolate the benefit of the explicit bi-level gating.\n\nLimited Novelty of Components: The novelty is somewhat limited, as individual components are adapted from prior work. The use of virtual nodes for clusters and global tokens has appeared in models like GraphViT, Cluster-GT, and Exphormer. The Mixture-of-Experts (MoE) gating mechanism is also an established technique in GNNs (e.g., GCN-MoE, MOWST). While the specific combination of three expert types within a bi-level gating framework for GTs appears new, the main contribution lies in the integration and empirical validation of these components rather than a fundamentally new algorithmic paradigm.\n\n\nWeaknesses related to component-level novelty and a narrow task focus still remain.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes a unified framework that reconceptualizes various Graph Transformer architectures as the design of hierarchical attention masks, leading to a key design principle: an effective mask must balance a large receptive field with high label consistency. Guided by this theory, the authors introduce M³I, a novel Mixture-of-Experts-based model that adaptively combines local, cluster, and global interaction masks to achieve state-of-the-art results on multiple node classification benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper is well-organized.
2. There is comprehensive theoretical analysis.
3. The performance is great.

Weaknesses:
1. The proposed method does not truly achieve state-of-the-art performance. The results reported in papers such as https://arxiv.org/pdf/2410.02296 and https://arxiv.org/pdf/2410.13798 are competitive. More baselines should be included for comparison.
2. The experimental evaluation is insufficient. Additional experiments, such as analyses of time and space complexity, should be conducted.
3. The paper claims that the code is publicly available, but the code for the proposed method has not been released.
4 .The authors only evaluate performance on node classification tasks, which limits the persuasiveness of the proposed method.

QUESTIONS:
See Weaknesses. If the more convincing experimental results can be provided, I might consider increasing the score.

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02