Title: HubGT: Fast Graph Transformer with Decoupled Hierarchy Labeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Graph neural networks; Graph Transformer; Shortest Path Distance
Abstract: Graph Transformer (GT) has recently emerged as a promising neural network architecture for learning graph-structured data. However, its global attention mechanism with quadratic complexity concerning the graph scale prevents wider application to large graphs. Effectively representing graph information while ensuring learning efficiency remains challenging, as our analysis reveals that current GT designs targeting scalability still suffer from the computational bottleneck related to graph-scale operations. In this work, we tackle the GT scalability issue by proposing HubGT, a scalable Graph Transformer boosted by fully decoupled graph processing and simplified learning. HubGT represents the graph by a novel hierarchical scheme exploiting hub labels, which is shown to be more informative than plain adjacency by offering global connections while promoting locality, and is particularly suitable for handling complex graph patterns such as heterophily. We also design algorithms for efficiently constructing and querying the hub label hierarchy tailored for the GT attention training in scalable deployments. Notably, the precomputation and training processes of HubGT achieve complexities linear to the number of graph edges and nodes, respectively, while the training stage completely removes graph-related computations, leading to favorable mini-batch capability and GPU utilization. Extensive experiments demonstrate that HubGT is efficient in terms of computational enhancement and mini-batch capability over existing GT designs on large-scale benchmarks, while achieving top-tier effectiveness on both homophilous and heterophilous graphs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a hierarchical method to improve the efficiency of graph transformers. In particular they provide an algorithm which is linear in the number of graph edges and nodes, in precomputing and training. The training stage completely removes graph-related computation. In the paper a thorough analysis is made of the complexities of related algorithms and also the\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n- Thorough design optimizing several aspects in the computation\n- Good analysis of the complexity of different methods.\n- Significant speed ups with respect to state-of-the-art methods \n\nWeaknesses:\n- It remains a bit unclear where the real benefits lie. Table 2 which is the main experimental result shows that for many of the datasets other methods are better in terms of accuracy. In terms of precomputation, time for epoch, and time for infer there is no clear winner. It is also hard to determine what the overall timing means. Is every epoch for the different methods comparable? Do all methods require the same number of epochs?\n- It remains unclear what the competing methods especially the ones with similar performance do. There is some more description in the appendix but also there not much is said about the actual differences with the proposed method. \n\nImprovements: \n- In the caption of table 1 it would be good to mention all variables that play a role in the complexity. L, n, F, M etc.\n- Indicate in bold / underline which method is based in computation time for the different elements.\n- Can't you create a scatterplot like visualization with the tradeoff between acc and computation? Probably separate for training and testing. Your main message is that you can reach a similar accuracy in much less time. A table doesn't easily convey this message.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces HubGT, a novel Graph Transformer (GT) framework that addresses the scalability bottlenecks of existing GTs. The paper introduces a decoupled hub-based hierarchical labeling scheme that supports efficient subgraph sampling and distance-based positional encoding (SPD). The method avoids graph operations during training and enables O(1) SPD querying, making it suitable for large-scale graphs with millions of nodes and edges. Empirical results on 14 benchmark datasets demonstrate state-of-the-art efficiency and strong predictive performance, especially in heterophilous and sparse graphs.\n\nSTRENGTHS AND WEAKNESSES:\n> Strengths of the Paper:\n\n1. The paper introduces a 3-level hub-labeling scheme (H-0, H-1, H-2) to capture both local and global graph structures. It leverages Pruned Landmark Labeling for efficient index construction and SPD approximation and cleanly decouples precomputation from training to support mini-batch learning without graph traversal.\n2. The framework achieves linear-time precomputation and training complexity independent of edge count. It consistently demonstrates the fastest inference speed across all large-scale benchmark datasets.\n3. The paper enables the use of Shortest Path Distance (SPD) as a dense positional encoding, a strategy previously too expensive for large graphs, effectively enhancing representation power, especially under heterophily.\n4. The model is evaluated on 14 benchmark datasets covering a wide range of scales and homophily levels. Ablation studies confirm the importance of SPD bias, global hub connections, and attention-based node readout for performance gains.\n\n> Weaknesses of the Paper:\n\n1. The core Transformer module is largely standard. The proposed approach could benefit from integrating efficient attention mechanisms to improve scalability.\n2. The hub-label index is constructed in a one-time pre-computation phase and remains fixed. This design may be inadequate for dynamic or temporal graphs, where incremental updates to the label hierarchy are necessary.\n3. While the focus is on Graph Transformers, the paper lacks a direct comparison with recent scalable convolutional GNNs, which also address large-scale and heterophilous settings. The only non-GT baseline included is SGFormer, which limits the scope of empirical benchmarking.\n4. The paper does not provide an in-depth analysis of hub selection, such as the statistical properties of selected hubs, cross-dataset patterns, or their structural significance, which limits the interpretability of the hub hierarchy used in H-1 and H-2 indices.\n5. Model performance depends on choices like subgraph size and the balance between \"s_in\" and \"s_out\". While the impact is partially evaluated (Figure 7), the method lacks adaptive mechanisms to optimize these parameters across diverse datasets.\n\nQUESTIONS:\n1. Can the authors provide an in-depth analysis of hub selection, such as the statistical properties of selected hubs, cross-dataset patterns, or their structural significance?\n2. Can the authors provide evidence or discuss how the hub label index could be updated incrementally in dynamic or temporal graph settings?\n3. Could the authors expand the experimental comparison to include scalable non-GT baselines?\n4. Is it feasible to combine HubGT with efficient Transformer variants such as FlashAttention or Linformer, and what are the anticipated gains or trade-offs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper proposes HubGT, a novel and efficient Graph Transformer architecture that leverages hierarchical hub labeling to enable scalable training and inference on large graphs. The key idea is to decouple the graph computation from the transformer training by introducing a three-level indexing structure for precomputing neighborhood and shortest-path information. The authors demonstrate both theoretical and empirical improvements over existing Graph Transformer baselines, especially on large-scale graphs under both homophily and heterophily settings.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. The idea of decoupling graph computation from transformer training is novel and leads to significant efficiency gains.

2. The three-level hierarchical hub labeling scheme (local, global, and 2-hop caching) is theoretically well-grounded and empirically effective.

3. The paper conducts comprehensive experiments on 14 datasets, including million-scale graphs, showing consistent improvements in speed and accuracy compared to state-of-the-art graph transformers.

**Weakness**

The evaluation only includes comparisons against Graph Transformer baselines. Stronger message-passing GNNs are missing, which limits understanding of the accuracy-efficiency tradeoff relative to GNNs.

QUESTIONS:
NA

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors describe HubGT, a Graph Transformer (GT) framework designed to tackle scalability issues that hinder the application of traditional GTs to large graphs. HubGT achieves this by decoupling graph structure processing from model training through an offline precomputation step. This step constructs a 3-level hierarchical index based on hubs, which are specially selected nodes with high connectivity that serve as central reference points within subgraphs. The index, inspired by techniques like Pruned Landmark Labeling (PLL), assigns labels to nodes based on their relationships to hubs, enabling efficient encoding of shortest path distances. During training, this hierarchical index allows shortest path distance (SPD) computation in $O(1)$ time and enriches node tokens with global \"hub context\", enhancing the model’s ability to capture broader graph structures. As a result, HubGT simplifies GT training to resemble that of a standard Transformer, supporting CPU/GPU pipelining, reducing memory overhead, and delivering state-of-the-art performance on graph benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nSome strengths of the paper are obviously related to the benefits of PLL. For example, HubGT’s hub labeling scheme has constant time for SPDs during training. Moreover a nice property is that the hierarchical indexing enriches node tokens with global “hub context,” capturing long-range dependencies and structural patterns. Finally, the idea of decoupling graph structure processing from model training should enable CPU/GPU pipelining which is an advantageous property.\n\nI do note some possible weaknesses also related to hub based labeling. For example, the dependence on quality of hub selection could potentially limit generalizability in graphs with uneven degrees or more complex network topologies (e.g. Erdos-Reyni, Barabasi-Albert, Stochastic Block Model). What about secondary effects like thrashing? For example, it is possible that the $H_0$ cache’s fixed size limit ($s^2$) could thrash in the face of many alternative paths especially with denser graphs.\n\nIn terms of clarity, mostly ok, but some problematic notation, viz. the terms theorem and property are used interchangeably, so property should be used throughout the text as the observations do not require proofs.\n\nQUESTIONS:\n- For practitioners/application developers, how are $s_(sample size)$ and $s_out$ (in/out-neighbor ratio) determined heuristically? Are optimal values related to properties of the entire graph (e.g. diameter, average degree, sparsity, etc.)?\n\n- Your hub selection is based on node degree. How robust is this strategy on graph topologies that lack clear high-degree hubs, such as random graphs or stochastic block models?\n\n- Have you experimented with other centrality measures for hub selection, and how does HubGT's performance change when high-degree nodes are not the most effective structural landmarks?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is an approach to deal with scalability of Graph Transformers (GT). This paper uses graph structure to solve two separate issues for scaling GT. \n1. Since a large graph cannot be held in memory and the attention mechanism is quadratic, naive attention does not work. Mini batching is a way to deal with large graphs. Batching might only capture parts of graph, which can be problematic for hetrophilic graphs. \n2. Positional encoding is an important problem to solve for graphs. On regular sequences the relative position of the tokens leads to natural positional encodings which do not work for permutation-invariant structure of graphs.\n\nThe area of GraphTransformers are using many techniques from the graph theory literature to deal with efficient representation learning and improved accuracy on large graphs.  This paper uses Pruned Landmark Labelling (PLL) approach by Akiba et al. [33,34] to represent the graph topology. PLL was introduced in 2013-14 in the context of large (web, social etc.) graphs to solve the shortest paths between a pair of nodes. It uses the idea of using landmark nodes to quickly compute distance between a pairs of nodes.  These are used for both batching to include representation from other nodes (especially in heterophilous graphs) and for positional encoding.\n\nEven though HubGT does not outperform many of the methods (Table 2), it is close to the performance metrics of the other good methods.  The idea proposed is good for the field and I would like the authors can address the weaknesses described below.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is well written. The time complexity analysis is very useful.\n\nWeaknesses/potential future directions:\n(I) The paper compares against NAGphormer, SGFormer and DIFFormer, which are good benchmarks for mini batching.  There are at least a couple of approaches that are not cited/compared against in the paper. \n(1) The first is Exphormer (Shirzad et al. 2023) which uses (1) virtual nodes and (2) uses an overlay graph (like in HubGT, but using a different intuition).  It would be interesting to see the mixing properties due to the two overlay graph approaches. How many additional edges are needed in HubGT are comparable to the degree of the expander graph in Exphormer? How many virtual nodes smooth the network vs. in HubGT? Exphormer might not scale well with minibatching, but SpExphormer might be better in that aspect. So comparison to HubGT will be valuable.\n\n(2) The second paper that might be relevant is Graphormer (Ying et al., 2021), where structural features (centrality and spatial encoding) for GT was proposed.  Graphormer was before the introduction of Long Range benchmarks. I am not sure if there have been studies in applying it to the LongRange benchmark. So a study here will be valuable to understand which structural approaches are more valuable.\n\nFor the final version, a discussion and some simple comparison would be valuable. And further work will be useful to understand the space well.\n\n(II) Description of the HLL algorithm.\nI found it hard to follow the labelling ideas for HubGT. I went back and forth between section 4 and appendix, but had to read the original paper to understand the approach better. The complexity of BFS approach for SPD as described in the paper is clear. I would like an easier-to-follow description of the algorithm --- how pruning works, what labels represent (Appendix A refers to algorithm 1, which uses a 3-tuple. It is too hard to follow). An easier English description would be valuable.\n\n(III) Scaling properties. The authors spend a lot of time on the complexity analysis of the paper. Scaling to REDDIT, ogbn-mag and Pokec is impressive.   However, I would also like to see the scalability in terms of memory, compute times and graph sizes. I would like a study (instead of brief paragraphs) on the memory and scalability issues. From the description it is not clear what the limitations are on a single GPU or in a multiGPU setting. A thorough description of the experimentations would be very helpful.\n\nQUESTIONS:\nDescribed in the weaknesses section above.\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}