Title: Generalizable Insights for Graph Transformers in Theory and Practice
Venue: NeurIPS 2025
Accepted: yes
Keywords: graph transformers

expressivity

structural encodings

structural embeddings

positional encodings
Abstract: Graph Transformers (GTs) have shown strong empirical performance, yet current architectures vary widely in their use of attention mechanisms, positional embeddings (PEs), and expressivity. Existing expressivity results are often tied to specific design choices and lack comprehensive empirical validation on large-scale data. This leaves a gap between theory and practice, preventing generalizable insights that exceed particular application domains. Here, we propose the Generalized-Distance Transformer (GDT), a GT architecture using standard attention that incorporates many advancements for GTs from recent years, and develop a fine-grained understanding of the GDT's representation power in terms of attention and PEs. Through extensive experiments, we identify design choices that consistently perform well across various applications, tasks, and model scales, demonstrating strong performance in a few-shot transfer setting without fine-tuning. Our evaluation covers over eight million graphs with roughly 270M tokens across diverse domains, including image-based object detection, molecular property prediction, code summarization, and out-of-distribution algorithmic reasoning. We distill our theoretical and practical findings into several generalizable insights about effective GT design, training, and inference.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new graph transformer architecture called the Generalized Distance Transformer (GDTransformer). Unlike some existing graph transformer architectures that use architectural components specialized to graphs, GDTransformer uses the standard transformer architecture. Despite this, the paper proves that GDTransformer matches the expressive power of GDGraphormer. Given that the expressive power of the GDTransformer is determined by the choice of relative positional encoding, the authors prove several theoretical results comparing the expressive power of different positional encodings. The authors then empirically compare the performance of GDTransformer with different positional encoding on a mixture of real-world and algorithmic benchmarks. The authors also perform a set of experiments on the few-shot learning and size generalization capabilities of GDTransformer.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- This paper prove (Theorem 1) that the standard transformer architecture has the same expressive power as specially designed graph transformers like GDGraphormer; namely, both have the same expressive power as the GD-WL test. In contrast to GDGraphormer that used the relative PE both inside and outside of the softmax, GDTransformer proves that it is sufficient to use the relative PE inside of the softmax. Most standard transformer implementation only use relative positional encodings inside of the softmax. This results means that the one can use standard transformer implementations rather than specialized implementations. This will significantly simplify future implementations of graph transformers.\n- The authors prove (Theorem 2, Propositions 3 & 4) several new results comparing the expressive power of different positional encodings, e.g. SPE >= LPE, RRWP. \n- The authors provide several experiments comparing different choices of positional encodings, including experiments testing few-shot generalization and size generalization. This and the previous result can help practitioners choose among the different options of positional encodings.\n\n# Weaknesses\n\n- There is a disconnect between the theoretical results and the experiments. The theoretical results prove that GDTransformer matches the expressive power of GDGraphormer (Theorem 1) and prove some PEs are more expressive than others (Theorem 2, Propositions 3 & 4). The experiments in Table 1 does compare different positional encodings so validates Theorem 2 and Propositions 3 & 4; however, I am not sure how the experiments in Figure 3 connect to the theoretical results. These experiments test the generalization capabilities of GDTransformer, but I am not sure how this fits into the larger message of the paper. Is there a reason why GDTransformer would generalize better than GDGraphormer or other graph transformer or GNN models? Conversely, there is no experimental validation of Theorem 1.  \nTo the this point, I feel like the biggest weakness of the paper is that there is no comparison between GDTransformer and existing graph transformer models. I feel like such a comparison is necessary to the message of the paper. I see these experiments as serving two purposes.\n    - Theorem 1 shows that GDTransformer has the same expressive power as GDGraphormer while being simpler. Therefore, I feel like there should be an experimental comparison between GDTransformer and GDGraphormer to empirically validate this theorem. Potentially they could be compared on the BREC dataset [Wang and Zhang, 2024], which tests the expressive power of a GNN.\n    - Figure 3 tests the generalization abilities of GDTransformer. However, there is no point of comparison for generalization, so it is unclear how unique GDTransfomers generalization capabilities are.\n- While the authors compare some of the most prominent positional encodings, there are many others that are not included in their comparison, most notably the shortest-path or resistance distances used in GD-Graphormer, which is perhaps the most similar architecture to GDTransformer in the literature.\n\nQUESTIONS:\n## Suggestions\n\n- I feel the term \"distance\" is misused in this paper, as it is applied to functions that are not metrics by the mathematical definition. For example, line 109 uses $d_G$ for the adjacency function, which is not a metric. Thus, it feels like a misnomer to call the proposed architecture GDTransformer, when it can use any relative positional encoding, not just distances.\n\n- The name \"GDTransformer\" is very similar to GDGraphormer introduced in [Zhang et al, 2023]. It might be a good idea to change the name to avoid confusion.\n\n- One of the key selling points of this paper is that GDTransformer is a simplificiation of previous graph transformer architectures while maintaining the same theoretical expressive power. However, there is little dicussion of previous graph transformer architectures, so the novelty of GDTransformer may not be obvious to readers. I would suggest adding some discussion comparing GDTransformer to the existing graph transformer architectures it improves on.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper starts with introducing GDT, a generalization of recent graph transformer architectures. It then proves that GDTs can be designed and parameterized to simulate the GD-WL algorithm in general. Importantly, in this proof, the paper works with full softmax attention. Using this framework, it evaluates the relative expressiveness of position embeddings used in recent graph transformers. Finally, empirical results are provided to evaluate the applicability of the developed theory.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Being able to rigorously show that a graph transforner with softmax attention is able to emulate GD-WL is a big contribution.\n    \n- Since the goal of this work is to takl about expressibility of different position embeddings, and not necessarily their “optimizability”, the choice of restricting experimental evaluation to only large graphs was a good idea.\n    \n\n### Weaknesses\n\n- One real-world dataset per task-type (graph/node/edge classificaiton), seems a little low.\n    \n- While the theory predicts SPE is atleast as expressive as LPE, the results in Table 1 show that the performance of SPE is much worse than LPE. How would you explain that?\n    \n- Since all 3 real-world datasets considered have a similar number of tokens, what causes RRWP to be OOT on COCO and Code? Also, what would be the estimated training time for RRWP on COCO and Code? If getting results for RRWP on these datasets is feasible, I would suggest including the results and noting down the extra compute time needed for it in the paper.  \n    Excluding a method based on the inefficiency of current implementation seems a bit wrong. After all, transformers is also slow in theory, but once researchers designed fast attention kernels, using attention became feasible, even for large datasets.\n\nQUESTIONS:\n- And I correct in assuming that GDT always assumes the existence of edge features. And if they are not present in the dataset then these are assigned to 1/0 based on the adjacency?\n\nLIMITATIONS:\nYes, limitations have been properly addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Generalized-Distance Transformer (GDT), a new graph transformer (GT) architecture designed to bridge the gap between theoretical expressiveness and practical performance across diverse domains. The authors aim to provide generalizable insights for the design and application of GTs by leveraging standard attention mechanisms alongside advanced positional embeddings (PEs). Experiments over various datasets and tasks show the effectiveness of the proposed GT.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- I am not an expert in this field. But it seems that the GDT is shown to be provably expressive. The work advances GT theory by linking standard attention-based architectures to the GD-WL algorithm.\n\n- Experiments span graph-, node-, and edge-level tasks, in- and out-of-distribution settings, and model scales (15M to 160M parameters). Datasets include both real-world benchmarks and synthetic algorithmic tasks. The results show the generalizability and effectiveness of the proposed GT.\n\nWeaknesses:\n\n- The scope of the theoretical contributions and practical applications in this paper needs to be clearly defined. In Section 2.2, the authors state: “While many variations of GTs exist, we consider the standard transformer encoder based on Vaswani et al. [2017].” However, they do not explain why other variants are excluded. Additionally, the paper focuses primarily on four types of positional embeddings (PEs). Are there other PEs available? If so, why were those not considered? This point was not sufficiently clarified.\n\n- Some experimental findings may be difficult to interpret for readers who are not specialists in the field. For example, in Section 4.3, the performance of LPE slightly declines on the MST task with the 160M-parameter model (compared to the 90M model). The paper merely states that it “still outperforms both 15M models” but does not further analyze the cause of this fluctuation. It is recommended to include additional experiments or analyses to improve the robustness of the conclusion.\n\nQUESTIONS:\n- Why do you not consider other GT variants?\n\n- How would GDT extend to dynamic graphs (e.g., temporal social networks) or heterogeneous graphs (e.g., knowledge graphs with multiple node/edge types)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Generalized-Distance Transformer (GDT), a graph transformer architecture that aims to bridge the gap between theoretical expressivity and practical performance in graph learning. Sepcifically, the authors provide comprehensive comparison among position encodings and experimental validation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper provides strong thoeretical proof to compare each of variants of GT.\n\n2. The experiments span 8 million graphs across diverse tasks and domains, including out-of-distribution reasoning. The GDT’s ability to learn transferable representations (e.g., few-shot COCO-to-PASCAL transfer) highlights its practical utility.\n\n3. The work provides practical guidelines for GT design, such as the importance of PEs and attention mechanisms, which are broadly applicable beyond the proposed architecture.\n\nWeaknesses:\n\n1. While the authors provide detailed proof. These information mostly are not surprising, and the authors basically confirm these intuitive idea. For example, insight 1 & 2 are sort of intuitive, as it seems that the authors are simply claiming biased attention and positional encodings are useful. Also, insight 3 has been widely observed in previous work, as these models' performances do not vary too much. I believe the interesting part is in insight 4 & 5.\n\n2. The rankings experimantal results do not align with the theoretical results, which means the Table 1 does not match the expressive power of positional embeddings.\n\n3. There are also some other works[1] analyzes the expressive power of graph transformer. It seems that there's overlap between [1] and Section3 in this paper. Could the authors further discuss the difference from the previous work? \n\n4. Few-shot experiments seem not sufficient. As the authors test this work on 6 benchmarks, the authors only investigate two few shot cases (Bridges -> Cycle detection & COCO -> Pascal). Could the authors conduct more experiments to provide a more convincing results?\n\n[1]What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}