Title: Enhancing Graph Transformers with Hierarchical Distance Structural Encoding
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Transformers

Graph Neural Networks

Graph Classification

Node Classification

Large Graphs

Scalability
Abstract: Graph transformers need strong inductive biases to derive meaningful attention scores. Yet, current methods often fall short in capturing longer ranges, hierarchical structures, or community structures, which are common in various graphs such as molecules, social networks, and citation networks. This paper presents a Hierarchical Distance Structural Encoding (HDSE) method to model node distances in a graph, focusing on its multi-level, hierarchical nature. We introduce a novel framework to seamlessly integrate HDSE into the attention mechanism of existing graph transformers, allowing for simultaneous application with other positional encodings. To apply graph transformers with HDSE to large-scale graphs, we further propose a high-level HDSE that effectively biases the linear transformers towards graph hierarchies. We theoretically prove the superiority of HDSE in terms of expressivity and generalization. Empirically, we demonstrate that graph transformers with HDSE excel in graph classification, regression on 7 graph-level datasets, and node classification on 11 large-scale graphs.

Enhancing Graph Transformers with Hierarchical Distance Structural Encoding

Abstract

Graph transformers need strong inductive biases to derive meaningful attention scores. Yet, current methods often fall short in capturing longer ranges, hierarchical structures, or community structures, which are common in various graphs such as molecules, social networks, and citation networks. This paper presents a Hierarchical Distance Structural Encoding (HDSE) method to model node distances in a graph, focusing on its multi-level, hierarchical nature. We introduce a novel framework to seamlessly integrate HDSE into the attention mechanism of existing graph transformers, allowing for simultaneous application with other positional encodings. To apply graph transformers with HDSE to large-scale graphs, we further propose a high-level HDSE that effectively biases the linear transformers towards graph hierarchies. We theoretically prove the superiority of HDSE in terms of expressivity and generalization. Empirically, we demonstrate that graph transformers with HDSE excel in graph classification, regression on 7 graph-level datasets, and node classification on 11 large-scale graphs.

1 Introduction

The success of Transformers [74] in various domains, including natural language processing (NLP) and computer vision [18], has sparked significant interest in developing transformers for graph data [19, 86, 46, 11, 68, 61, 89, 60, 82]. Scholars have turned their attention to this area, aiming to address the limitations of Message-Passing Graph Neural Networks (MPNNs) [30] such as oversmoothing [54] and over-squashing [1, 73].

However, Transformers [74] are known for their lack of strong inductive biases [18]. In contrast to MPNNs, graph transformers do not rely on fixed graph structure information. Instead, they compute pairwise interactions for all nodes within a graph and represent positional and structural data using more flexible, soft inductive biases. Despite its potential, this mechanism does not have the capability to learn hierarchical structures within graphs. Developing effective positional encodings is also challenging, as it requires identifying important hierarchical structures among nodes, which differ significantly from other Euclidean domains [10]. Consequently, graph transformers are prone to overfitting and often underperform MPNNs when data is limited [61], especially in tasks involving large graphs with relatively few labeled nodes [82]. These challenges become even more significant when dealing with various molecular graphs, such as those found in polymers or proteins. These graphs are characterized by a multitude of substructures and exhibit long-range and hierarchical structures. The inability of graph transformers to learn these hierarchical structures can significantly impede their performance in tasks involving such complex molecular graphs.

HDSE

key

VV23V456VggV1011

MatMul

V1

GHD2

V2

V3

V4

Scale

query

V5

MatMul

V7

Linear

Linear

Linear

V8

Q

K

4

Vg

V10

V2

V10

GHD0

V11

V3

V6

GHD°(v,V11)

GHD(v,V1)

GHD²（v,U11)

= 7

= 2

= 1

NodeFeature

PositionEncoding

Further, the global all-pair attention mechanism in transformers poses a significant challenge due to its time and space complexity, which increases quadratically with the number of nodes. This quadratic complexity significantly restricts the application of graph transformers to large graphs, as training them on graphs with millions of nodes can require substantial computational resources. Large-scale graphs, such as social networks and citation networks, often exhibit community structures characterized by closely interconnected groups with distinct hierarchical properties. To enhance the scalability and effectiveness of graph transformers, it is crucial to incorporate hierarchical structural information at various levels.

To address the aforementioned challenges and unlock the true potential of the transformer architecture in graph learning, we propose a Hierarchical Distance Structural Encoding (HDSE) method ( Sec. 3.1 ), which can be combined with various graph transformers to produce more expressive node embeddings. HDSE encodes the hierarchy distance, a metric that measures the distance between nodes in a graph, taking into account multi-level graph hierarchical structures. We utilize popular coarsening methods [43, 65, 31, 6, 57] to construct graph hierarchies, enabling us to measure the distance relationship between nodes across various hierarchical levels.

HDSE enables us to incorporate a robust inductive bias into existing transformers and address the issue of lacking canonical positioning. To achieve this, we introduce a novel framework ( Sec. 3.2 ), as illustrated in Figure 1. We utilize an end-to-end trainable function to encode HDSE as structural bias weights into the attentions, allowing the graph transformer to integrate both HDSE and other positional encodings simultaneously. Our theoretical analysis demonstrates that graph transformers equipped with HDSE are significantly more powerful than the ones with the commonly used shortest path distances or without relative positional encodings, in terms of both expressiveness and generalization . We rigorously evaluate our HDSE in ablation studies and show that it successfully improves different kinds of baseline transformers, from vanilla graph transformers [19] to state-of-the-art graph transformers [68, 11, 91, 61], across 7 graph-level datasets.

To enable the application of graph transformers with HDSE to large graphs ranging from millions to billions of nodes, we introduce a high-level HDSE ( Sec. 3.3 ), which effectively biases the linear transformers towards the multi-level structural nature of these large networks. We demonstrate our high-level HDSE method exhibits high efficiency and quality across 11 large-scale node classification datasets, with sizes up to the billion-node level.

Our implementation is available at https://github.com/LUOyk1999/HDSE.

2 Background and Related Works

We refer to a graph as a tuple G = ( V, E, X ) , with node set V , edge set E ⊆ V × V , and node features X ∈ R | V |× d . Each row in X represents the feature vector of a node, with | V | denoting the number of nodes and feature dimension d . The features of node v are denoted by x v ∈ R d .

2.1 Graph Hierarchies

Given an input graph G , a graph hierarchy of G consists of a sequence of graphs ( G k , ϕ k ) k ≥ 0 , where G 0 = G and ϕ k : V k → V k +1 are surjective node mapping functions. Each node v k +1 j ∈ V k +1 represents a cluster of a subset of nodes { v k i } ⊆ V k . This partition can be described by a projection matrix ˆ P k ∈ { 0 , 1 } | V k |×| V k +1 | , where ˆ P k ij = 1 if and only if v k +1 j = ϕ k ( v k i ) . The normalized version can be defined by P k = ˆ P k C k -1 / 2 , where C k ∈ R | V k +1 |×| V k +1 | is a diagonal matrix with its j -th diagonal entry being the cluster size of v k +1 j . We define the node feature matrix X k +1 for G k +1 by X k +1 = P k ⊤ X k , where each row of X k +1 represents the average of all entries within a specific cluster . The edge set E k +1 of G k +1 is defined as E k +1 = { ( u k +1 , v k +1 ) |∃ v k r ∈ ϕ -1 k ( u k +1 ) , v k s ∈ ϕ -1 k ( v k +1 ) , such that ( v k r , v k s ) ∈ E k } .

Graph hierarchies can be constructed by repeatedly applying graph coarsening algorithms. These algorithms take a graph, G k , and generate a mapping function ϕ k : V k → V k +1 , which maps the nodes in G k to the nodes in the coarser graph G k +1 . A summary and comparison of popular graph coarsening algorithms, along with their computational complexities, can be found in Table 1. We define the coarsening ratio as α = | V k +1 | | V k | , which represents the proportion of the number of nodes in the coarser graph G k +1 to the number of nodes in the original graph G k . Consequently, each graph G k , where k > 0 , captures specific substructures derived from the preceding graph.

2.2 Graph Transformers

Transformers [74] have recently gained significant attention in graph learning, due to their ability to learn intricate relationships that extend beyond the capabilities of conventional GNNs [44, 33, 59, 58], and in a unique way. The architecture of these models primarily contain a self-attention module and a feed-forward neural network, each of which is followed by a residual connection paired with a normalization layer. Formally, the self-attention mechanism involves projecting the input node features X using three distinct matrices W Q ∈ R d × d ′ , W K ∈ R d × d ′ and W V ∈ R d × d ′ , resulting in the representations for query ( Q ), key ( K ), and value ( V ), which are then used to compute the output features described as follows:

Technically, transformers can be considered as message-passing GNNs operating on fully-connected graphs, decoupled from the input graphs. The main research question in the context of graph transformers is how to incorporate the structural bias of the given input graphs by adapting the attention mechanism or by augmenting the original features. The Graph Transformer (GT) [19] represents an early work using Laplacian eigenvectors as positional encoding (PE), and various extensions and alternative models have been proposed since then [64]. For instance, the structureaware transformer (SAT) [11] extracts a subgraph representation rooted at each node before computing the attention. Most initial works in the area focus on the classification of smaller graphs, such as molecules; yet, recently, GraphGPS [68] and follow-up works [92, 81, 80, 82, 12, 45, 72, 16, 26] also consider larger graphs.

2.3 Hierarchy in Graph Learning

In message passing GNNs, hierarchical pooling of node representations is a proven method for incorporating coarsening into reasoning [5, 28, 87, 48, 36, 69]. With GNNs, coarsened graph

representations are further considered in the context of molecules [42] and virtual nodes [39]. Additionally, HTAK [3] employ graph hierarchies to develop a novel graph kernel by transitively aligning the nodes across multi-level hierarchical graphs. The recent HC-GNN [94] demonstrates competitive performance in node classification on large-scale graphs, utilizing hierarchical community structures for message passing.

In graph transformers, there are currently only a few hierarchical models. ANS-GT [91] use adaptive node sampling in their graph transformer, enabling it for large-scale graphs and capturing long-range dependencies. Their model groups nodes into super-nodes and allows for interactions between them. Similarly, HSGT [96] aggregates multi-level graph information and employs graph hierarchical structure to construct intra-level and inter-level transformer blocks. The intra-level block facilitates the exchange and transformation of information within the local context of each node, while the interlevel block adaptively coalesces every substructure present. Our concurrent work directly incorporates hierarchy into the attention, a fundamental building block of the transformer architecture, making it flexible and applicable to existing graph transformers. Additionally, Coarformer [47] utilizes graph coarsening techniques to generate coarse views of the original graph, which are subsequently used as input for the transformer model. Likewise, PatchGT [27] starts by segmenting graphs into patches using spectral clustering and then learns from these non-trainable graph patches. MGT [66] learns atomic representations and groups them into meaningful clusters, which are then fed to a transformer encoder to calculate the graph representation. However, these approaches typically yield coarse-level representations that lack comprehensive awareness of the original node-level features [41]. In contrast, our model integrates hierarchical information from a broader distance perspective, thereby avoiding the oversimplification in these coarse-level representations.

3 Our Method

3.1 Hierarchical Distance Structural Encoding (HDSE)

Firstly, we introduce a novel concept called graph hierarchy distance (GHD), which is defined as follows.

Definition 1 ( Graph Hierarchy Distance ) . Given two nodes u, v in graph G , and a graph hierarchy ( G i , ϕ i ) i ≥ 0 , the k -level hierarchy distance between u and v is defined as

where SPD( · , · ) is the shortest path distance between two nodes ( ∞ if the nodes are not connected), and ϕ k -1 ...ϕ 0 ( · ) maps a node in G 0 to a node in G k .

Note that the k -level hierarchy distance adheres to the definition of a metric, being zero for v = u , invariably positive, symmetric, and fulfilling the triangle inequality. As illustrated on the left side of Figure 1, it can be observed that GHD 0 ( v 1 , v 11 ) = 7 , whereas GHD 1 ( v 1 , v 11 ) = 2 .

Graph hierarchies have been observed to assist in identifying community structures in graphs that exhibit a clear property of tightly knit groups, such as social networks and citation networks [31]. They may also aid in prediction over graphs with a clear hierarchical structure, such as metal-organic frameworks or proteins. Fig. 2 shows that with the graph hierarchies generated by the Newman coarsening method, GHD 1 is capable of capturing chemical motifs, including CF3 and aromatic rings.

Based on GHD, we propose hierarchical distance structural encoding (HDSE), defined for each pair of nodes i, j ∈ V :

where GHD k is the k -level hierarchy distance matrix, and K ∈ N controls the maximum level of hierarchy considered.

We demonstrate the superior expressiveness of HDSE over SPD using recently proposed graph isomorphism tests inspired by the Weisfeiler-Leman algorithm [78]. In particular, [89] introduced the Generalized Distance Weisfeiler-Leman (GD-WL) Test and applied it to analyze a graph transformer architecture that employs SPD( i, j ) as relative positional encodings. They proved that the graph

GHD

𝟏

𝐂𝐨𝐚𝐫𝐬𝐞𝐧𝐢𝐧𝐠	𝐫𝐞𝐬𝐮𝐥𝐭

𝐌𝐨𝐥𝐞𝐜𝐮𝐥𝐞	𝐠𝐫𝐚𝐩𝐡

GHD

𝟏

𝐂𝐨𝐚𝐫𝐬𝐞𝐧𝐢𝐧𝐠	𝐫𝐞𝐬𝐮𝐥𝐭

𝐌𝐨𝐥𝐞𝐜𝐮𝐥𝐞	𝐠𝐫𝐚𝐩𝐡

GHD

!

GHD

"

GHD

"

GHD

!

!"#$%&'$#(")	+(&,'

!$-&(+.$-	+(&,'

Q-

2.5

3.0

2.00

1.75

10

2.0

1.5

10

10

1.00

1.50

1.25

15

1.0

0.75

20

10

15

20

0.0

1

1

15

10

15

15

10

15

050

0.25

0.00

0

1.75

1.50

10

1.25

15

10

10

1.00

0.75

20

C

0.50

25

0.25

10

1520

25

10

15

10

15

0.00

1

transformer's maximum expressiveness is the GD-WL test with SPD. Here, we also use the GD-WL test to showcase the expressiveness of HDSE.

Proposition 1 ( Expressiveness of HDSE ) . GD-WL with HDSE (D i,j ) is strictly more expressive than GD-WL with the shortest path distance SPD( i, j ) .

The proof is provided in Appendix 4. Firstly, we show that the GD-WL test using HDSE can differentiate between any two graphs that can be distinguished by the GD-WL test with SPD. Next, we show that the GD-WL test with HDSE is capable of distinguishing the Dodecahedron and Desargues graphs (Figure 2) while the one with SPD cannot.

3.2 Integrating HDSE in Graph Transformers

We integrate HDSE ( D i,j ) into the attention mechanism of each graph transformer layer to bias each node update. To achieve this, we use an end-to-end trainable function Bias : R K +1 → R to learn the biased structure weight H i,j = Bias(D i,j ) . We limit the maximum distance length to a value L , based on the hypothesis that detailed information loses significance beyond a certain distance. By imposing this limit, the model can extend acquired patterns to graphs of varying sizes not encountered in training. Specificially, we implement the function Bias using an MLP as follows:

where [ e k 0 , e k 1 , · · · , e k L ] 0 ≤ k ≤ K ∈ R d × ( L +1) collects L +1 learnable feature embedding vectors e k i for hierarchy level k . By embedding the hierarchy distances at different levels into learnable feature vectors, it may enhance the aggregation of multi-level graph information among nodes and expands the receptive field of nodes to a larger scale. We assume single-headed attention for simplified notation, but when extended to multiheaded attention, one bias is learned per distance per head.

We incorporate the learned biased structure weights H to graph transformers, using the popular biased self-attention method proposed by [86], formulated as:

where the original attention score A is directly augmented with H . This approach is backbone-agnostic and can be seamlessly integrated into the self-attention mechanism of existing graph transformer architectures. Notably, we have the following results on expressiveness and generalization.

Proposition 2. The power of a graph transformer with HDSE to distinguish non-isomorphic graphs is at most equivalent to that of the GD-WL test with HDSE. With proper parameters and an adequate number of heads and layers, a graph transformer with HDSE can match the power of the GD-WL test with HDSE.

See the proof in Appendix 5. This result provides a precise characterization of the expressivity power and limitations of graph transformers with HDSE. Combining Proposition 1, 2 and Proofs A.1 in [86] immediately yields the following corollary:

Corollary 1 ( Expressiveness of Graph Transformers with HDSE ) . There exists a graph transformer using HDSE (with fixed parameters), denoted as M , such that M is more expressive than graph transformers with the same architecture using SPD or using no relative positional encoding, regardless of their parameters.

This is a fine-grained expressiveness result of graph transformers with HDSE. It demonstrate the superior expressiveness of HDSE over SPD in graph transformers.

Proposition 3 ( Generalization of Graph Transformers with HDSE ) . (Informal) For a semisupervised binary node classification problem, suppose the label of each node i ∈ V is determined by node features in the ' hierarchical core neighborhood ' S i ∗ = { j : D i,j = D ∗ } for a certain D ∗ , where D i,j is HDSE defined in (3). Then, a properly initialized one-layer graph transformer equipped with HDSE can learn such graphs with a desired generalization error, while using SPD or using no relative positional encoding cannot guarantee satisfactory generalization.

The formal version and the proof are given in Appendix 6. Proposition 3 is a corollary and extension of Theorem 4.1 of [53] from SPD to HDSE. It indicates that learning with HDSE can capture the labeling function characterized by the hierarchical core neighborhood, which is more general and comprehensive than the core neighborhood for SPD proposed in [53].

3.3 Scaling HDSE to Large-scale Graphs

For large graphs with millions of nodes, training and deploying a transformer with full global attention is impractical due to the quadratic cost. Some existing graph transformers address this issue by sampling nodes for attention computation [91, 96]. While our HDSE can enhance these models, this sampling approach compromises the expressivity needed to capture interactions among all pairs of nodes. However, in the NLP domain, Linformer [77] utilizes a learnable low-rank projection matrix ˜ P to reduce the complexity of the self-attention module to linear levels:

Inspired by Linformer, models like GOAT [45] and Gapformer [56] in the graph domain also employ projection matrices to reduce the number of nodes by projecting them onto fewer supernodes, consequently compressing the input node feature matrix X into a smaller dimension. This transformation enables the aggregation of information from super-nodes and reduces the quadratic complexity of attention computation to linear complexity. Here, we can replace the projection matrix with our coarsening partition matrix P (see Sec. 2.1) to obtain representations of coarser graphs at higher levels. Observe that we can still calculate meaningful distances at these higher hierarchy levels, using a high-level HDSE as follow:

where each row of the projection matrix P l (see Sec. 2.1) is a one-hot vector representing the l -level cluster that an input node belongs to, and D c ∈ R | V 0 |×| V c |× ( K +1 -c ) . Note that GHD m ( ∏ c -1 l =0 P l ) , c ≤ m ≤ K computes distances from input nodes to clusters at c -level graph hierarchy (see App. A). In practice, these distances can be directly obtained by calculating the hierarchy distance between all node pairs at the c -level. When c = 0 , D c becomes D in Eq 3. In this way, the high-level HDSE establishes attention between nodes in the input graph G and clusters at high level hierarchies. For example, we can integrate the high-level HDSE into Linformer by adapting Equation (6):

where X k ∈ R | V k |× d (see Sec. 2.1) represents the features of clusters at k -level, and Bias : R K +1 -k ↦→ R is a end-to-end trainable function as defined in Sec. 3.2.

4 Evaluation

We evaluate our proposed HDSE on 18 benchmark datasets, and show state-of-the-art performance in many cases. Primarily, the following questions are investigated:

Can HDSE improve upon existing graph transformers , and how does the choice of coarsening algorithm affect performance? ( Sec. 4.2 )

Does our adaptation for large graphs also show effectiveness , is it marked by efficiency , and how does high-level HDSE impact the performance? ( Sec. 4.3 )

4.1 Experiment Setting

Datasets. We consider various graph learning tasks from popular benchmarks as detailed below and in Appendix B.

Graph-level Tasks. For graph classification and regression, we evaluate our method on five datasets from Benchmarking GNNs [20]: ZINC, MNIST, CIFAR10, PATTERN, and CLUSTER. We also test on two peptide graph benchmarks from the Long-Range Graph Benchmark [23]: Peptides-func and Peptides-struct, focusing on classifying graphs into 10 functional classes and regressing 11 structural properties, respectively. We follow all evaluation protocols suggested by [68].

Node Classification on Large-scale Graphs. We consider node classification over the citation graphs Cora, CiteSeer and PubMed [44], the Actor co-occurrence graph [14], and the Squirrel and Chameleon page-page networks from [71], all of which have 1K-20K nodes. Further, we extend our evaluation to larger datasets from the Open Graph Benchmark (OGB) [35]: ogbn-arxiv, arxiv-year, ogbn-papers100M, ogbn-proteins and ogbn-products, with node numbers ranging from 0.16M to 0.1B. We maintain all the experimental settings as described in [82].

Baselines. We compare our method to the following prevalent GNNs: GCN [44], GIN [84], GAT [75], GatedGCN [9], GatedGCN-RWSE [22], JKNet [85], APPNP [29], PNA [15], GPRGNN [14], SIGN [70], H2GCN [95]; and other recent GNNs with SOTA performance: LINKX [55], CIN [8], GIN-AK+ [93], HC-GNN [94]. In terms of transformer models, we consider GT [19], Graphormer [86], SAN [46], Coarformer [47], ANS-GT [91], EGT [38], NodeFormer [81], Specformer [7], MGT [66], AGT [62], HSGT [96], Graphormer-GD [89], SAT [11], GOAT [45], Gapformer [56], Graph ViT/MLP-Mixer [34], LargeGT [21], NAGphormer [12], Exphormer [72], DRew [32], VCR-GT [26], CoBFormer [83] and recent SOTA graph transformers GraphGPS [68], GRIT [61], SGFormer [82].

Models + HDSE. We integrate HDSE into GT, SAT, GraphGPS, GRIT (and ANS-GT in appendix) only modifying their self-attention module by Eq. 5. For fair comparisons, we use the same hyperparameters (including the number of layers, hidden dimension etc.), PE and readout as the baseline transformers. Given one of the baseline transformers M , we denote the modified model using HDSE by M+HDSE . Additionally, we integrate our high-level HDSE into GOAT , denoted as GOAT + HDSE . We fix the maximum distance length L = 30 and vary the maximum hierarchy level K in { 0 , 1 , 2 } in all experiments. A sensitivity analysis of these two parameters is presented in Appendix C. During training, the steps of coarsening and distance calculation [17] can be treated as pre-processing, since they only need to be run once. We detail the choice and runtime of coarsening algorithms for HDSE in the appendix. Detailed experimental setup and hyperparameters are in Appendix B due to space constraints.

4.2 Results on Graph-level Tasks

Benchmarks from Benchmarking GNNs, Table 2. We observe that nearly all four baseline graph transformers, when combined with HDSE, demonstrate performance improvements. Note that the enhancement is overall especially considerable for GT. On CIFAR10, we also obtain similar improvement for GraphGPS. Among them, GT shows the greatest enhancement and becomes competitive to more complex models. Our model attains the best or second-best mean performance for all datasets. While the improvement for GRIT is smaller, as its relative random walk probabilities (RRWP) already incorporate distance information [61], we still observe improvements in three datasets. This indicates that HDSE can provide additional information beyond what is captured by RRWP. Notably, it is observed that the SOTA SGFormer tailored for large-scale node classification underperforms in graph-level tasks.

Long-Range Graph Benchmark, Table 3. We consider GraphGPS due to its superior performance. Note that our HDSE module only introduces a small number of additional parameters, allowing it to remain within the benchmark's ∼ 500k model parameter budget. In the Peptides-func dataset, HDSE yields a significant improvement of 6.21%. This is a promising result and hints at potentially great benefits for macromolecular data more generally.

Ablation Study and Visualization, Table 4, 13, 16, Figure 3, 5. First, we conduct several ablation experiments of coarsening algorithms on ZINC and observe that the dependency on the coarsening varies with the transformer backbone. For instance, the multi-level graph structures extracted by the Newman algorithm yields the largest improvement for GraphGPS. More generally, our experiments indicate that Newman works best for molecular graphs. We visualize the attention scores on the ZINC and Peptides-func datasets respectively, as shown in Figure 3. The results indicate that our HDSE method successfully leverages hierarchical structure.

We also conduct a sensitivity analysis on maximal hierarchy level K and maximum distance length L in Appendix C. The variation in the optimal K and L could stem from the distinct hierarchical structures inherent in different graphs.

Molecular graph

Transformer attn

HDSE attn

0.08

0.35

!

0.07

V

0.30

0.06

0.25

0.05

0.20

0.04

0.15

0.03

0.10

0.02

0.05

0.01

10.00

0.00

Peptides graph

Transformer attn

HDSE attn

0.14

0.020

0.12

0.10

0.015

0.08

0.06

0.010

0.04

0.005

0.02

10.00

0.000

Synthetic Community Graphs, Table 19. We evaluate our methods on the community datasets from [88], generated using the Erdos-Renyi model [24]. These graphs have adjacency matrices that obey the certain clustered structure. As evidenced in Table 19, the GT struggles to detect such structures; and solely utilizing SPD proves inadequate; however, our HDSE, enriched with coarsening structural information, effectively captures these structures.

4.3 Results on Large-scale Graphs

Overall Performance, Table 5. We select four categories of baselines: GNNs, graph transformers with proven performance on graph-level tasks, graph transformers with hierarchy, and scalable graph transformers. It is noteworthy that while some graph transformers exhibit superior performance on graph-level tasks, they consistently result in out-of-memory (OOM) in large-scale node tasks. The results are remarkably consistent. In relatively smaller datasets graphs (on the left side), the integra-

tion of high-level HDSE enables GOAT to demonstrate competitive performance among baseline models. This could be due to the coarsening projection filtering out the edges from neighboring nodes of different categories and providing a global perspective enriched with multi-level structural information. For all larger graphs (on the right side), our high-level HDSE method significantly enhances GOAT's performance beyond its original version. This indicates that the structural bias provided by graph hierarchies is capable of handling the node classification task in such larger graphs and effectively retains global information. We investigated this in more detail in our ablation experiments. Furthermore, we also observed that all graph transformers with hierarchy suffer from serious overfitting, attributed to their relatively complex architectures. In contrast, our model's simple architecture contributes to its better generalization.

Efficiency Comparison, Table 6. We report the efficiency results on PubMed, ogbn-proteins, ogbnarxiv, ogbn-products and ogbn-100M. It is easy to see that our model outperforms NodeFormer in speed, matching the pace of the latest and fastest model, SGFormer [82]. It achieves true linear complexity with a streamlined architecture.

Ablation Study, Table 7. To determine the utility of our architectural design choices, we conduct ablation experiments on GOAT + HDSE over three datasets. The results presented in Table 7, include (1) removing the high-level HDSE and (2) replacing the coarsening projection matrix with the original projection matrix used in GOAT. These experiments reveal a decline in all performance, thereby validating the effectiveness of our architectural design.

5 Conclusions

We have introduced the Hierarchical Distance Structural Encoding (HDSE) method to enhance the capabilities of transformer architectures in graph learning tasks. We have developed a flexible framework to integrate HDSE with various graph transformers. Further, for applying graph transformers with HDSE to large-scale graphs, we have introduced a high-level HDSE approach that effectively biases linear transformers towards the multi-level structure. Theoretical analysis and empirical results validate the effectiveness and generalization capabilities of HDSE, demonstrating its potential for various real-world applications.

Acknowledgments and Disclosure of Funding

We extend our sincere gratitude to Veronika Thost, Yicheng Pan, Zixu Zhao and Jaan Li for their invaluable guidance in our experiments. We also express our appreciation to all the anonymous reviewers and ACs for their insightful and constructive feedback. This work received support from National Key R&D Program of China (2021YFB3500700), NSFC Grant 62172026, National Social Science Fund of China 22&ZD153, the Fundamental Research Funds for the Central Universities, State Key Laboratory of Complex & Critical Software Environment (CCSE), HK PolyU Grant P0051029, HK PolyU Grant P0038850, and HK ITF Grant ITS/359/21FP. Lei Shi is with Beihang University and State Key Laboratory of Complex & Critical Software Environment.

References

Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. arXiv preprint arXiv:2006.05205 , 2020.

Davoud Ataee Tarzanagh, Yingcong Li, Xuechen Zhang, and Samet Oymak. Max-margin token selection in attention mechanism. Advances in Neural Information Processing Systems , 36:48314-48362, 2023.

Lu Bai, Lixin Cui, and Hancock Edwin. A hierarchical transitive-aligned graph kernel for un-attributed graphs. In International Conference on Machine Learning , pages 1327-1336. PMLR, 2022.

Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. arXiv preprint arXiv:2110.02910 , 2021.

Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling. In International conference on machine learning , pages 874-883. PMLR, 2020.

Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment , 2008(10):P10008, 2008.

Deyu Bo, Chuan Shi, Lele Wang, and Renjie Liao. Specformer: Spectral graph neural networks meet transformers. arXiv preprint arXiv:2303.01028 , 2023.

Cristian Bodnar, Fabrizio Frasca, Yuguang Wang, Nina Otter, Guido F Montufar, Pietro Lio, and Michael Bronstein. Weisfeiler and lehman go topological: Message passing simplicial networks. In International Conference on Machine Learning , pages 1026-1037. PMLR, 2021.

Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiv:1711.07553 , 2017.

Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 , 2021.

Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In International Conference on Machine Learning , pages 3469-3489. PMLR, 2022.

Jinsong Chen, Kaiyuan Gao, Gaichao Li, and Kun He. Nagphormer: A tokenized graph transformer for node classification in large graphs. In The Eleventh International Conference on Learning Representations , 2022.

Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Unveiling induction heads: Provable training dynamics and feature learning in transformers. arXiv preprint arXiv:2409.10559 , 2024.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations , 2020.

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veliˇ ckovi´ c. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems , 33:13260-13271, 2020.

Chenhui Deng, Zichao Yue, and Zhiru Zhang. Polynormer: Polynomial-expressive graph transformer in linear time. In The Twelfth International Conference on Learning Representations , 2024.

EW Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik , 1(1):269-271, 1959.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699 , 2020.

Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research , 24(43):1-48, 2023.

Vijay Prakash Dwivedi, Yozen Liu, Anh Tuan Luu, Xavier Bresson, Neil Shah, and Tong Zhao. Graph transformers for large graphs. arXiv preprint arXiv:2312.11109 , 2023.

Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations , 2021.

Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. arXiv preprint arXiv:2206.08164 , 2022.

P ERDdS and A R&wi. On random graphs i. Publ. math. debrecen , 6(290-297):18, 1959.

Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428 , 2019.

Dongqi Fu, Zhigang Hua, Yan Xie, Jin Fang, Si Zhang, Kaan Sancak, Hao Wu, Andrey Malevich, Jingrui He, and Bo Long. VCR-graphormer: A mini-batch graph transformer via virtual connections. In The Twelfth International Conference on Learning Representations , 2024.

Han Gao, Xu Han, Jiaoyang Huang, Jian-Xun Wang, and Liping Liu. Patchgt: Transformer over non-trainable clusters for learning graph representations. In Learning on Graphs Conference , pages 27-1. PMLR, 2022.

Hongyang Gao and Shuiwang Ji. Graph u-nets. In international conference on machine learning , pages 2083-2092. PMLR, 2019.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997 , 2018.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Michelle Girvan and Mark EJ Newman. Community structure in social and biological networks. Proceedings of the national academy of sciences , 99(12):7821-7826, 2002.

Benjamin Gutteridge, Xiaowen Dong, Michael M Bronstein, and Francesco Di Giovanni. Drew: Dynamically rewired message passing with delay. In International Conference on Machine Learning , pages 12252-12267. PMLR, 2023.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Xiaoxin He, Bryan Hooi, Thomas Laurent, Adam Perold, Yann LeCun, and Xavier Bresson. A generalization of vit/mlp-mixer to graphs. In International Conference on Machine Learning , pages 12724-12745. PMLR, 2023.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Jingjia Huang, Zhangheng Li, Nannan Li, Shan Liu, and Ge Li. Attpool: Towards hierarchical feature representation in graph convolutional networks via attention mechanism. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6480-6489, 2019.

Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. In Forty-first International Conference on Machine Learning , 2024.

Md Shamim Hussain, Mohammed J Zaki, and Dharmashankar Subramanian. Global selfattention as a replacement for graph convolution. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 655-665, 2022.

EunJeong Hwang, Veronika Thost, Shib Sankar Dasgupta, and Tengfei Ma. An analysis of virtual nodes in graph neural networks for link prediction (extended abstract). In The First Learning on Graphs Conference , 2022.

Samy Jelassi, Michael Sander, and Yuanzhi Li. Vision transformers provably learn spatial structure. Advances in Neural Information Processing Systems , 35:37822-37836, 2022.

Bo Jiang, Fei Xu, Ziyan Zhang, Jin Tang, and Feiping Nie. Agformer: Efficient graph representation with anchor-graph transformer. arXiv preprint arXiv:2305.07521 , 2023.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning , pages 48394848. PMLR, 2020.

George Karypis and Vipin Kumar. A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices. University of Minnesota, Department of Computer Science and Engineering, Army HPC Research Center, Minneapolis, MN , 38:7-1, 1998.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Kezhi Kong, Jiuhai Chen, John Kirchenbauer, Renkun Ni, C. Bayan Bruss, and Tom Goldstein. GOAT: A global transformer on large-scale graphs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 17375-17390. PMLR, 23-29 Jul 2023.

Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems , 34:21618-21629, 2021.

Weirui Kuang, WANG Zhen, Yaliang Li, Zhewei Wei, and Bolin Ding. Coarformer: Transformer for large graph via graph coarsening. 2021.

Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In International conference on machine learning , pages 3734-3743. PMLR, 2019.

Jure Leskovec and Andrej Krevl. Snap datasets: Stanford large network dataset collection. 2014. 2016.

Hongkang Li, Meng Wang, Sijia Liu, and Pin-Yu Chen. A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity. In The Eleventh International Conference on Learning Representations , 2023.

Hongkang Li, Meng Wang, Songtao Lu, Xiaodong Cui, and Pin-Yu Chen. Training nonlinear transformers for chain-of-thought inference: A theoretical generalization analysis. arXiv preprint arXiv:2410.02167 , 2024.

Hongkang Li, Meng Wang, Songtao Lu, Hui Wan, Xiaodong Cui, and Pin-Yu Chen. Transformers as multi-task feature selectors: Generalization analysis of in-context learning. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning , 2023.

Hongkang Li, Meng Wang, Tengfei Ma, Sijia Liu, ZAIXI ZHANG, and Pin-Yu Chen. What improves the generalization of graph transformers? a theoretical dive into the self-attention and positional encoding. In Forty-first International Conference on Machine Learning , 2024.

Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. arXiv preprint arXiv:1803.03324 , 2018.

Derek Lim, Felix Matthew Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Prasad Bhalerao, and Ser-Nam Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Chuang Liu, Yibing Zhan, Xueqi Ma, Liang Ding, Dapeng Tao, Jia Wu, and Wenbin Hu. Gapformer: Graph transformer with graph pooling for node classification. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI , pages 2196-2205, 2023.

Andreas Loukas. Graph reduction with spectral and cut guarantees. J. Mach. Learn. Res. , 20(116):1-42, 2019.

Yuankai Luo, Lei Shi, and Veronika Thost. Improving self-supervised molecular representation learning using persistent homology. Advances in Neural Information Processing Systems , 36, 2024.

Yuankai Luo, Lei Shi, Mufan Xu, Yuwen Ji, Fengli Xiao, Chunming Hu, and Zhiguang Shan. Impact-oriented contextual scholar profiling using self-citation graphs. arXiv preprint arXiv:2304.12217 , 2023.

Yuankai Luo, Veronika Thost, and Lei Shi. Transformers over directed acyclic graphs. Advances in Neural Information Processing Systems , 36, 2024.

Liheng Ma, Chen Lin, Derek Lim, Adriana Romero-Soriano, Puneet K Dokania, Mark Coates, Philip Torr, and Ser-Nam Lim. Graph inductive biases in transformers without message passing. arXiv preprint arXiv:2305.17589 , 2023.

Xiaojun Ma, Qin Chen, Yi Wu, Guojie Song, Liang Wang, and Bo Zheng. Rethinking structural encodings: Adaptive graph transformer for node classification task. In Proceedings of the ACM Web Conference 2023 , pages 533-544, 2023.

Julian McAuley, Rahul Pandey, and Jure Leskovec. Inferring networks of substitutable and complementary products. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 785-794, 2015.

Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, and Yu Rong. Transformer for graphs: An overview from architecture perspective. arXiv preprint arXiv:2202.08455 , 2022.

Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems , 14, 2001.

Nhat Khang Ngo, Truong Son Hy, and Risi Kondor. Multiresolution graph transformers and wavelet positional encoding for learning long-range and hierarchical structures. The Journal of Chemical Physics , 159(3), 2023.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. In International Conference on Learning Representations , 2019.

Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. arXiv preprint arXiv:2205.12454 , 2022.

Ekagra Ranjan, Soumya Sanyal, and Partha Talukdar. Asap: Adaptive structure aware pooling for learning hierarchical graph representations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5470-5477, 2020.

Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael Bronstein, and Federico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.11198 , 7:15, 2020.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks , 9(2):cnab014, 2021.

Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J Sutherland, and Ali Kemal Sinop. Exphormer: Sparse transformers for graphs. arXiv preprint arXiv:2303.06147 , 2023.

Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. arXiv preprint arXiv:2111.14522 , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, et al. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315 , 2019.

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020.

Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. nti, Series , 2(9):12-16, 1968.

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning , pages 6861-6871. PMLR, 2019.

Qitian Wu, Chenxiao Yang, Wentao Zhao, Yixuan He, David Wipf, and Junchi Yan. DIFFormer: Scalable (graph) transformers induced by energy constrained diffusion. In The Eleventh International Conference on Learning Representations , 2023.

Qitian Wu, Wentao Zhao, Zenan Li, David P Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. Advances in Neural Information Processing Systems , 35:27387-27401, 2022.

Qitian Wu, Wentao Zhao, Chenxiao Yang, Hengrui Zhang, Fan Nie, Haitian Jiang, Yatao Bian, and Junchi Yan. Simplifying and empowering transformers for large-graph representations. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Yujie Xing, Xiao Wang, Yibo Li, Hai Huang, and Chuan Shi. Less is more: on the overglobalizing problem in graph transformers. arXiv preprint arXiv:2405.01102 , 2024.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning , pages 5453-5462. PMLR, 2018.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems , 34:28877-28888, 2021.

Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems , 31, 2018.

Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. In International conference on machine learning , pages 5708-5717. PMLR, 2018.

Bohang Zhang, Shengjie Luo, Liwei Wang, and Di He. Rethinking the expressive power of GNNs via graph biconnectivity. In The Eleventh International Conference on Learning Representations , 2023.

Yihua Zhang, Hongkang Li, Yuguang Yao, Aochuan Chen, Shuai Zhang, Pin-Yu Chen, Meng Wang, and Sijia Liu. Visual prompting reimagined: The power of activation prompts, 2024.

Zaixi Zhang, Qi Liu, Qingyong Hu, and Chee-Kong Lee. Hierarchical graph transformer with adaptive node sampling. Advances in Neural Information Processing Systems , 35:21171-21183, 2022.

Jianan Zhao, Chaozhuo Li, Qianlong Wen, Yiqi Wang, Yuming Liu, Hao Sun, Xing Xie, and Yanfang Ye. Gophormer: Ego-graph transformer for node classification. arXiv preprint arXiv:2110.13094 , 2021.

Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. arXiv preprint arXiv:2110.03753 , 2021.

Zhiqiang Zhong, Cheng-Te Li, and Jun Pang. Hierarchical message-passing graph neural networks. Data Mining and Knowledge Discovery , 37(1):381-408, 2023.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in neural information processing systems , 33:7793-7804, 2020.

Wenhao Zhu, Tianyu Wen, Guojie Song, Xiaojun Ma, and Liang Wang. Hierarchical transformer for scalable graph learning. arXiv preprint arXiv:2305.02866 , 2023.

A Proof

Proposition 4. (Restatement of Proposition 1) GD-WL with HDSE (D i,j ) is strictly more expressive than GD-WL with shortest path distances (SPD( i, j )) .

Proof. First, we show that GD-WL with HDSE is at least as expressive as GD-WL with shortest path distances (SPD). Then, we provide a specific example of two graphs that cannot be distinguished by GD-WL with SPD, but can be distinguished by GD-WL with HDSE.

Let SPD( i, j ) ∈ R denote the encoding for shortest path distance. It is worth mentioning that

Thus, D i,j is a function of SPD( i, j ) , and hence D i,j refines SPD( i, j ) . To conclude this, we utilize Lemma 2 from [4], which states that refinement is maintained when using multisets of colors. This observation confirms that GD-WL with HDSE is at least as powerful as GD-WL with SPD.

To show that GD-WL with HDSE is strictly more expressive, we provide an example of two nonisomorphic graphs that can be distinguished by the HDSE but not the SPD: the Desargues graph and the Dodecahedral graph. As depicted in Figure 6 of [89], it has been observed that GD-WL with SPD fails to distinguish these graphs. However, GD-WL with our HDSE can. Figure 4 shows the coarsening results of the Girvan-Newman Algorithm [31]. We can demonstrate that, for the Dodecahedral graph, each node has 1 -level hierarchy distances of length 2 to other nodes. On the other hand, in the Desargues graph, there are no such distances of length 2 between any pair of nodes.

GHD

𝟎

GHD

𝟏

GHD

𝟏

GHD

𝟎

𝐃𝐨𝐝𝐞𝐜𝐚𝐡𝐞𝐝𝐫𝐨𝐧	𝐠𝐫𝐚𝐩𝐡

𝐃𝐞𝐬𝐚𝐫𝐠𝐮𝐞𝐬	𝐠𝐫𝐚𝐩𝐡

D

2.00

1.75

1.50

1.25

10

10

1.00

0.75

15

15

- 0.50

0.25

10

15

10

15

0.00

2.00

1.75

1.50

1.25

10

10

1.00

0.75

15

15

0.50

0.25

10

15

10

15

0.00

Proposition 5. (Restatement of Corollary 2) The power of a graph transformer with HDSE to distinguish non-isomorphic graphs is at most equivalent to that of the GD-WL test with HDSE. With proper parameters and an adequate number of heads and layers, a graph transformer with HDSE can match the power of the GD-WL test with HDSE.

Proof. The theorem is divided into two parts: the first and second halves. We begin by considering the first half: The power of a graph transformer with HDSE to distinguish non-isomorphic graphs is at most equivalent to that of the GD-WL test with HDSE.

Recall that the GD-WL with HDSE is quite straightforward and can be expressed as:

where χ t G ( v ) represents a color mapping function.

Suppose after t iterations, a graph transformer with HDSE M has M ( G 1 ) = M ( G 2 ) , yet GD-WL with HDSE fails to distinguish G 1 and G 2 as non-isomorphic. It implies that from iteration 0 to t in the GD-WL test, G 1 and G 2 always have the same collection of node labels. Particularly, since G 1 and G 2 have the same GD-WL node labels for iterations i +1 for any i = 0 , . . . , t -1 , they also

̸

share the same collection of GD-WL node labels { (D v,u , χ i G ( u )) : u ∈ V } . Otherwise, the GD-WL test would have produced different node labels at iteration i +1 for G 1 and G 2 .

We show that within the same graph, for example G = G 1 , if GD-WL node labels χ i G ( v ) = χ i G ( w ) , then the graph transformer node features h i v = h i w for any iteration i . This is clearly true for i = 0 because GD-WL and graph transformer start with identical node features. Assuming this holds true for iteration j , if for any v, w , χ j +1 G ( v ) = χ j +1 G ( w ) , then we must have

By our assumption at iteration j , we deduce that

Hence,

By induction, if GD-WL node labels χ i G ( v ) = χ i G ( w ) , we always have the graph transformer node features h i v = h i w for any iteration i . Consequently, from G 1 and G 2 having identical GD-WL node labels, it follows that they also have the same graph transformer node features.

Therefore, h i +1 v = h i +1 w . Given that the graph-level readout function is permutation-invariant with respect to the collection of node features, M ( G 1 ) = M ( G 2 ) . This leads to a contradiction.

This completes the proof of the first half of the theorem. For the theorem's second half, we can entirely leverage the proof of Theorem E.3 by [89] (provided in Appendix E.3), which presents a similar situation.

Proposition 6. (Full version of Proposition 3) For a semi-supervised binary node classification problem, suppose the label of each node i ∈ V in the whole graph is determined by the majority vote of discriminative node features in the 'hierarchical core neighborhood': S i ∗ = { j : D i,j = D ∗ } for a certain D ∗ , where D i,j is HDSE defined in (3). Assume noiseless node features. Then, as long as the model is large enough, the batch size B ≥ Ω( ϵ -2 ) , the step size η < 1 , the number of iterations T satisfies T = Θ( η -1 / 2 ) and the number of known labels satisfies |L| ≥ max { Ω((1 + δ 2 D ∗ ) · log N ) , BT } , where δ D ∗ measures the maximum number of nodes in the hierarchical core neighborhood S n ∗ for all nodes n , then with a probability of at least 0 . 99 , the returned one-layer graph transformer with HDSE trained by the SGD variant Algorithm 1 and Hinge loss in [53] can achieve a generalization error which is at most ϵ for any ϵ > 0 . However, we do not have a generalization guarantee to learn such a graph characterized by the hierarchical core neighborhood with a one-layer graph transformer with SPD encoding or without any relative positional encoding.

Before starting the proof, we first briefly introduce and extend some notions and setups used in [53]. The major differences are that (1) we extend their core neighborhood from based on SPD to HDSE (2) we use HDSE in the transformer by encoding it as a one-hot encoding for simplicity of analysis.

Their work focuses on a semi-supervised binary node classification problem on structured graph data, where each node feature corresponds to either a discriminative or a non-discriminative feature, and the dominant discriminative feature in the core neighborhood determines each ground truth node label. The node features are sampled from a set of orthonormal vectors following other feature-learning works [40, 50, 2, 37, 52, 51, 90, 13] in theoretically analyzing Transformers. For each node, the neighboring nodes with features consistent with the label are called class-relevant nodes, while nodes with features opposite to the label are called confusion nodes. Denote the class-relevant and confusion nodes set for node n as D n ∗ and D n # , respectively. A new definition here is the distanceD neighborhood N n D , which is the set of nodes { j : D n,j = D,j ∈ V } . D is the HDSE defined in (3). Then, by following Definition 1 in [53], we define the winning margin for each node n of distance D as ∆ n ( D ) = |D n ∗ ∩ N n D | - |D n # ∩ N n D | . The core distance D ∗ is the distance D where the average winning margin over all nodes is the largest. We call the set of neighboring nodes

S n ∗ = { j : D n,j = D ∗ } the core neighborhood. We then make the assumption that ∆ n ( D ∗ ) > 0 for all nodes n ∈ V , following Assumption 1 in [53]. The one-layer transformer we study is formulated as

where W O ∈ R d ′ × d ′′ and a ∈ R d ′′ are the hidden and output weights in the two-layer feedforward network, and b ∈ R Z is the trainable parameter to learn the relative positional encoding. The one-hot relative positional encoding B(D n,u ) is defined as

where c s is the s -th standard basis in R Z . Z is the total number of all possible values of D n,u for n, u ∈ V . B( · ) is a bijection from { d ∈ R K +1 : d = D n,u , for certain n, v ∈ V } to { c 1 , c 2 , · · · , c Z } .

Then, we provide the proof for Proposition 6.

Proof. The proof follows Theorem 4.1 in [53] given the above reformulation. Note that (10) can also map the SPD relationship, which is a special one-dimensional case of D v,u , between nodes as (2) in [53] by the definition of itself. It means that (9) with HDSE can achieve a generalization performance on the graph characterized by the core neighborhood as good as in [53].

However, we cannot have an inverse conclusion, i.e., providing a generalization guarantee on the graph characterized by the hierarchical core neighborhood using (9) with SPD. This is because SPD cannot distinguish nodes with the same SPD but different HDSE to a certain node. Hence, for a certain node i ∈ V , aggregating nodes using SPD may include nodes outside the hierarchical core neighborhood, of which the labels are inconsistent with the node i , and lead to a wrong prediction. Likewise, we cannot guarantee the generalization using a Graph Transformer without any relative positional encoding since such a model cannot distinguish nodes with different HDSE.

Proposition 7. For a semi-supervised binary node classification problem, suppose the label of each node i ∈ V in the whole graph is determined by the majority vote of discriminative node features in the 'core neighborhood': S i ∗ = { j : D i,j = D ∗ } for a certain D ∗ , where D i,j is HDSE defined in (3). Then, for each node i ∈ V , a properly initialized one-layer graph transformer (i) without HDSE (ii) and only aggregate nodes from S i ∗ can achieve the same generalization error as learning with a one-layer graph transformer (a) with HDSE (b) aggregate all nodes in the graph without knowing S ∗ i in prior .

Proof. The proof follows Theorem 4.3 in [53]. When b = 0 is fixed during the training, but the nodes used for training and testing in aggregation for node n are subsets of N n D ∗ , the bound for attention weights on class-relevant nodes is still the same as in (63) and (64) of [53]. Given a known core neighborhood S n ∗ , the remaining parameters follow the same order-wise update as Lemmas 4, 5, and 7. The remaining proof steps follow the contents in the proof of Theorem 4.1 of [53], which leads to a generalization on a one-layer transformer with HDSE and aggregation with all nodes in the graph.

Explanation of GHD Computation in Equation 7. As defined in Eq. 2, GHD m ∈ R | V |×| V | represents the shortest path distance between any two input nodes at the m -level graph hierarchy. ∀ m, GHD m has the same size as GHD 0 .

In Eq. 7, our high-level HDSE computes, at each level c ≤ m ≤ K , distances between input nodes and clusters obtained by coarsening (i.e., super nodes at the c -level graph hierarchy). This is achieved by multiplying the projection matrices ∏ c -1 l =0 P l to GHD m . In effect, it is equvalent to selecting corresponding columns from GHD m . For instance, referring to Figure 1, GHD 1 P 0 ∈ R 11 × 3 calculates the distances from input nodes to the super nodes at 1 -level graph hierarchy, essentially selecting the first, fourth, and tenth columns from GHD 1 .

Likewise, GHD m ( ∏ c -1 l =0 P l ) ∈ R | V |×| V c | selects | V c | columns from GHD m to represent the distances, at the m -level graph hierarchy, between the input nodes and the c -level super nodes (i.e., clusters obtained through coarsening).

B Experimental Details

B.1 Computing Environment

Our implementation is based on PyG [25] and DGL [76]. The experiments are conducted on a single workstation with 4 RTX 3090 GPUs and a quad-core CPU.

B.2 Description of Datasets

Table 8 presents a summary of the statistics and characteristics of the datasets. The initial five datasets are sourced from [20], followed by two from [23], and finally the remaining datasets are obtained from [44, 14, 67, 71, 35, 63, 49].

ZINC, MNIST, CIFAR10, PATTERN, CLUSTER, Peptides-func and Peptides-struct. For each dataset, we follow the standard train/validation/test splits and evaluation metrics in [68]. For more comprehensive details, readers are encouraged to refer to [68].

Cora, Citeseer, Pubmed, Actor, Squirrel, Chameleon, ogbn-proteins, ogbn-arxiv, ogbn-products and ogbn-papers100M. For each dataset, we use the same train/validation/test splits and evaluation metrics as [82]. For detailed information on these datasets, please refer to [82].

Arxiv-year is a citation network among all computer science arxiv papers, as described by [55]. In this network, each node corresponds to an arxiv paper, and the edges indicate the citations between papers. Each paper is associated with a 128-dimensional feature vector, obtained by averaging the word embeddings of its title and abstract. The word embeddings are generated using the WORD2VEC model. The labels of arxiv-year are publication years clustered into fve intervals. We use the public splits shared by [55], with a train/validation/test split ratio of 50%/25%/25%.

B.3 Hyperparameter and Reproducibility

Models + HDSE. For fair comparisons, we use the same hyperparameters (including training schemes, optimizer, number of layers, hidden dimension etc.) as baseline models for all of our HDSE versions. Taking GraphGPS + HDSE as an example, Tables 9 and 10 showcase the corresponding hyperparameters and coarsening algorithms. It is important to note that our HDSE module introduces only a small number of additional parameters. And each experiment is repeated 5 times to get the mean value and error bar.

GOAT + HDSE. To accelerate training, we do not adopt the neighbor sampling (NS) method from GOAT to sample neighbors; instead, we train directly on the entire graph. For graphs with over one million nodes, we randomly sample nodes within the graph and select their induced subgraph for batch training. For the hyperparameter selections of our high-level HDSE model, in addition to what

we have covered in the setting part of the experiment section that datasets share in common, we list other settings in Table 11. It's important to note that our hyperparameters were determined within the SGFormer's grid search space. Furthermore, all other experimental parameters, including dropout, batch size, training schemes, optimizer, etc., are consistent with those used in the SGFormer [79]. The testing accuracy achieved by the model that reports the highest result on the validation set is used for evaluation. And each experiment is repeated 10 times to get the mean value and error bar.

SGFormer on Graph-level Tasks. To accurately demonstrate the capabilities of SGFormer on these datasets, we use all the same experimental settings and conduct the same grid search as outlined in GraphGPS [68].

C Additional Experimental Results

C.1 Performance on Large-Scale Heterophilic Graphs

We run additional experiments on the Pokec and snap-patents datasets [55]. Pokec represents the friendship network of a Slovak online social platform, where nodes correspond to users and directed edges denote friendship relationships. Each node is labeled with the user's reported gender, and features are derived from profile information, including geographical region, registration time, and age. The snap-patents dataset comprises utility patents in the United States, where nodes represent individual patents and edges indicate citation relationships between them. Node features are extracted from patent metadata.

We use the default splits and features from the LINKX [55] and reported the mean accuracy over 5 runs. The results further demonstrate the effectiveness of our HDSE on large-scale heterophilic graphs.

C.2 Sensitivity Analysis of Maximal Hierarchy Level K

We conduct a sensitivity analysis on the maximum hierarchy level K on ZINC. The results are shown in Table 13. Note that when K = 0 , HDSE degenerates into SPD, leading to a worse performance. This result of K is about graph classification tasks, where the size of graphs is typically small. Therefore, at level 1 ( K = 1 ), the quantity of coarsened nodes is quite small, eliminating the necessity for a higher K . We further investigate the impact of K on large-graph node classification, across 3 datasets: Squirrel, arxiv-year, and ogbn-arxiv. Based on the results displayed in Table 14, we can make the following observations: (1) K = 1 does not consistently yield the best results. Optimal performance is achieved with K = 2 on some datasets. (2) The improvement brought about by K = 2 over K = 1 is relatively minor.

The variation in the optimal K could stem from the distinct hierarchical structures inherent in different graphs. Larger graphs may possess more pronounced multi-level structures, thus necessitating a higher K . However, the slight improvement resulting from a larger K could suggest limitations in the coarsening algorithm.

This study reinforces our selection of K = 1 , aligning with results from other hierarchical graph transformer papers such as HSGT [96] and ANS-GT [91]. We anticipate that the real potential of a higher K will be revealed through the application of a proper, effective coarsening algorithm on graphs with hierarchical community structures. We look forward to exploring this in the future.

C.3 Sensitivity Analysis of Maximum Distance Length L

For each graph classification dataset, we calculate the graph diameter of each graph in the dataset and then compute the average graph diameter and maximum graph diameter for the entire dataset, as detailed in Table 15. Note that we use high-level HDSE to deal with node classification on large graphs; therefore, we do not calculate the distances between the nodes in these large graphs. The data indicates L = 30 is a reasonable choice, as it encompasses most of the graph diameters. We do not use a larger number as we hypothesize that for graphs with larger diameters, the utility of detailed information loses significance beyond a certain distance.

Additionally, we conducted a sensitivity analysis regarding the selection of L , as outlined in Table 16, which confirms that L = 30 is an appropriate choice.

C.4 Coarsening Runtime

Table 17 gives the runtime of coarsening algorithms (including distance calculation) on graph-level tasks, illustrating the practicality of our method. The Newman algorithm is unsuited for larger graphs due to high complexity. In addition, our HDSE module almost does not increase the runtime of the baselines. For example, GraphGPS runs at 10 seconds per epoch, compared to 11 seconds per epoch with HDSE module on ZINC.

Additionally, for all large-scale graphs, we employ METIS due to its efficiency with a time complexity of O ( | E | ) . This makes it highly effective for partitioning large graphs, such as ogbn-products, in less than 5 minutes, and even the vast ogbn-papers100M, with a size of 0.1 billion nodes, requires only 59 minutes.

C.5 Impact of Coarsening Algorithms on Large-scale Graphs

Our study on coarsening algorithms in Table 4 focuses on the ZINC dataset, where the size of graphs is typically small (around 20 nodes). The Newman algorithm exhibits optimal performance on these small graphs; however, on large-scale graphs, we use a linear complexity algorithm METIS.

To further assess the impact of linear coarsening algorithms, we conduct additional experiments to study the impact of linear coarsening algorithms on node classification across three datasets: Cora, CiteSeer, and PubMed. The results, as shown in Table 18, demonstrate the advantage of METIS, which is the coarsening algorithm used for node classification in our experiments.

C.6 Synthetic Community Dataset

We evaluate the Community-small dataset from GraphRNN [88], a synthetic dataset featuring community structures. It comprises 100 graphs, each with two distinct communities. These communities are generated using the Erdos-Renyi model (E-R). Node features are generated from random numbers and node labels are determined by their respective cluster numbers with accuracy as the chosen evaluation metric. We use the a random train/validation/test split ratio of 60%/20%/20%.

We select the Louvain method as our coarsening algorithm and integrate the HDSE module into the Graph Transformer (GT). As shown in Table 19, the GT struggles to detect such structures; and solely utilizing SPD proves inadequate; however, our HDSE, enriched with coarsening structural information, effectively captures these structures.

C.7 ANS-GT + HDSE

We validate the performance of our HDSE framework using the efficient ANS-GT [91], which uses a multi-armed bandit algorithm to adaptively sample nodes for attention. We use the Louvain method as our coarsening algorithm. And for each pair of nodes sampled adaptively by the ANS-GT, we calculate their HDSE and bias the attention computation. For fair comparisons, we tune the hyperparameters using the same grid search as reported in their paper [91]. Note that we report the supervised learning setting (different from the text), since this is the one considered in the ANS-GT [91]. Overall, Table 20 shows that HDSE yields consistent performance improvements, even in this challenging scenario, where nodes are sampled.

C.8 Gapformer + HDSE

To further validate the effectiveness of our HDSE framework, we also integrate our high-leve HDSE with Gapformer [56], and observe promising results, as reported in Table 21.

Note that we follow the supervised split setting (48%/32%/20% training/validation/test sets) used in the Gapformer [56] here.

C.9 Clustering Coefficients Analysis

We check if there is a correlation with the cluster structure according to [35], by computing clustering coefficients on five benchmarks in Table 22, but we do not observe a direct correlation. Notably, the

ZINC dataset, which comprises small molecules, has a low clustering coefficient; however, our HDSE shows a significant improvement on it. This improvement could be attributed to the HDSE capturing chemical motifs that cannot be captured by the clustering coefficient, as illustrated in Figure 2.

D HDSE Visualization

Here, we demonstrate that our HDSE method also provides interpretability compared to the classic GT. We train the GT + HDSE and GT on ZINC and Peptides-func graphs, and compare the attention scores between the selected node and other nodes. Figure 5 visualizes the attention scores on ZINC and Peptides-func. The results indicate that, after integrating the HDSE bias, the attention mechanism tends to focus on certain community structures rather than individual nodes as seen in classic attention. Furthermore, different selected nodes lead to different attention weights and consistently demonstrate our HDSE's capability to capture a multi-level hierarchical structure.

E Additional Discussion

Positional Encoding or Structural Encoding?

We would like to clarify that our HDSE method aligns with the definitions of structural encoding. While GraphGPS [68] does classify SPD encoding under relative positional encoding, it defines

relative structural encoding as "allow two nodes to understand how much their structures differ". Given that our HDSE not only incorporates SPD information but also embeds multi-level graph hierarchical structures, it is reasonable to classify it under the category of structural encoding.

Limitations. In larger graphs, the presence of multi-level structures may require a higher maximal hierarchy level, K . The marginal improvements observed with increased K may indicate limitations in the coarsening algorithm. We anticipate that the real potential of a higher K will be revealed through the application of a proper, effective coarsening algorithm on graphs with hierarchical community structures. We look forward to exploring this in the future.

Broader Impacts. This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

F Further Related Works

Graph Transformers over Clustering Pooling. [45] employs a hybrid approach that integrates a neighbor-sampling local module with a global module, the latter featuring a trainable, fixed-size codebook obtained by K-Means to represent global centroids, which is noted for its efficiency. Meanwhile, Gapformer [56] involves the incorporation of a graph pooling layer designed to refine the key and value matrices into pooled key and value vectors through graph pooling operations. This approach aims to minimize the presence of irrelevant nodes and reduce computational demands. However, the performance of these methods remains constrained due to a lack of effective inductive biases.

Graph Transformers over Virtual Nodes. Several graph transformer models utilize anchor nodes or virtual nodes for message propagation. For instance, Graphormer [86] introduces a virtual node and establishes connections between the virtual node and each individual node. AGFormer [41] selects representative anchors and transforms node-to-node message passing into an anchor-to-anchor and anchor-to-node message passing process. Additionally, AGT [62] extracts structural patterns from subgraph views and designs an adaptive transformer block to dynamically integrate attention scores in a node-specific manner.

Moleculargraph

Transformerattn

HDSE attn

0.08

0.35

0.07

0.30

0.06

0.25

0.05

0.20

0.04

0.15

0.03

0.10

0.02

0.05

0.01

0.00

0.00

Peptidesgraph

Transformerattn

HDSE attn

Q

0.14

0.020

0.12

0.10

0.015

0.08

0.06

0.010

0.04

0.005

0.02

0.00

0.000

Moleculargraph

Transformerattn

HDSE attn

0.5

0.25

0.4

0.20

0.3

0.15

0.2

0.10

0.1

0.05

0.0

0.00

Moleculargraph

Transformerattn

HDSE attn

2.5

0.12

2.0

0.10

0.08

1.5

0.06

1.0

0.04

0.5

0.02

0.0

0.00

Moleculargraph

Transformerattn

HDSE attn

0.5

0.25

0.4

0.20

0.3

0.15

0.2

0.10

0.1

0.05

0.0

0.00

Moleculargraph

Transformerattn

HDSE attn

0.25

0.4

0.20

0.3

0.15

0.2

0.10

0.1

0.05

0.0

0.00

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions and complete proofs in Appendix A. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Code, data, and instructions are available at https://github.com/LUOyk1999/ HDSE.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code, data, and instructions are available at https://github.com/LUOyk1999/ HDSE.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe the datasets, splits and hyperparameters in Appendix B. Full configuration files are provided at https://github.com/LUOyk1999/HDSE.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We include standard deviations over several random seeds depending on the dataset evaluation protocol, more details are in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We elaborate on the compute and used resources in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss impacts in Appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the creators. For datasets see Appendix B.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The source code is available at https://github.com/LUOyk1999/HDSE.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This work proposes a new structural encoding for graphs, which can be applied in e.g. graph transformers. The method relies on some graph partitioning / coarsening methods to generate hierarchical clustering of a graph, and the structural encoding is more informative than shortest path structural encoding. The paper proves the the new structural encoding has higher expressivity, and empirical results are good on most graph datasets.

STRENGTHS:
The method is novel, instead of running a graph transformer on different hierarchical clusters, it makes use of the shortest path on different hierarchies. 
Overall the writing is pretty good, the narrative is clear and easily understandable. The illustration is also nice. 
The theoretical and empirical results are strong. The experiments are exhaustive.

WEAKNESSES:
The graph partitioning / coarsening preprocess, including the paritioning algorithm and shortest path computation, can also be complex.

QUESTIONS:
According to the definition of edge set $E^{k+1}$ in line 78, wouldn't there be information loss? If there are multiple edges across two clusters, the number of edges is not represented. 
In section 3.3, is the partition matrix low rank? Especially if the coarsening ratio is close to 1, I guess the partition matrix would be almost full rank.

LIMITATIONS:
According to authors, for larger graphs, more hierarchies are required.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leverages graph coarsening techniques to help the graph transformer capture hierarchical distance information on a graph, improving its performance on both node-level and graph-level tasks. Besides empirical validation, the paper also provides theoretical guarantee about the better expressiveness and generalization of the graph transformer with the proposed HDSE.\n\nSTRENGTHS:\n- This paper is well organized and clearly written.\n- The method incorporates coarsening technology, integrating distance information at different hierarchical levels into the attention score. It is easy to understand and implement and is somewhat novel. Additionally, it has a certain theoretical guarantee regarding the expressiveness and generalization of the proposed method.\n- The experiments are comprehensive, evaluating the model's performance on homophilic and heterophilic graphs, as well as on large-scale graphs. Efficiency and ablation studies are also conducted, along with visualizations. The improvements are significant on some datasets, such as Peptides-func, Chameleon, and Squirrel.\n\nWEAKNESSES:\n- There are some missing results in the experiments (mainly in Tables 2 and 5). For example, the results for models such as ANS-GT, NAGphormer, and LINKX on Actor, Squirrel, and Chameleon are missing. Although implementing all work on all datasets is time-consuming, the missing results reduce the paper's convincingness. Additionally, some results differ from previous reports. For instance, LINKX reported its performance as about 56 on the arxiv-year dataset [1], but in this paper, it is around 53.53. Is this because of different experimental settings?\n- Besides, the performance of the model on large-scale heterophilic graphs remains unknown. The few heterophilic graphs shown in Table 5 are not large. To further demonstrate the model's performance on large-scale heterophilic datasets, experiments on datasets such as pokec, snap-patents, and wiki [1] are recommended. This would enhance the quality of the experiments.\n\n[1] Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods\n\nQUESTIONS:\nBesides the questions mentioned above, there are some additional questions:\n\n**Q1:** In Equation 7 and the illustration below, could you please explain why $\\text{GHD}^m\\prod \\limits_{l=0}^{c-1}P^l$  computes distances from input nodes to clusters at the c-level graph hierarchy? For the input node, why is it not $\\text{GHD}^0$? $\\text{GHD}^m$ represents the distance at hierarchy $m$. What is the meaning of $\\text{GHD}^m$ with the product of projection (coarsening) matrices from level $0$ to $c-1$ (i.e. $\\prod \\limits_{l=0}^{c-1}P^l$)? \n\n**Q2:** In Table 6, the efficiency of GOAT+HDSE, SGFormer, and NodeFormer was compared. However, since NodeFormer and SGFormer use linear attention, is GOAT+HDSE still competitive in terms of efficiency when the graphs are larger (i.e., ogbn-products and ogbn-papers100m)? As the method is claimed to have scalability, illustrations on such large-scale graphs would be beneficial. Additionally, as HDSE is based on graph coarsening and GHD, the cost of coarsening and computation of GHD at each level should also be considered, especially on graphs with more nodes than those used in Table 17.\n\n**Q3:** In the Experiment Setting, the same hyperparameters are used for the proposed method and the baseline transformers. It is also claimed that the hyperparameters are determined within SGFormer's grid search space. However, the optimal hyperparameters for the proposed method may not be suitable for the baseline models. Could you clarify this?\n\n**Q4:** In Appendix 4, the attention scores are displayed between the selected node and the other nodes. How was the node selected? Would different selected nodes lead to different attention weights and still reflect the capability to capture a multi-level hierarchical structure?\n\nLIMITATIONS:\nWhen graphs are larger, a higher maximal hierarchy level $K$, is needed to obtain multi-level structure information, which may be limited by current coarsening algorithms. This limitation can be addressed with more effective coarsening algorithms.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo enhance the effectiveness and scalability of graph transformers, this paper proposes a hierarchical distance structural encoding (HDSE) method to incorporate hierarchical structural information with graph transformers. Theoretical analysis of graph transformer equipped with HDSE shows the improvements of both expressiveness and generalization. Experiments have been conducted to verify the effectiveness and efficiency of the proposed method.\n\nSTRENGTHS:\n1. It is interesting and reasonable to improve the performance of graph transformer based on the hierarchical structural information of the input graph. Moreover, the proposed method is easy to be applied on existing graph transformer models.\n2. Theoretical analysis are provided to guarantee the improvements of expressiveness and generalization of combining HDSE with graph transformer.\n3. A technique of speeding up HDSE to large-scale graphs is provided and verified in experiments on large graphs with millions of nodes.\n4. The paper is well-organized and describes the motivation and methodology clearly.\n\nWEAKNESSES:\n1. It can be seen from Table 4 that the performance of the proposed method varies with different coarsening algorithms and transformer backbones, which should be discussed in details.\n\n2. The experimental settings are not described clearly. Four model + HDSE methods are used for graph-level tasks but only one GOAT + HDSE used for node classification, which makes the results slightly inconvincible.\n\nQUESTIONS:\n1. Although combining the hierarchical structural information into graph transformer indeed improves the performance of the transformer, the improvements vary with different coarsening algorithms. Thus, can the authors provide theoretical analysis on what is the optimal hierarchical structural information and conduct more experiments on how to select suitable coarsening algorithm in practice?\n\n2. The authors integrate HDSE into GT, SAT, GraphGPS and GRIT for graph classification and regression experiments (in Table 2), but only use GOAT + HDSE in the node classification tests (in Table 5), why? Moreover, the results of node classification on classical datasets Core, CiteSeer and PubMed should be reported and discussed in the Evaluation section rather than Appendix.\n\nLIMITATIONS:\nPlease refer to weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}