Tailoring Self-Attention for Graph via Rooted Subtrees
Authors: Siyuan Huang
Yunchong Song
Jiayue Zhou
Zhouhan Lin
Venue: NeurIPS 2023 poster (2023)
Keywords: Graph Based Learning
TL;DR: We design an efficient multi-hop graph attention mechanism based on rooted subtrees.

Abstract: Attention mechanisms have made significant strides in graph learning, yet they still exhibit notable limitations: local attention faces challenges in capturing long-range information due to the inherent problems of the message-passing scheme, while global attention cannot reflect the hierarchical neighborhood structure and fails to capture fine-grained local information. In this paper, we propose a novel multi-hop graph attention mechanism, named Subtree Attention (STA), to address the aforementioned issues. STA seamlessly bridges the fully-attentional structure and the rooted subtree, with theoretical proof that STA approximates the global attention under extreme settings. By allowing direct computation of attention weights among multi-hop neighbors, STA mitigates the inherent problems in existing graph attention mechanisms. Further we devise an efficient form for STA by employing kernelized softmax, which yields a linear time complexity. Our resulting GNN architecture, the STAGNN, presents a simple yet performant STA-based graph neural network leveraging a hop-aware attention strategy. Comprehensive evaluations on ten node classification datasets demonstrate that STA-based models outperform existing graph transformers and mainstream GNNs. The code
is available at https://github.com/LUMIA-Group/SubTree-Attention.

Tailoring Self-Attention for Graph via Rooted Subtrees
Siyuan Huang
Yunchong Song Jiayue Zhou Shanghai Jiaotong University
Zhouhan Lin"
ycaongesjtu euw.tu..uuuuueeuuuuee.... en
lin. zhouhan@gmail.com
Abstract
Attention mechanisms have made significant strides in graph learning, yet they still exhibit notable limitations: local attention faces challenges in capturing long-range information due to the inherent problems of the message-passinng passing scheme, while global attention cannot reflect the hierarchical neighborhood structure and fails to capture fine-grained local information. In this paper, we propose novel multi- hop graph attention mechanism, named Subtree Attention (STA), to address the aforementioned issues. STA seamlessly bridges the fully- cully-attentional structure and the rooted subtree, with theoretical proof that STA approximates the global atten- tion under extreme settings. By allowing direct computation of attention weights among multi-hop neighbors, STA mitigates the inherent problems in existing graph attention mechanisms. Further we devise efficient form for STA by employing kernelized softmax, which yields a linear time complexity. Our resulting GNN architecture, the STAGNN, presents a simple yet performant STA-based graph ral network leveraging hop-aware attention strategy. Comprehensive evaluations node classification datasets demonstrate that STA-based models outper- form existing graph transformers and mainstream GNNs. The code is available at https://g Stam:/LLMMAA-GGrroroSureeeeeee
1 Introduction
Graph Neural Networks (GNNs) have achieved remarkable performance in various tasks, such as drug discovery 35 15]. social networks [29 31. traffic flow [11) and recommendation systems [44). Most GNNs are based on the message-passing scheme [16]. hierarchically aggregating information from multi-hop neighbors by stacking multiple layers. During this procedure, a rooted subtree can be generated for each node, representing the node's ' neighborhood structure. Nodes with similar neighborhood structures possess similar subtrees, which leads to similar node representations [3].
Local attention in graph learning can be seen as natural combination of the pesssae--assiing scheme and the self-attention mechanism. By adaptively assigning weights among one-hop neighbors in single layer, the local attention mechanism allows each node focus on the most task-relevant neigh- bors (39 36 40 3] However, local attention limits the receptive field to one-hop neighbors. While
stacking multiple local-attention layers to build deep model can increase the receptive field, such wawggg--pasinnn--bee
deep architectures face challenges in capturing long-range dependencies [TO)
due to issues such as over-smoothing
( and over-squashing [2].
On the other hand, the global attention mechanism originated from vanilla Transformer B8 has been widely adopted graph learning domain 23 48
28 30 leveraging the fully-attentional
architecture to address the aforementioned issues. However, the global attention employed by graph transformers cannot reflect the hierarchical neighborhood structure and fails capture fine-grand fine-grained
local information, which is crucial in many real-works scenarios [2] 17
46 49
To mitigate
this deficiency, recent studies try to directly assemble global attention and message-passe www.agge-assnnnb-aseeeee
ZZhouhan Lin the corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

models by combining GNNs and Transformers, including simultaneously applying GNNs and Transformers [50] or building Transformers on top of GNNs [33 27 45).
Considering the limitations of both local and global attention, we propose a multi-hop graph attention mechanism, termed SubTree Attention (STA). It allows the root node to directly attend further neighbors in the subtree, enabling the root node to gather information from the entire rooted subtree within one layer. It provides two main advantages: (i) Compared to deep architectures with multiple local attention layers, STA avoids issues associated with the message- paasgeggasinnggggg scheme such as over- smoothing and werrssqushinn. (ii) Compared to global attention, STA can hierarchically capture the neighborhood structure by enabling each node to focus on its own rooted subtree.
Due to the exponential growth in neighborhood size with increased hops, directly calculating attention among multi-hop neighbors becomes impractical. Meanwhile, powers of the adjacency matrix have to be stored for the calculation among multi-hop neighbors. To address these issues, we employ kernelized softmax [19] to develop an algorithm that reduces the quadratic time complexity to linear while avoiding the need to store the powers of the adjacency matrix. This efficient algorithm can be viewed as keys and values performing a random walk on the graph and eventually landing on queries.
Furthermore, we provide theoretical analysis of STA, demonstrating that under extreme settings, STA converges to the global self-attention. As a result, STA can be regarded as a bridge between local and global attention, effectively combining the benefits of both approaches, In addition, present the STA module with multiple attention heads. We introduce a hop- hoip gating mechanism, enabling attention heads to be specialized in capturing information from specific hops.
We then propose a simple yet performant multi-hop graph attention network, named STAGNN, which can leverage multi-hop information and in the meantime acts as a fully-attentional model. As for the evaluation, we test the performance of STAGNN on ten common node classification datasets. Despite its relatively simple architecture, STAGNN consistently outperforms existing GNNs and graph transformers. Furthermore, we demonstrate that STAGNN maintains a competitive performance with an extremely deep architecture. Additional ablation studies conducted to show the effectiveness of subtree attention even in the presence of global attention.
2 Background
Let G = ()) be an undirected graph, with the associated nodes set / and edges set E. We use N = WV to represent the number of nodes. X E Rxx denotes the node feature, where s denotes the number of features. Let A be the adjacency matrix of G and let D be the diagonal degree matrix of A. Aym 1-2 AD 11/2 denotes the symmetric normalized adjacency matrix, while AT = AD-1 denotes the random walk matrix. Let Á be an arbitrary transition matrix, including A pw A sym or other matrices representing message propagation. We use M, and M, to indicate the th row and the jhh column of the matrix M, respectively. And let [0, K] denote the set {0, 1,.... K}.
2.1 Multi-Hop Representations
Many existing GNNs and diffusion-basse models perform multi- hop cressage-passingg in a single layer, taking advantage of multi-hop representations GCN [42 is a typical representative. One important reason for the over-smoothing problem in GCN is that neighborhood aggregation and feature transformation are coupled [4. To address this issue, decoupled GCNs perform feature transformation and neighborhood aggregation, respectively. A
47 22 (1). Among them, decoupled
general form of decoupled GCN can be described as follows (8):
K
0
PROPAG (H), PROPAG (H) A H- MLP(X)
(1)
k=
where {ax (ox k] are the aggregation weights for different hops. K represents the number of propagation steps, which also corresponds to the height of the resulting rooted subtree. Although there exist various strategies assign weights to different hops, these methods all apply K-step propagation with transition matrix A, which inevitably results in over-smoothing when K is large.
There are relatively few attention-base -aased methods that leverage multi-hop information. As represen- tative, Wang et al. (2021) [4] generalized GAT by Personalized PageRank (22). Yet, this strategy

---

only employs attention scores from one-hop neighbors and necessarily acts as low- low-frequeney filter. In a recent work, Chen et al. (2022) [6] began with aggregating Khhop representations and subsequently processed them as sequence of length K using Transformer model. Despite the decent performance, this approach still adopts a Ktstep propagation to capture multi-hop information before taking the attention mechanism into account, which leads to the aforementioned issues. This observation inspires us to incorporate the attention mechanism into the propagation phase, rather than adhering to the K K-step propagation with transition matrix A.
2.2 Global Self-Attention and Kernelized Softmax
In graph learning domain, the Global Self-Attention function SA(:::.) computes a weighted sum of all positions for each node. It first projects the node feature matrix into three subspaces:
Q = XW@ Q: K = XWK, =
(2)
where We E Raxdx WK E RRxdx and Wy E RRdxdv are learnable projection matrices. Then the new representation of the 1th node computed as follows:
S, V): = Σj= sim Qu: Kj.)V
ΣØ=1 sim Qu: (y)
where sim(:.)) : Rd x Rd + R is a function used to evaluate the similarity between queries and keys. common form of self-attention is called softmax attention, which applies the exponential of the dot
Q
product to compute similarity: sim( Qu. K,,) exp
vdk
In fact, we can use an arbitrary positive-definite kernel x to serve as sim( and its corresponding feature map o, we can rewrite sim(., -) as: sim Qu K,:) (K),) Thus, Equation 3 becomes:
:). Given a selected kernel
ES (Q),),,) ΣS= (Q=))(()))
6(Q)) E O(K Y'V (QQ.. Σ ((,))
SA(Q. K, V):
(4)
There are many potential choices for the feature map σ. e.g., Tsai al. (2019) 337 verified that RBF kernels perform on par with exponential kernels on neural machine translation and sequence prediction, and Choromanski et al. (2021) 9 opted for Positive Random Features (PRF).
The key advantage of Equation 4 is that all nodes share two identical summations Σ o (K3)) V'V,
and == o(K,) j:)² )T, which only need to be computed once. By doing so, avoid computing the full attention matrix (sim(Qu. Kj:) Kj:)JeNNJeN and reduce the complexity from O(N2) to O(N).
3 The Proposed Attention Mechanism: SubTree Attention
In this section, we present detailed introduction to an efficient multi-hop attention mechanism called SubTree Attention (STA). First, we give the definition of STA, followed by an efficient algorithm for computing STA based on kernelized softmax and the message-passing scheme. We then explain how multi-head STA makes attention heads hop-aware by incorporating gate into each hop. Finally, prove that STA approximates the global self-attention when the height of the subtree is O (log N).
3.1 SubTree Attention
In this subsection, we give the definition of our proposed multi-hop graph attention mechanism named subtree attention. Similar to the global self-attention function SA(:,:.)), subtree attention takes queries, keys, and values as inputs and outputs new values.
We first define the method for computing attention weights among the 1 hop neighbors, which refer to as STAg. For the ,th node, this process can be described as follows:
STA, Q K, V)
Vin.
Ak E [1, E/ Ak sim (im(Qn, K
(5)
STA Q K. V);
Σ/=
N
A sim (0,, K,:)
3

---

STA
STA2
STAx
STA
HopAggergation
Σ1 A simi R.
>n1 Ag sim(Q.
Input
Transform
Propagate
A
STA+
Censer
STA1 : STA.
Hop 1
AHM,
T
x(Ke)
Hop l
Q:
((Q))
STA. 400EE AgM,/ /WWE// App(Ars))
(a) SubTree Altention
(b) Efficient algorithm of SubTree Attention
Figure 1: (Left)): Definition of SubTree Attention (STA). STA, represents that each node attends the kth level of the rooted subtree, and then STA aggregates information from the entire subtree. (Right): An efficient algorithm for computing subtree attention. We first compute <(K;) and ((K,))" V' for each node, and then let ((K;) and ((K,))VV perform K-step random walk, respectively. After each step of random walk, we compute attention weights using ø ((Qi:) and the aggregated keys and values. The computation {STAs ke]L,K can be seen as nested process.
STA, (Q, K, V) represents the 7th node attending to its 1.th hop neighbors, which are also the athh level of its rooted subtree. SubTree Attention STA (IR"xde, Rexds Raxd.) Raxdd can then be calculated by aggregating the results from all levels of the rooted subtree:
STA(Q, K, V) AGGR ((STA (Q, (Q, V) k E {])
AGGR can be any aggregation function, such as sum, concat GPR-like aggregation 8 which present in /sectiion
[88] attention-basee readout 66, or the
In simple terms, STA, allows each node to attend to the 4: level of its own subtree. Meanwhile, STA aggregates the results from all levels to gather information from the entire subtree. FFigure I shows the process of STA.
In contrast to global attention, subtree attention enables each node to compute attention weights on its own subtree, thus taking advantage of the key insight of Message-Passingg GNNs (MP-GNNs). e.g.. even if the majority of nodes within the receptive fields are the same for two different nodes, they will still gather different information through subtree attention if they have different subtree structures.
3.2 An Efficient Algorithm for Sub'Tree Attention
Equation 3) gives a straightforward method to calculate STA,: we first compute the complete similarity matrix {sim(Q,, Kj)JENNENN and then calculate its Hadamard product with A to obtain {A, sim Qu. Ki:NNEENJJEN In short, we treat A as mask for the similarity matrix. This algorithm exhibits two primary disadvantages: (i) The computational cost associated with calculating the entire similarity matrix is ( (ii) A quickly converges to a dense matrix. Storing A* in the GPU memory for computing the Hadamard product leads to considerable space complexity. Considering these two limitations, utilizing A mask for the similarity matrix is suboptimal.
We now present an efficient algorithm for subtree attention. Considering the close relationship between rooted subtrees and MP-GNNs, we leverage the message--assingg scheme to implement the computation of subtree attention. By permitting keys and values to propagate along the edges, we can achieve an algorithm that has linear time complexity and avoids the need to store A&
Learning from Equation 4 is not the main locus of our work. Our model adopts a simple yet effective approach proposed by Katharopoulos et al. (2020) [19] that chooses ((z) elu(z) + as the feature map and demonstrates empirically that performs on par with softmax attention. We can rewrite Equation 5 as follows:
a feature map ó to replace sim(:.)). The choice of feature map
Σ/ A ((K)) )TV,
Φ(Q E 1
Å (K,,)) TV
STA, V);
(7)
Σ
N Å ((O)))())))
((Qa) Σx= A o(K,:)T

---

Note that there are two summations jj=ı A ((K))) VV; and Σ Ag ; in Equation 7
We can think of these two summations as a kind of message propagation. That is to say, we first compute (KK)) and ((K,))" V'. for each node. Then we let ((K,)) and ((K,) V undergo steps message passing. Finally, we use the aggregated keys and values Σ)= A ((K))) VV,
).V j: and Σラ- M o(5;)² in conjunction with the node's own query (Qq:) to complete the computation of subtree attention. Figure T illustrates the whole process of this efficient algorithm. When we choose Arw as the transition matrix, this process can be regarded as keys and values performing random walk on the graph, eventually landing on different queries. Note that message passing occurs on each edge, thus reducing the computational cost from O to O(EE). Additionally, message passing only requires the sparse adjacency matrix A, thereby circumventing the need to store A#. Furthermore, [STA:]ee[LKK] can be viewed as a nested process, calculated one after another.
Based on this algorithm, we can say that STA is an attempt to incorporate the message-pashan passing scheme into the fully-attentional architecture. In fact, STA serves as a pesssage-passingg module for keys and values. In[section 4) we design novel multi-hop graph attention network employing STA for message propagation. We also provide a detailed complexity analysis in Appendix A
3.3 SubTree Attention with Multiple Heads
In this subsection, present STA with multiple attention heads. Kim et al. (2022) 20 discovered empirically that different attention heads tend to concentrate on neighbors at different hops. Certain attention heads attend to remote nodes, while others consistently focus nearby nodes, sug- gesting that attention heads can be specialized in capturing information from specific hops. To make better of multiple attention heads in this context, we propose a hop-aware method of mixing them.
Suppose there are a total of H attention heads. STA with multiple attention heads, noted as MSTA, can be described as follows:
MSTA(Q, K, V) AGGR ((MSTA) (Q, K. V) & []}
MSTA, Q, V) [head]...... head H!
Wo
Yk E [L, M V (8)
head = d/ STA:(Q" Kh, vh) Vh [h,
9N softmax(gs)
where [] denotes row-wise concatenation. Q", Kh, and V represent the query, key, and value matrices for the htth head, respectively. Wo denotes a linear projection matrix. 9& E IR" is an -Iddimensional vector and of is its 11th element, representing the weight of the At attention head the htth hop. Compared to STA with single attention head, introduce in total H K additional learnable parameters: {g he We can regard 9₁ as a hop-wise gate that determines the weight of each attention head at the Ethh hop.
In other words, can reconsider the multi-hop attention mechanism in terms of multi-task learning. Different attention heads are seen as different experts, while aggregating information from different hops is seen as different task. 93 signifies the process of selecting appropriate experts for each task.
3.4 Theoretical Analysis of SubTree Attention
MP-GNNs suffer from issues like over-smoothing or over squashing when the height of the subtree increases. In this subsection, we theoretically demonstrate that STA avoids the issues associated with the message-passing scheme despite employing the same rooted subtree as MP-GNNs. Notice that we employ the random walk matrix An as the transition matrix in the STA module,i.e. we have Å Anw in this subsection.
We first employ slightly modified approach to rewrite the global self-attention module SA, which can be described as:
N n. sim Qu K V
SA(Q,K,V),
ビj=1 n. sim Qu,, Kj,)
where π, > 4() and d(j) denotes the degree of the 7th node. Note that Equation 9 is consistent
d(s)
with Equation 3
5

---

Transformation
STA-based Propagation
Wq
Mesage passing
MLP
Q
1
STA
STAK
W
K
GiR-like HopAggrogation
Wv
000
α1
Q2
X
H
VE
ag... og Learnahle parameters
Softmax
Figure 2: Overall architecture of STAGNN. STAGNN can decomposed into two parts: Transforma- tion and STA-based Propagation, the latter of which be seen an STA module using GPR-like aggregation as HopAggregation function. According to the algorithm introduced in subsection 3.2 employing STA here is equivalent to letting keys and values propagate the graph.
Comparing [qquation and Equation we find that the only difference between SA and STA, lies in the mask for the similarity matrix. STA, employs A, the mask, whereas SA employs π, We now demonstrate that when the height of the rooted subtree takes the same order of magnitude as log(N)) STA, can be approximately considered SA.
Theorem Let Å NNN denote the random walk matrix of connected and non-bipartite graph, ie.. Á AD-1, Let 1 A1 ... 2 AN be the eigenvalues of A. Then we have the following results:
Vi,j E NI, Ve > O, 3Kg E N, Vk > Ko. I - -T
(10)
And for given the smallest Ky that satisfies the condition shown in Equation 10 is at most log
-ax (n)l) we had more information about V, we could specify the convergence rate of STA 6.g, if V is computed by V = (XWv) where a is a non-negative activation function, then:
O
1- STA, K, V) I+ SA(Q, K, V)
1 ++
Vi,j E [I,, Vn EJO, 1[, 3K1 EN, AK K1,
(11)
-I-1
holds true when none of the denominators is equal to zero. And for a given v, the smallest K₃ that N log 7
satisfies the condition shown Equation 11 at most O
max (A2,111)
with logarithmic complexity. indicating that under general conditions, STA; quickly tends to SA. Notice that although STA, is multi-hop graph attention module implemented using the pressage-passing mechanism, displays the characteristics of global attention when the height of the subtree O (log N). This property prevents STA from the issues associated with the message-pastee passing scheme such over-smoothing and over-squashing. From this perspective, subtree attention serves as a bridge connecting local and global attention. STA1 plays the role of local attention, while STA N) as the global self-attention. Subtree attention achieves hicrarchical attention computation by hop- wise aggregation of (STA, h Olou N) A detailed proof of Theorem provided in Appendix
Equation 10 demonstrates that A converges
The Proposed Multi-Hop Graph Attention Network: STAGNN
In this section, present simple yet effective multi-hop graph attention network, named STAGNN. This model is built upon decoupled GCN, but employs STA as the cessagg-ppssingg module instead. STAGNN can be divided into two steps: first, apply MLP to compute queries, keys, and values for each node; then, we use STA to propagate information. Formally, it can be described as:
Σ ( Q -H-Q. K HWR, K. V HWy. H MLP(X) (12)

---

GCN
MixHop
LDS
NodeFarmar
STAGNN
Cora
CiteSeer
Deezer
Actor
90
Accuracy
Accuracy
76
ROC-AUC
70
Accuracy
89
74
88
63
30
72
Figure 3: Comparison of four GNN baselines, three graph structure learning baselines and STAGNN
on four common node classification datasets. The missing result of Deezer due to out-of-memory.
Table I: Comparison of four GNN baselines, five Graph Transformer baselines and STAGNN on six common node classification datasets. The best results appear in bold.
Method
Pubmed
CoraFull
Computer
Photo
CS
Physics
GCN
86.54
61.7620.14
89.65+0.52
92.70+0.20
92.92+0.12
96.18+0.07
GAT
86.3220.16
64.4720.18
90.784..15
93.87
93.6140.14
96.17
APPNP
88.43
65, 15.1622
90.18 18
94.32220.14
94.4944.00
96.5440.07
GPRGNN
89.34g0.25
67. 12
89.32
94.4944..1
95.1320.09
96.85
GT
88.79
61. 61.0550.38
11.18
94.744..113
94.64
97.05
Graphormer
00M
OOM
0OM
92.74.0.14
0OM
0OM
SAN
88, 15
59.01
89.83
94.86420110
94.51
OOM
GraphGPS
88.9420.16
55. 55.7620.22
OOM
95.06
93.9310.12
OOM
NAGphormer
89.70gu.19
71.51 #0.13
91.22
95.4944.21
95.75+0.09
97.34+0.0z
STAGNN
90.4610.2 22
12..55oo.0
91.72:20.30
95.64
95.7710.16
97.09
where X is the input node feature and O is the learned representation for each node. We adopt the GPR-like aggregation [ for Aggggggenn in STA. To be precise, we assign learnable parameters {aw [c] to each hop (initialized simply to 1), and then the nodes aggregate information from each hop based on these learned weights. Figure 2) shows the overall architecture of STAGNN.
Comparing Equation I and Equation 12 the only difference between decoupled GCN and STAGNN lies in the propagation method. Decoupled GCN relies on high powers the normalized adjacency matrix to capture long-range information, which inevitably results in over-smoothing. In contrast, STAGNN utilizes subtree attention for message propagation, effectively learning more informative representations from multi-hop neighbors without suffering from the inherent problems associated with the message-passing scheme.
Wu et al. (2022) 43) have drawn attention an issue named over-normalization: In the context of graphs with a large volume of nodes, the use the global attention module may lead to a situation where the attention scores for the majority of nodes nearly zero and thus resulting in gradient vanishing. Subtree attention can alleviate this problem by providing a hierarchical calculation focusing each level of the rooted subtree instead of the whole graph.
5 Evaluation
We evaluate the performance of STAGNN on ten common node classification datasets, with detailed dataset information provided in Appendix C We then verify the performance of STAGNN under extreme settings, empirically showing its capacity to tackle over-smoothing. Furthermore, We conduct an experiment that confirms the necessity of subtree attention even the presence global attention. Additional ablation studies are conducted for further discussions. For implementation, we fix the number of hidden channels at 64. More implementation details are presented in Appendix All experiments are conducted on an NVIDIA RTX4090 with 24 GB memory.
7

---

Cora
Actor
GPR weights when K 100
39.0
Cora
Accuracy
89.5
Actor
02.0
37.5
88.5
37.5
85.0
36.5
97.5
75.00
29
60
100
Propagation steps
Propagation steps
Hop
Figure (Leff and Middle): Performance of STAGNN across different heights of the rooted subtree. (Right): GPR weights of STAGNN when the height of the subtree K = 100.
5.1 Experiments on Node Classification
To compare STAGNN with wide variety of baselines, select two recent works 43 their experiment settings and their choice of baselines. The metries of the baselines are adopted from these works [33 []. The code for reproduction can be found in supplementary materials.
6 and follow
Comparison with Multi-Hop GNNs and Structure Learning Methods For a fair comparison, we strictly follow the experiment settings from Wu al. (2022)
43. We test the performance of STAGNN on four common datasets: Cora, Citeseer, Deezer, and Actor The first two are homogeneous
graphs, while the latter two are heterophilic [24 52 34
compare STAGNN with two mainstream GNNs: GCN (21] GAT (39 two multi-hop GNNs: JKNet [47 MixHop M.. and three graph
structure learning methods: LDS 14, IDGL. 77. NodeFormer 433. We apply the same random splits with train/valle crain/vald/ttst ratios of 50%/25%/255% as 44). Further details can be found in Appendix D.I Figure didiplays the experimental results, showing that STAGNN, which features a relatively simple model architecture, outperforms all the baselines all four datasets. This result highlights the effectiveness of STAGNN in managing both homogeneous and heterophilic graphs.
Comparison with Decoupled GCNs and Graph Transformers For a fair comparison, we strictly follow the experiment settings from Chen et al. (2022) 6 We test the performance of STAGNN on six common datasets: Pubmed, Corafull, Computer, Photo, CS and Physics. More information about these datasets can be found in Appendix C We compare STAGNN with two mainstream GNNs:
g and five graph transformers: 6 We apply the same random as [6]. The experimental results are shown in Table I STAGNN shows comparable or superior performance compared to all the baselines, which highlights the competitiveness of STAGNN when compared to existing GNNs and graph transformers.
GCN 21). GAT 33). two decoupled GCNs: APPNP (22], GPRGNN
GT [12], Graphormer 4S). SAN (23]. GraphGPS 32), NAGphormer
splits with train/valid/test ratios of 60%/20%
5.2 Experiments on Deep STAGNN
We evaluate the performance of deep STAGNN on Cora and Actor, with the height of the subtree ranging from 3 to 100. The experimental results are presented in Figure 4 In contrast to MP-GNNs, STAGNN maintains robust performance even when the height of the subtree reaches 100. For Cora, the accuracy of STAGNN peaks
10, demonstrating its ability effectively collect information from a large receptive field. We further visualize the GPR weights of STAGNN when the height of the subtree is set K 100, and we observe distinct characteristics for Cora and Actor. In the case of Cora, the GPR weights exhibit monotonic decrease, which aligns with the witnessed performance drop on Cora as the height of the subtree increases. Therefore, for Cora, we may consider keeping the height of the subtree within reasonable range. In the case of Actor, the GPR weights eventually stabilize at value close to This finding suggests that the limiting state of STA,, i.e., SA,, is suitable for Actor, which is confirmed by the robust performance of deep STAGNN on Actor. More visualizations of GPR weights can found in Appendix In summary, STAGNN can achieve impressive results even with an extremely deep architecture.

---

Table 2: Necessity of subtree attention in the presence of global attention. We compare two scenarios:
(i) Only Global attention and (ii) Global attention supplemented by subtree attention.
Method
Pubmed
CoraFull
Computer
Photo
CS
Physics
GAT
86.32+o.16
64.47
90.78+0.13
93.87
93.61
96.17
Global Attn (GA)
88.87
62.3/
85.7
2..32
94.7420.37
96. 20.24
1-hop STA + GA
90.1610.51
70.65
11.52
10.4
95.49 26
97.09_0.22 0.
2-hops STA + GA
90.5610.49
91.9310.3s 12.24
as
95.75
95.70
97.17 1:
3 hops STA GA
90. 20.6641.22
70.36+0.37
95.88 91.89+0.28
95.81
97. 97.15_0.23
GAT
STAGAN-SUM
CONCAT
STAGNN ATTN
STAGNN- GPR
Cora
CiteSeer
Deazer
Actor
78
Accuracy
89
76
ROC-AUC
Accuracy
25
74
30
87
70
Figure 5: Comparison of different HopAggregation methods
5.3 Ablation Study
Study on the Necessity of SubTree Attention in the Presence of Global Attention We demon- strate that STA, converges to the global self-attention, which naturally leads to critical question:
Is it still necessary to employ subtree attention when global attention is already present?
In this experiment, we seek to answer the question above by exploring two scenarios: (i) applying global attention independently and (ii) combining global attention with subtree attention. Formally, we extend STAGNN by replacing the STA module with global attention enhanced by 0, 1, 2. or hop/hops of subtree attention. More details the experiment settings can be found in Appendix D.2
The experimental results are presented in Table 2 We find that incorporating subtree attention as an auxiliary to global attention significantly improves overall performance, implying that it is still necessary to employ subtree attention to capture the neighborhood structure even in the presence of global attention. This observation also inspires us that leveraging subtree attention to augment global attention could be an interesting approach for enhancing Graph Transformers.
Study on HopAggregation methods In this experiment, we investigate the influence of differ- ent PoopAggreattion functions within STAGNN. We compare GPR-like aggregation with sum, concat [18), and attention-based readout (6). Details of the experiment settings can be found in Appendix D.3 The experimental results are shown in Figure 5 GPR-like aggregation outperforms the alternatives on all four datasets. This observation highlights the importance of enabling nodes to adaptively learn the weight of each hop, which is the main advantage of GPR-like aggregation.
Conclusion
We propose novel multi-hop graph attention mechanism called Subtree Attention (STA), which effectively addresses the limitations of local and global attention in graph learning. The proposed STA mechanism bridges the gap between local and global attention, hierarchically capturing neighborhood structures while addressing issues associated with the message-passian computing subtree attention, utilizing kernelized sofumax and the message-passing scheme, reduces the computational cost from ((N2) to O(EE) while avoiding the need to store high powers of the adjacency matrix. This process can be approximately viewed as keys and values performing random walk on the graph. We further prove theoretically that under extreme settings. STA approximates the global self-attention. Based on the STA module, we propose a simple yet performant multi-hop graph attention network, coined STAGNN. Comprehensive evaluations datasets demonstrate that STAGNN outperforms mainstream GNNs and graph transformers. Ablation studies further verify the effectiveness of subtree attention, even in the presence of global attention.
scheme. The key algorithm for
various node classification

---

Current Limitations, Potential Impacts and Further Discussions In the present work, we mainly evaluate STAGNN, novel multi-hop graph attention network incorporating STA. However, there are many other potential applications of STA, including combining STA with other graph learning methods that utilize self-attention mechanisms or supplementing global attention with subtree attention to enhance graph transformers. Furthermore, evaluating the robustness of STA and digging into its interpretability can also be part of future works. We provide a detailed discussion of potential impacts in Appendix G And further analysis of the gate mechanism within the mixture of attention heads can be found in Appendix
Acknowledgement
This work was sponsored by the National Natural Science Foundation of China (NSFC) grant (No. 62106143), and Shanghai Pujiang Program (No. 21PJ1405700).
References
[1] S. Abu-El-Haija B. Perozzi. A. Kapoor, N. Alipourfard. K. Lerman, H. Harutyunyan, G. V. Steeg, and A. Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 hune 2019, Long Beach, California, USA, volume 97 Proceedings of Machine Learning Research, pages 21 21-29. PMIR, 2019.
[2] U. Alon and E. Yahav, On the bottleneck of graph neural networks and its practical implications. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OvenReview.net, 2021.
[3] S. Brody, U. Alon, and E. Yahav. How attentive are graph attention networks? In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[4] D. Chen, Y. Lin, W. Li, P. Li, Zhou, and X. Sun. Measuring and relieving the smoothigg problem for graph neural networks from the topological view. CoRR, abs//1999.0 19999032111, 2019.
[5] D. Chen, Y. Lin, W. Li, P. Li, J. Zhou, and X. Sun. Measuring and relieving the over-smog problem for graph neural networks from the topological view. In The Thirty-Four Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 3445.. AAAI Press, 2020.
[6] J. Chen, K. Gao, G. Li, and K. He. Nagphormer: Neighborhood aggregation graph transformer for node classification in large graphs. CORR, abs/2206. 049900, 2022.
[7] Y. Chen, L. Wu, and M. J. Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. In H. Larochelle, M. Ranzato. R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[8] E. Chien, J. Peng. Li, and Milenkovie. Adaptive universal generalized pagerank graph neural network. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[9] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins. J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. Colwell, and A. Weller. Rethinking attention with performers. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[10] H. Dai, Z. Kozareva, B. Dai, A. Smola, and L. Song. Learning steady-states of iterative algorithms over graphs. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 11 1114-1122. PMLR, 2018
[11] A. Derrow-Pinion, J. She, D. Wong, O. Lange, T. Hester, L. Perez, M. Nunkesser, S. Lee. X. Guo, P. W. Battaglia, V. Gupta. A. Li, Z. Xu, A. Manchez-Gonzalee Y. Li, and P. Veličković. Traffic Prediction with Graph Neural Networks in Google Maps. 2021.
10

---

[12] V. Dwivedi and X. Bresson. A generalization of transformer networks to graphs. CoRR. abs/2012.09699 2020.
[13] J. Feng, Y. Chen, F, Li, A. Sarkar, and M. Zhang. How powerful are k-hop message passing graph neural networks. NeurlPS, 2022.
[14] L.. Franceschi, M. Niepert, M. Pontil, and X. Hc. Learning discrete structures for graph neural networks. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Muchine Learning, 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1972-1982. PMLR, 2019.
[15] T. Gaudelet, B. Day, A. Jamash, J. Soman, C. Regep, G. Liu, J. B. Hayter, R. Vickers, Roberts, Tang, et al. Utilising graph machine learning within drug discovery and development. arXiv preprint arXiv:2012.05716 2020.
[16] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1263-1272. PMLR, 2017.
[17] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang. J. Yu, W. Han, S. Wang. Z. Zhang, Wu, and R. Pang. Conformer: Convolution-augmenn
transformer for speech recognition. In H. Meng. B. Xu, and T. F. Zheng, editors. Interspeech 2020, 21 Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pages 5036-5040 ISCA, 2020.
[18] W. L. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning large graphs. In Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, V. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
2017.
[19] A. Katharopoulos, A. Vyas, N. Pappas. and F. Fleuret. Transformers are mas: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 11661165. PMLR, 2020.
[20] J. Kim, D. Nguyen, S. Min, S. Cho, M. Lec, H. Lee, and S. Hong. Pure transformers are powerful graph learners. In NeurIPS, 2022.
[21] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Reprexentations, ICLR 2017, Toulon. France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[22] Klicpera, A. Bojchevski, and Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[23] D. Kreuzer, D. Beaini, W. L. Hamilton, V. Létourneau, and P. Tossou. Rethinking graph transformers with spectral attention. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 1021, NeurlPS 2021, December 6 14, 2021, virtual, pages 21618 216622, 2021.
[24] D. Lim, X. Li, F. Hohne, and S. Lim. New benchmarks for learning on non homohoohhhlous graphs. CoRR, abs/2104.01404, 2021.
[25] K. Lin, L. Wang. and Z. Liu. Mesh graphormer. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 12919-12928. IEEE, 2021.
[26] L. László, L. Lov, and O. Erdos. Random walks on graphs: survey. pages 1-46, 01 1996.
[27] G. Mialon, D. Chen, M. Selosse, and J. Mairal. Graphit: Encoding graph structure in transformers. CoRR, abs/2106.05667, 2021.
[28] E. Min, R. Chen, Y. Bian, T. Xu, K. Zhao, W. Huang, Zhao, J. Huang, S. Ananiadou, and Y. Rong. Transformer for graphs: An overview from architecture perspective. CoRR, abs/2202.08455, 2022.
[29] F. Monti, F. Frasca, D. Eynard, D. Mannion, and M. M. Bronstein. Fake news detection social media using geometric deep learning, arXiv preprint arXiv: wwx.v:1902.0600 002700673. 2019.
11

---

[30] L. Müller. M. Galkin, C. Morris, and L. Rampásek. Attending to graph transformers. CoRR. abs/2302.04181, 2023.
[3]] A. Pal, C. Eksombatchai, Y. Zhou, B. Zhao, C. Rosenberg. and J. Leskovec. Pinnersage: Multi-modal user emhedding framework for recommendations at pinterest. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2311 1-220, 2020.
[32] L. Rampásck, M. Galkin, Dwivedi, T. Luu, G. Wolf, and D. Beaini. Recipe for a general, powerful, scalable graph transformer. In NeurIPS, 2022.
[33] Y. Rong. Bian, T. Xu, W. Xie, Y. Wei, W. Huang. and J. Huang. Sel--super graph transformer on large-scale molccular data. In Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[34] P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Gallagher, and T. Eilassi-Rad. Collective classification in network data. Al Mag., 29(3).933111)) 1069 2008
[35] J. M. Stokes, K. Yang. K. Swanson, W. Jin, A. Cubillos-Ruiz, N. M. Donghia, C. R. MacNair, S. French, L. A. Carfrae, Z. Bloom-Ackermann, et al. A deep learning approach to antibiotic discovery. Cell. 180(4):688-700 702, 2020.
[36] K. K. Thekumparampil, C. Wang. S. Oh, and L. Li. Attention-based graph neural network for semi- supervised learning. CoRR, abs/1 803.03735, 2018
[37] Y. H. Tsai, S. Bai, M. Yamada, L. Morency, and R. Salakhutdinov. Transformer dissection: An unified understanding for transformer's attention via the lens kernel. In K. Inui, J. Jiang, V. Ng, and X. Wan. editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, MMNLP-IJCNLA 2019, Hong Kong, China, November 3-7, 2019, pages 4343 4352. Association for Computational Linguistics, 2019.
[38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. von Luxburg. S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 6008, 2017.
[39] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canuda, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[40] G. Wang, R. Ying, J. Huang, and J. Leskovec. Improving graph attention networks with large margin-based constraints. CoRR, abs/1910. 1945, 2019.
[41] G. Wang, R. Ying, J. Huang, and J. Leskovec. Multi-hop attention graph neural networks. In Z. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 3089-3096. ijcai.org. 2021.
[42] F. Wu, A. H. S. Jr., T. Zhang, C. Fifty, T. Yu, and K. Q. Weinberger. Simplifying graph convolutional networks. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Confer- ence on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Califormia, USA, volume 97 of Proceedings of Machine Learning Research, pages 6861 18611. PMLR, 2019.
[43] Q Wu, W. Zhao, Z. Li, D. P. Wipf, and J. Yan. Nodeformer: A scalable graph structure learning transformer for node classification. In NeurlPS, 2022.
[44] S. Wu, F. Sun, W. Zhang, X. Xic, and B. Cui. Graph neural networks in recommender systems: A survey, ACM Comput. Sure, 55(5):97:
2023.
[45] Z. Wu, P. Jain, M. A. Wright, A. Mirhoseini, J. E. Gonzalez, and I. Stoica. Representing long-range context for graph neural networks with global attention. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, Liang, and W. Vaughan. editors. Advances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, pages 13266- 10666-13799 13279, 2021.
[46] Z. Wu, Z. Liu. J. Lin, Y. Lin, and S. Han. Lite transformer with long- range attention. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30. 2020. OpenReview.net, 2020.
12

---

[47] K. Xu, Li, Y. Tian. T. Sonobe, K. Kawarabayashi, and S. Jegelka. Representation learning on graphs with jumping knowledge networks. In J. G. Dy and A. Krause, editors, Proceedingy of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 5449-544 5458, PMLR, 2018.
[48] C. Ying, T. Cai. Luo, Zheng, Ke, He, Shen, for graph representation? In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang. and J. W. Vaughan. editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, pages 28877-28888 2021.
Liu, transformers really perform badly
[49] A. W. Yu, D. Dohan, M. Luong. R. Zhao, K. Chen, M. Norouzi, and Q. V. Le. Qanet: Combining local convolution with global self-attention for reading comprehension. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[50] Zhang, H. Zhang, C. Xia, and Sun. Graph-bert: Only attention needed for learning graph representations. CoRR, abs/2001.0S 105140, 2020.
[51] Zhao, Y. Dong, M. Ding, E. Kharlamov, and J. Tang. Adaptive diffusion in graph neural networks. In Ranzato, A. Beygelzimer, N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 102 NeurlPS 2021, December 6414, 2021, virtual, pages 23321-23333 2021.
[52] J. Zhu, Y. Yan, L. Zhao, M. Heimann, L., Akoglu, and D. Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
13

---

A Complexity Analysis of STA
In this subsection, we analyze the time complexity of SubTree Attention (STA). STA has two key components: the feature map and the HopAAggeeation function. Both of these components offer wide range of potential options for consideration. Different options will affect the time complexity of STA. In the following analysis, we will adopt the configuration used by STAGNN, i.e. we choose (z) = elu(z) + 1 as the feature map and use GPR-like aggregation as the HopAggregation function.
The computation of STA can be seen aggregation of (STA, Jee [i x]. which refer to the attention- based aggregation of each level of the rooted subtree. Therefore, we can start by analyzing the time complexity of STAg-
STA1
STA
STAK
Input
Transform
Propagate
Vi
(KK)) VV
M
AyMs
MB
Mg
T
N
K
(Ka))
Σ 5k) 3.)
Σ (k
Σ (k))
Q:
(QQ))
STANQQ K, V) -Q() +Q:)) Ny (() Eg) Abo(K33)
Figure 6: Efficient algorithm of SubTree Attention
The calculation of STA, can be divided into three steps. In the first step, we compute Φ(K,) and &(K,.)" V"V; V;, model, we chose ((x) = elu(x) + as the feature map. Thus, the time complexity of computing ((K,.) ) ((Nd)). We also need to compute @ K;,"VV for each node, the time complexity of this part is O(Nddu)). Therefore, the overall time complexity of the first step is O(Nd, + Nd4d,).
s; for each node. The time complexity of this step depends on the feature map. In our
In the second step, we let ((K,)) and d(,,)""V propagate the graph. For STAR, we need to propagate A times. The time complexity of propagating <(K;) once is O(dx), ). and the time complexity of ((K,)""V edge. Considering that there are in total |E| edges and &s times propagation, the overall time complexity
propagating once is (dkd,). The message propagation occurs on each
of this step Ok [d4 + k|E (d
J: and E A o TT aggregated by each node, along with the node's 's own query o(Q.). to complete the computation of STA. For each node, we need to calculate of Q Σ/= A ((K,,)) T f:, the time complexity of this part is dg))). At the same time, for each node, we need to calculate &f (QQ.:) Σ A o(K,:)" the time complexity of this part is (N So the total time complexity of this step is ON d2 + +Nddd,). In summary, the total time complexity of STA, is O(2Nd: 2Nd,d, El + k k|Eld,do).
In the third step, we use the information Σ/ A o(K):) (VK
VV,
Next, we analyze the time complexity of STA when the height of the rooted subtree is K. should be noted that (STA, heefi, KJ can be viewed as a nested process, calculated one after another. Therefore, the first two steps of the above-mentioned calculation of STA, not need to be repeated. We only need to complete the full calculation of STA and perform the third step mentioned above K times. Therefore, the time complexity of calculating STA is O((K + Nd + (K + N) + K|E/d K [l In general, we can think of the time complexity of STA as OKKE (.
14

---

B Proof for Theorem
B.1 Proof for Equation 10
Let Á denote the random walk matrix of a connected and non-bipartite graph, and let Asym denote the symmetric normalized adjacency matrix. Let a
AN be the eigenvalues of A, which
also the eigenvalues Asm [26]. Let V1, V2. Vn be the corresponding orthonormal eigenvectors (V1, V2: ................................ Vn here are column vectors). Let πr node. Á
d(f) and d(i) denotes the degree of the 2t Dm d(²)
max A2, |arl) denotes the corresponding spectral gap, and let D be the diagonal degree matrix. I't denotes all-ones column vector.
In this subsection, we prove the following results:
Vi,j [I, Ve > 0, 3Kg E N, Ak Ko, |A Tx| Se
log
And for a given the smallest Ko that satisfies the condition above is at most
T-max A2.1
We begin by considering an arbitrary distribution E RN, which is a column vector and ||Pi||2 1.
Notice that V1, V2, Yn form an orthonormal basis, we can rewrite D- pp as:
$ pl = cv
(13)
We next consider the new distribution obtained when Pi undergoes k-step random walk. Notice that D
Á AD-1
D D AD-
Thus we have:
A A
D A D
pi
D sym +p
D
C.Vi
(14)
A Xym Vi
N sym V1 + Σ
= c D
A
sym Vi
1=2
We now consider e D A sym V1. As know that D I
is eigenvector of Asy with eigenvalue
1. We then have:
D V1
(15)
| T
15

---

Notice that |D 1 12 VE d(i). Using the fact that V1, V2, form orthonormal basis
and D 3p: Ei=1 Ci,i, we then have:
c
D
$ Pi
81
D 1
Pi D
[ I 112
Pi
(16)
D 1
I
1l
| I 1|
V E d(i)
Notice that A V1 = v and A = 1. Thus we have:
D A sym
v1 e D Xfv1
=cD
D
1
D
ΣM d(i)
|D 1 1|
(17)
DÍ
E d(i)
N
Considering Equation 14 and Equation 17
we have:
A A Pi = +
Σ
G
D A
sym Vi
(18)
and immediately:
A Pi
π
D
A
vil
sym
N
vi vill2
(19)
sym
N
k
vil.
smm
2=2
where D |||,
sup =ERN
| ||=||
Vmnx- Thus we have:
N
| k P - 21/ dmax
Σ
c
k
sym vill2
(20)
1=2
16

---

And using the fact that V1, 2 ...... Vn are orthonormal and 1 Å then have:
max |Nr
max ..l...... .........BBB B3
A $ pj T12 - dmax
Σ
Gj
c
vill
N
dma
2
A
(21)
1=2
S dmax
(1
\)
1=
dmax (1 1)2h
p:l12
Notice that |D
< <||D=>]
l|p: 1. Therefore:
dmax (1 2)26
| Di
(22)
dmii
and immediately:
dmax
A Pi
4/22 5
(1 2)
dmin
v N(1 2)*
(23)
Vee -
Using Cauchy-Schwarz, we then have:
A Pi T/I S VVIIA Pi T/12 S -A
(24)
In conclusion, given an arbitrarily small positive number for all kip greater than or equal to log e.
the L1 norm of the difference between À & P; and the vector æ is less than or equal to e. This result establishes that } log N indeed serves upper bound.
Notice that the vector pi is arbitrary distribution. Thus, may consider pi to be one of jth unit basis vector in the N-dimensional space: {Pi, ........ PN.. where each vector has only one element equal to (the 7th element) and all other elements equal to O. Thus we have | * π Σ
A 5
n.|- Then given O, we have that:
N
Yj E [I,]] AA 2 log
ΣIA
TilSe
(25)
which demonstrates immediately the first part Theorem
N
Vi.j E N1, Vk N i
log
| T Sc
(26)
B.2 Proof for Equation 11
In this subsection, we prove the following results: is computed by V = a (XWv) where a is non-negative activation function, then:
1 1- STA, V)
1 1
Vi.j E [I? An EJ0, 1[. 3K₁ N, Vk K1.
<
1 1+
SA(Q, V)j
1 1-
holds true when none of the denominators is equal to zero. And for given y, the smallest K₁ that satisfies the condition shown in Equation ii at most O
1>..x(22.12..1
This result can, indeed, viewed as a straightforward corollary of Equation TT). The crucial
prerequisite is that all elements of the vector V must be positive. Importantly, there specific
17

---

requirements imposed on the non non-negative activation function d, meaning it can be any function that ensures non-negativity.
STA, and SA are defined as follows:
jj=1 A
i sim Qu., V,
STA, K. V)2.
ンj=1 by sim Qu: Kj:) Σ أ==
(27)
sim Q K.)) V; Σラ क. sim Qu. Kj))
SA(Q, K, V)
N
Equation 27 shows their form as row vectors. Their jth elements are:
Σ A sim Qu Va N
Σ A sim Qu: K.,) STA, V)
(28)
Σ sim Qui, K V Em nI sim Qu.. K..)
SA(Q, K, V)u
Hence, we have:
STA Q. SA
E b sim Qu., Vo
E Ti sim Qu: K..)
(29)
EE sim Q K
2t=1 A sim Qu. K..)
For clarity, proceed under the assumption that none of the denominators equal zero, which is reasonable considering the context. Let da represent the difference between A and πं A π. + 6g. Given EJ0, we aim to determine an upper bound of the convergence rate between STA, and SAA
Using Equation 10
we take = N and we have immediately:
Vi,t E [I, Vk N 2og
A
T = 111==10cl= < N2
17
(30)
We can rewrite Equation 29las: as:
STA, STA,(Q,K, V)sg
EN (TT, + 6a) sim Qi: K
Σ 1= n, sim Qu, K.)) E (T, 5x)) sim Qu: K..)
(31)
SA(Q, K, V)
Em T. sim Qa:. Kr.)V
2 2log Considering the fraction
>N Bhe sim K. in the first part of 21-1 Te K.)
Assuming that &
>
Equation
Using Equation 30 and the fact that sim( (Qu. K.:) and Vis are all positive, we have:
N
Sa sim (Q:: K,y
S5e (ic V
(32)
N
"
sim Qú, V
N
Notice that Vt E [1, n; 2 NZ. Hence, we have:
N
N
म. sim Qie K. Vo
sim Qa, K..) Vij
(33)
N2
Therefore:
dy sim Qu. Vn
de sim Q2., K. V
< 1 <
(34)
1= T. sim Qu., K.)
Ti sim Qu,, K. V.yl
Thus have
E (T;; + 5a) sim Qu. K.,V ΣN ni sim Qu., K.) V
1 1-
< 51+1
(35)
18

---

Table Statistics on datasets
Dataset
Context
# Nodes
# Edges
# Features
# Classes
Cora
Citation
2,708
5,429
1,433
7
Citeseer
Citation
3.327
4,732
3,703
6
Deezer
Social Connection
28,281
92,752
31,241
Actor
Co-oceurrence
7,600
29,926
931
Pubmed
Citation
19,717
44.324
500
CoraFull
Citation
19,793
126,842
8,710
70
Computer
Co-purchasing
13,752
491,722
767
10
Photo
Co-purchasing
7,650
238,163
745
8
CS
Co-authorship
18,333
163,788
6,805
15
Physics
Co-authorship
34,493
495,924
8,415
Considering the second part reasoning, obtain:
s1 ( (Q... K..)
of Equation 31
Utilizing the same line of
1= (Tt 4-4e) sim K..)
1= du sim Qú. K.))
n <
(36)
1= n. sim Qi. K.:)
and
1
Σ -= ग. sim Q K
(37)
1+
1=1 n + 6τ)) sim Qu: K..)
Considering Equation 35
Equation 37 and Equation 31 we finally prove that:
STA, Q K, V)
1+
<
<
(38)
+
SA(Q. K, V)j 1-1
which proves the second part of Theorem 1.
C Dataset Information
In this section, we present the datasets used our experiments. These different types of data provide robust platform evaluate the performance of our methods.
The detailed information for each dataset is presented in Table These datasets are drawn from the areas of citation networks, c0-purchasing networks, co-authorship networks, and social networks: Citation Networks: The citation networks datasets include Cora, Citeseer, Pubmed, and CoraFull. Nodes in these networks correspond to scientific publications, while the edges represent citations between these documents. In addition to the topological structure, each node carries a binary attribute vector, encoding the presence absence of specific words from pre-determined dictionary. The dimensionality of these attribute vectors varies from 1,433 in Cora to 8,710 in CoraFull. Moreover, each document node is associated with unique class label, signifying the document's overarching scientific discipline. Co-authorship Networks: We utilize the CoauthorCSDataset and Coauthor- PhysicsDataset that capture co-authorship relationships in Computer Science and Physics domains, respectively. Nodes represent individual authors and edges encode co-authorship relations, thus creating undirected graph. Co-purchasing Networky: We utilize the AmazonCoBBuyCompure Dataset and AmazonCoBuy PhrrronoooByPPhooooooooo PhotoDataset, derived from Amazon's co-purchasing network. Nodes denote products and edges symbolize frequent co-purchase incidents. Moreover, the nodes can carry diverse product-specific information. Social Networks: The Deezer-Europe dataset is dataset representing social network of Deezer users collected via the public API in March 2020. The nodes in this network symbolize Deezer users hailing from various European countries, while the edges represent reciprocal follower relationships between these users. The features of each node are derived from the preferences of the users, specifically, the artists they have expressed an interest in. The task associated with this graph involves binary node classification, wherein the objective to predict the user's gender. Co-occurrence Networks: We utilize the Actor dataset, type of co-occurrence network based on the Microsoft Academic Graph. Nodes represent actors, and an edge signifies their co-appearance the same Wikipedia page.
19

---

D Implementation Details
Positional Encoding We use Laplacian positional encoding to capture the structural information. the focus of our work, we use simple approach to combine positional encoding with the original features of the nodes, which is also applied by [6]. Formally, we first calculate the eigenvectors corresponding to the smallest m eigenvalues of the Laplace matrix to
As positional encoding
construct the matrix P E R=xm Then we take X' = [X, P] as the new input, where [] denotes row-wise concatenation. For all the datasets, we set m 3.
D.1 Node Classification
Training Details
We choose two recent studies 45 6 and we adhere to their experimental config-
For Cora, Citeseer, Deezer and Actor, we apply the same random splits with train/valid/test ratios of 50%/25%/255 as 43). We conduct 5 runs with different splits and take the mean accuracy and standard deviation for comparison. For Pubmed, Corafull, Computer, Photo, CS and Physics, we apply the same random splits with train/valid/test ratios of 60%/20%/20% take the mean accuracy and standard deviation for comparison. Specifically, we utilize the ROC-AUC the Deezer dataset. For other datasets containing more than
urations. The metrics for the baselines are also derived from these works [43 6
[6]. We conduct 10 runs with different splits and
measure for binary classification two classes, we opt for Accuracy as the metric. We employ the Adam optimizer for gradient- based optimization. The training procedure can at most repeat until given budget of 3000 epochs and we set the patience of early stop to 200 epochs. We report the test accuracy of the epoch which has the highest accuracy the validation set.
Hyperparameters For the model configuration of STAGNN, we fix the number of hidden channels at 64. We use grid search for hyper-paramedd settings. The learning rate is searched within (0.001,0.01) J. dropout probability searched within (0.0,0.2.0.4.0.6). weight decay searched within (0.0001,0) www.05.0000000 1,,00,00,,..... 1,,00,,,,)), height of the rooted subtree K searched within (3,5,10), number of attention heads searched within (1,2,4.688). The best hyper-parameters are provided in supplementary materials.
D.2 Study on the Necessity of SubTree Attention in the Presence of Global Attention
In this experiment, we extend STAGNN by replacing the STA module with global attention enhanced by 0, 1, 2, or 3 hop/hops subtree attention. We now present a detailed mathematical description the experimental configurations. Formally, compare the performance of the STAGNN-basee model equipped with four different attention strategies: Global Attn Only, I-hop STA + GA, 2-hops STA + GA and 3-hops STA+ + GA six datasets: Pubmed, Corafull, Computer, Photo, CS and Physies, with the same experiment setting described in subsection D.T
First, we calculate keys, queries and values.
Q HWQ, o, K HW K. V = HWv, H MM)P(X)
(39)
Next, the output of the four different models (equipped with global attention enhanced by subtree attention of different heights) can be described as:
Global Attn Only:
O = SA(Q, K, V)
(40)
I-hop STA + GA:
0 = -= V)+ Σ
STA (Q,K, V)
(41)
2-hhpp STA + GA:
2 0 S. V) + Σ
STA, V)
(42)
20

---

h-hs STA + GA:
O ay V) + Σ (Q,
(43)
@T here represents the coefficient of teleportation, because we can regard the global attention enhanced by subtree attention here the random walk with teleportation. The only difference between these models that they use subtree attention of different heights as an auxiliary to global attention. As shown in Table 2 we can observe that 2-hops STA + GA and 3-hops STA + GA outperform Global Attn Only by large margin.
D.3 Study on HopAggregation Methods
In this experiment, we investigate different choices of the HopAggregation functions within the STA
module. We compare GPR-like aggregation with sum, concat [18]. and attention- based readout [6]. We now present detailed mathematical description of the experimental configurations. Formally, compare the performance of the following four models: STAGNN- SUM, LIGNN-CONCA and STAGNN-ATTN four datasets: Cora, Citeseer, Deezer-Europe and Actor, with the same experiment setting described in luussccttion D.I.
(origin STAGNN), STAGNN-
First, we calculate keys, queries and values.
Q HWQ, K HW K. V = HWr, = MLP(X)
(44)
Next, the output of the four different models (STAGNN with different HopAggregation methods) can be described as:
STAGNN-GPR (origin STAGNN):
O
Σ ag ST, V) k=0
(45)
STAGNN-SUM:
K 0
STAx(Q,
(46)
k=0
STAGNN-CONCAT:
O [STAa K, STA V).., V) Wo
(47)
where Wo is linear projection matrix.
STAGNN-ATIN:
O = STAA V) + Σ (. V),
(48)
Px exp (SSTAa K, V), STA, V)J W!)
Σ exp (SSTAo K, V), STA, QQ, V)J WT)
where W. is linear projection matrix and denotes row-wise concatenation.
E More Visualizations of GPR Weights
We conduct more visualizations of the GPR weights Cora and Actor, with heights K of the rooted subtrees ranging from 3 to 75. The results are shown in Figure
In the case of Cora, we observe that the depth K of the rooted subtree increases, STA keeps increasing the GPR weights of the local neighborhood in order to preserve the local information from being covered up by the global information.
21

---

film
GPR weights when K=
GPR weights when K
GPR weights when K
GPR weights when K=
2.5
2.0
1.5
1.0
3.0
GPR weights
GPR weights
GPR weights
GPR weights
2.0
2.0
15
1.0
GPR weights when K-
GPR weights when 30
GPR weights 50
GPR weights when
2.0
2.0
2.5
1.
1.0
6.
Figure 7: GPR weights STAGNN when the heights K of the subtree ranging from to 75.
F Further discussion of the Gate Mechanism within the Mixture of Attention Heads
In this subsection, we conduct an ablation study of the gate mechanism within the mixture of attention heads. The sub-tree attention module with multiple attention heads defined as follows:
MSTA w/ gate vector 9t. w/ softmax (origin STAGNN):
MSTA(Q, K, V) AGGR (IMSTA (Q, K, V) K [0, KJ})
MSTA+ K, V) head! .... head!" !! Wo
Ak E [1, MSTA K, V) V (49)
head 96 STAa(Q" Kh Vh) 4h E [1, H],
Ox softmax(gg)
The hop-wise gate vector here 9k IR is an H H-dimensional vector and gk is its A element. Compared STA with single attention head, we introduce in total H K additional learnable parameters: hi.,,.))
For comparison, we consider two variants.
MSTA w/ gate vector 9v. w/o softmux:
MSTA(Q, K. V) = AGGR ((MSTA (. & E [0, K]}) MSTA, Q, K, V) head! heead Wo Yk E [1,
MSTA, = =V (50)
head of (QA Kh vh) Vh E [I.
MSTA w/o gate vector gk:
MSTA(Q, K, V)- AGGR (IMSTA K, V) & E [0, K]})
MSTA, (Q. K, V) head ... head H
Wo
vk E [L, MSTA, V (51)
head STA,, (Q K², vh) Vh []
The experimental results are shown Table 4 We find that the performance of MSTA w/ gate vector kk. w/o softmax and MSTA w/o gate vector g almost the same, which means that using the gate
22

---

Table 4: Ablation study of the gate mechanism within the mixture of attention heads
Method
Pubmed
CoraFull
Computer
Photo
CS
Physics
STAGNN (origin)
90.4640.22
72.65
91.72=0.30
95. 04.
95. 20.17
97 97.09
w/ gate, w/o softmax
90.37 10.23
71.6220.39
95.37 91.8910.27
91.7210011
96.9610.20
w/o gate
90.31 10.25 71.67
91.80
95.3210.29
91.7010.18
96.9710.18
vector without softmax is approximately equivalent to not using the gate vector. In fact, on closer examination, we find that without softmax, the learned gate vector would be vector with all equal elements, which means that it is difficult for the model to learn different weights of attention heads at each hop without the help of softmax. Additionally, we observe that for most datasets, using the gating mechanism leads to improvement of the overall performance.
G Potential Impacts
Besides learning better node representations, proposed Subtree Attention (STA) has potential impacts on various aspects of graph learning. Compared to global attention, STA can help the model to better learn the hierarchical structure of the graph. Therefore, STA can be utilized module for designing local-aware Transformers on graph, acting as a competitor of all the GNN- assisted Transformers. STA opens new avenues for model design combining the message-passingg scheme with fully-attentional architectures, which can significantly enhance both the computational efficiency and expressive power of fully-attentional models graph data. Furthermore, STA bridges the gap between local and global graph attention methods. This opens possibilities for the design and application of hierarchical attention models that can leverage both local neighborhood and global structural information from graph data.
plug-in in
23


Reviews:
[{"review_id": "7ibP8kHzjP", "reviewer": "Reviewer_sk7A", "summary": "In this paper, the authors propose a novel Graph Transformer called STA-GNN (SubTree-attention-GNN) to address the over-smoothing and over-squashing in the message-passing scheme. Different from the previous Graph Transformer, STA-GNN's attention mechanism is called SubTree Attention (STA), which computes a node's attention according to different levels of its subtree. To overcome the exponentially growing computation cost STA brings, the authors leverage the linearized attention technique raised by [19]. Experiments show that STAGNN outperforms SOTA GNN on node classification tasks.", "questions": "1. In section 5.2, the authors claim \"In contrast to MP-GNNs, STAGNN maintains robust performance even when the height of the subtree reaches 100\". Where is the source of \"the performance of MP-GNNs doesn't maintain robust when the height of the subtree reaches 100\"?", "limitations": "No limitations are mentioned in this paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors provide a thorough demonstration of STA's workflow with a theoretical analysis of how STA addresses over-smoothing and over-squishing. Besides. The authors propose an effective computing algorithm, which decreases STA time complexity.\n\n2. The experiment result outperforms SOTA GCN and Graph Transformer.\n\n3. Ablation studies show that STA is beneficial to global attention in graph learning.", "weaknesses": "1. STA's timing performance is unclear. The authors leverage kernelized Softmax to design an efficient algorithm with time complexity O(epsilon) (the number of edges). However, it is hard to compare STA with Graph transformers' (O(N) where N is the number of nodes) from the time complexity aspect, since the number of nodes and edges are independent in an undirected graph. This paper lacks experiments on STA's and Graph Transformers' timing performance. Without evidence, people will worry about if STA can achieve superior performance within a reasonable time limit.\n\n2. Analysis of STA's space complexity is missing. STA efficient algorithm reduces the number of calculations by storing and re-using previous calculation results for future calculation. But there is no discussion on the potential problem of STA's memory cost. A memory cost comparison can better show the efficiency of STA.\n\n3. Hop2Token used in NAGphormer [1] also learns the graph at the level of hops like STA. The experiment results against with NAGphormer does not seem very superior, especially when the experiment is not comprehensive to include a comparison of time and memory cost.\n\n4. Equation (5) doesn't match the context in Figure 1 (a). N is defined as the number of nodes at the start of Section 2. However, figure 1 (a) says the STA_{k} should only attend the k-th hop neighbors, which is conflict with Equation (5).\n\t\n5. Lack of explanation when introducing the rewriting from equation (3) to equation (4) (in Sec. 2.2). It is not obvious how to rewrite a similarity function with a selected kernel and feature map. [2] is about how to rewrite the similarity function using a linear kernel, which should be cited here.\n\n[1] J. Chen, K. Gao, G. Li, and K. He. Nagphormer: Neighborhood aggregation graph transformer for node classification in large graphs. CoRR, abs/2206.04910, 2022.\n\n[2] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive\ntransformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5156–5165. PMLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments and valuable questions. We provide details to clarify your major concerns.\n\n> Q1 & Q2. (Concerns related to time and space complexity) \n\nA1 & A2.\nFirstly, we would like to highlight one point: while we indeed introduce an efficient algorithm, the primary motivation of this algorithm is to **address the implementation challenges associated with STA** (line 136). The goal **is not** to make STA faster or lighter compared to other graph attention mechanisms. \n\nSTA is specifically tailored for graph data with topological structures, which results in a more intricate computation process. We craft the algorithm to ensure the feasibility of STA,  not with the primary intent of outperforming other methods in computational efficiency. Thus, we do not compare time and space complexity with other graph learning methods in our paper. \n\nWe appreciate the reviewer for pointing out the importance of this aspect. Heeding the advice of the reviewer, we conduct an additional experiment focusing on time and space complexity:\n\n|              | Cora|        | Actor|        | Deezer|        |\n|--------------|:------------|:-------|:-----------|:-------|:-----------|:-------|\n|              | Memory(MB)  | Time(s/epoch)| Memory(MB) | Time(s/epoch)|Memory(MB) | Time(s/epoch)|\n|GCN|1,058| 0.007| 1,136 | 0.018| 4,012| 0.103|\n|GraphGPS|3,280| 0.043|  4,583 | 0.075|OOM| OOM|\n|STAGNN|1,263| 0.008|  1,341 | 0.025| 4,896| 0.136|\n\nIt's worth noting that the training time and memory required by STAGNN are similar to GCN. **This is not a coincidence.** This is due to our proposed efficient algorithm for STA (section 3.2) where we leverage **the Message-passing scheme** to compute graph attention, which ensures that **STAGNN has similar time and space complexity compared to vanilla Message-passing GNNs (especially, GCN)**, which is completely acceptable.\n\nAdditionally, we'd like to note that there is potential for making STA **even faster**. Since STA employs a Message-passing scheme to compute attention, it can be combined with existing GNN acceleration techniques (e.g. [1]) to architect an even quicker graph attention methodology. We are willing to delve deeper into it in future work.\n\n> Q3. \"Hop2Token used in NAGphormer also learns the graph at the level of hops like STA. The experiment results against with NAGphormer does not seem very superior, especially when the experiment is not comprehensive to include a comparison of time and memory cost.\"\n\nA3. Since this is a common question raised by multiple reviewers, we provide a unified reply in the Author Rebuttal at the top of this page. We kindly invite you to take a glance at **the second Q&A of the Author Rebuttal**.\n\nFurthermore, we kindly invite you to take a look at the pdf attached in the Author Rebuttal. This pdf clearly illustrates the superior theoretical properties of STA compared to NAGphormer.\n\n> Q4. \"Equation (5) doesn't match the context in Figure 1 (a). N is defined as the number of nodes at the start of section 2. However, figure 1 (a) says the $\\text{STA}_k$ should only attend the $k$-th hop neighbors, which conflicts with Equation (5).\n\nA4. Equation (5) **does match** Figure 1 (a). In fact, Equation (5) can be found at the top-right corner of Figure 1 (a).\n\nWe understand the reviewer's concern regarding the definition of \"N\". Please notice that within Equation (5), $A^k$ is incorporated, which implies $\\text{STA}_k$ should only attend to the $k$-th hop neighbors\".\n\nWe hope that this explanation has addressed the reviewer's concern.\n\n> Q5. \"Lack of explanation when introducing the rewriting from equation (3) to equation (4) (in section 2.2). It is not obvious how to rewrite a similarity function with a selected kernel and feature map. [2] is about how to rewrite the similarity function using a linear kernel, which should be cited here.\"\n\nA5. In fact, [2] has been cited in our paper (line 149). Additionally, we have cited multiple kernelized-related works (line 104,106). We will take the reviewer's suggestion and cite these papers at the beginning of our introduction to kernel methods.\n\n> Q6. \"In section 5.2, the authors claim \"In contrast to MP-GNNs, STAGNN maintains robust performance even when the height of the subtree reaches 100\". Where is the source of \"the performance of MP-GNNs doesn't maintain robust when the height of the subtree reaches 100\"?\"\n\nA6. Our claim about the depth limitation of MP-GNNs stems from the widely accepted consensus within the graph neural network community. In [3], it's mentioned that \"GNNs suffer a model depth limitation—they tend to perform increasingly worse on classifying graph nodes as the model gets deeper.\" Another work [4] highlights that \"GNNs are susceptible to a bottleneck when aggregating messages across a long path.\" (Here, \"long path\" is synonymous with deep subtree) From a more theoretical perspective, the study [5] mentions in Theorem 2 that there's \"an exponential information loss of GCNs in terms of the layer size.\"\n\nIn fact, there's no need for such a large number as 100. 7 is enough. [6] has empirically observed that GCN, when evaluated on the Cora dataset, exhibits an accuracy of approximately 37% upon reaching a subtree depth of 7. In contrast, our STAGNN showcases a performance of 89.2% at the same subtree depth. \n\n\n[1]. H. Zeng, et al. Graphsaint: Graph sampling based inductive learning method. ICLR 2020.\n\n[2]. A. Katharopoulos, et al. Transformers are rnns: Fast autoregressive transformers with linear attention. PMLR, 2020.\n\n[3]. K. Zhou, et al. Understanding and resolving performance degradation in deep graph convolutional networks. CIKM 2021.\n\n[4]. U. Alon, et al. On the Bottleneck of Graph Neural Networks and its Practical Implications. ICLR 2021.\n\n[5]. K. Oono, et al. Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. ICLR 2020.\n\n[6]. M. Liu, et al. Towards deeper graph neural networks. KDD 2020."}, {"review_id": "Yr7PUYALgp", "reviewer": "Reviewer_smZ8", "summary": "The present manuscript proposes a novel graph attention layer, that lies in the middle of the local aggregation scheme of message passing and the global (non-structured) nature of a full-attention. The newly proposed Subtree Attention constructs for each node the similarity pairs of key and query matrices from its k-hop neighborhood. \n\nGiven the transformed matrices, they perform an efficient aggregation throughout the neighborhood hops by considering the random walk adjacency matrix as the transition matrix that is used in the graph attention. They show that given the utilization of the random walk matrix combined with a kernelized softmax function, they reduce the computational cost to $\\mathcal{O}(\\mathcal{E})$, where $\\mathcal{E}$ the number of graph edges.\n\nThe authors prove theoretically that given a set of assumptions, the Subtree attention can approximate the global attention, showing that the proposed layer is able to tackle over-smoothing and over-squashing problems that come with MPNNs, while bridging the local with the global attention.\n\nTheir experimental study shows that STAGNN (their method) has a superior performance with respect to the state-of-the-art graph transformer models.\n\n", "questions": "- Can the authors show any empirical evidence that the examined datasets can show a need for deeper models? In other words, do these datasets consist of any long-range interactions, or do they consist only of shallow interactions?", "limitations": "The authors make a discussion on the supplementary material over possible extensions and ways to improve the STAGNN by incorporating it to more models.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The Subtree attention is a novel and interesting idea on bridging the notions of the local neighborhood attention with the global graph attention.\n- The theoretical study shows that subtree attention can approximate global attention, which can be very useful for treating over-smoothing in MPNNs.\n- Empirically, the model STAGNN seems to outperform its competitor baselines. \n- The paper is well-written and easy to follow.", "weaknesses": "- The experimental study is limited to six benchmark datasets, which it is unclear whether long-range interactions play any role, or instead shallower architectures are, also, effective. \n- It would be very useful testing STAGNN, also, in other datasets (e.g. from OGB), that reportedly depend on long-range interactions, or in synthetic datasets, so that the impact of the proposed model can be shown better. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's constructive feedback and positive remarks on our work. We provide the following detailed responses to your major concerns.\n\nIn order to organize our response logically, we jointly address the first weakness and the question raised by the reviewer.\n\n>Q1. \"The experimental study is limited to six benchmark datasets, which it is unclear whether long-range interactions play any role, or instead shallower architectures are, also, effective.\"\n\n> \"Can the authors show any empirical evidence that the examined datasets can show a need for deeper models? In other words, do these datasets consist of any long-range interactions, or do they consist only of shallow interactions?\"\n\nA1. In fact, we have conducted evaluations on 10 datasets in our paper. Kindly refer to Figure 3 (4 datasets) and Table 1 (6 datasets). These datasets come from real-world scenarios. Given the intricate relationships in real-world networks, it appears challenging to definitively state whether they rely more on long-range or short-range relationships or a blend of both. However, we would like to note that these datasets are frequently used as standards in assessing the capabilities of GNNs. In fact, numerous graph neural network methodologies [1,2,3] have utilized these datasets (or part of them) as performance benchmarks. Testing on these datasets offers a fair representation of our model's performance. \n\nAt the same time, we do have a piece of indirect evidence that reflects the level of involvement of the long-range interaction: **the GPR weights** within STAGNN. GPR weights are the weights learned for each hop when STAGNN adaptively aggregates the information across different hops. They can be considered as a series of 'importance scores' that the model deduces during training for different hops.  \n\nLet us give an example. The GPR weights for the Cora and Actor datasets are visualized in Figure 7 (Appendix section F). For instance, in the Cora dataset, we observe an increasing trend in the GPR weights over hops ranging from 0 to 10. This suggests that the model values relationships that span longer distances than relationships of just one or two hops. In contrast, for the Actor dataset, there's a noticeable decline in GPR weights over hops ranging from 0 to 10, highlighting the model's preference for shorter relationships. In summary, the GPR weights adaptively learned by STAGNN during the training process could serve as indirect evidence of which order of neighbors (longer or shorter relationships) the model primarily focuses on. We believe this also showcases the strong interpretability of STA.\n\nWe appreciate the valuable question from the reviewer. While these datasets might not clearly show STAGNN's ability to handle long-range interactions, we've added an additional experiment using the Long Range Graph Benchmark dataset (see our answer to question 2). \n\nFinally, we would like to note that STA isn't born for handling long-range interactions. STA is born to be an all-around player. We theoretically prove that STA includes both local and global attention (approximately), as well as every intermediate step between local and global attention (line 207), and thus STA is more like an all-around hand for both long and short interactions.\n\n>Q2. It would be very useful testing STAGNN, also, in other datasets (e.g. from OGB), that reportedly depend on long-range interactions, or in synthetic datasets, so that the impact of the proposed model can be shown better.\n\nA2. We greatly appreciate the suggestion provided by the reviewer. Based on this recommendation, we opt for two graph datasets from the **Long Range Graph Benchmark (LRGB)** to conduct an experiment [4].  LRGB includes 5 graph learning datasets that require long-range interactions to achieve strong performance in a given task. \n\nWe aim to demonstrate that STA (rather than STAGNN) possesses the capability to capture long-range interactions. Thus, we replaced the self-attention module in the vanilla transformer with our STA module and employed this new STA+Transformer model for long-range interaction learning tasks.\n\nWe elaborate on this experiment in detail in the Author Rebuttal. We kindly invite you to take a look at the **third Q&A in the Author Rebuttal**. Below are the experimental results:\n\n|Long Range Graph Benchmark (LRGB) | | Peptides-func (graph classification)| Peptides-struct (graph regression) | \n|--------------|------------:|------------:|-----------:|\n|              | #Params. |  AP ↑  |  MAE ↓ | \n| GCN|508k| 59.30±0.23| 34.96±0.13 | \n| GCNII|505k| 55.43±0.78 | 34.71±0.10 | \n| GINE |476k| 54.98±0.79| 35.47±0.45 | \n| GatedGCN |509k| 58.64±0.77|34.20±0.13 | \n| Transformer+LapPE |488k| 63.26±1.26| 25.29±0.16 | \n| **STA**+Transformer+LapPE |488k|**65.83**±**0.94**| **24.16**±**0.21**| \n\nBy solely substituting the self-attention module with STA, we achieved noticeable improvements in performance. This highlights STA's ability to handle long-range interactions effectively.\n\n\n[1]. E. Chien, et al. Adaptive Universal Generalized PageRank Graph Neural Network. ICLR 2021.\n\n[2]. Q. Wu, et al. NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification. NeurIPS 2022.\n\n[3]. J. Chen, et al. NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphs. ICLR 2023.\n\n[4]. VP. Dwivedi, et al. Long Range Graph Benchmark. NeurIPS 2022 Track on Datasets and Benchmarks."}, {"review_id": "XHT7B6Zym8", "reviewer": "Reviewer_K8yB", "summary": "This paper introduces a novel multi-hop graph attention mechanism called SubTree Attention (STA) to address the limitations of both local and global attention in Graph Neural Networks (GNNs). STA allows the root node to attend directly to further neighbors in the subtree, enabling it to gather information from the entire rooted subtree within one layer. This mechanism avoids issues associated with deep architectures using local attention layers and captures the hierarchical neighborhood structure more effectively than global attention. The authors propose an efficient algorithm based on kernelized softmax to calculate attention among multi-hop neighbors. They also provide a theoretical analysis of STA, demonstrating its convergence to a degree-masked global self-attention. The paper introduces STAGNN, a multi-hop graph attention network that integrates the STA module into decoupled GCNs, achieving superior performance compared to existing GNNs and graph transformers. Experimental evaluations on node classification datasets confirm the effectiveness of STA and the competitive performance of STAGNN.", "questions": "* How is Graph Transformer applied to node-classification? Is it the pipeline described? Because graph for node-classification is larger than graphs for graph-level tasks? Maybe subgraph sampling is needed?\n* Two questions about novelty. The STAGNN is an attention-based method to combine multi-hop information of graphs?  The difference of using vanilla transformer and STA is that STA has kernalized attention? If not, can you briefly describe the novelty here?\n* Is the multi-head attention applied in STAGNN?\n* If the attention will converge to degree-based global attention, why will STA work? Is the convergence achieved in training?", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The authors propose a strategy to incorporate attention in multi-hop propagation GNNs and get good performances.\n* Good theoretical analysis is provided.", "weaknesses": "* Lack of baselines (BernNet,...). The authors ignore some important baselines of propagation-based GNNs, including BernNet and successive work.\n* The kernalized attention is a method from Transformer variants. Need citation.\n* Datasets are not large, should include OGB-datasets, at least ogbn-arxiv", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the reviewer's thoughtful feedback. We provide the following detailed responses to your major concerns.\n\n> Q1. \"Lack of baselines (BernNet,..) The authors ignore some important baselines of propagation-based GNNs, including BernNet and successive work.\"\n\nA1. Indeed, BernNet is a compelling work in the realm of spectral GNNs. We conduct a supplementary experiment where we take BernNet as an additional baseline:\n\nMethod  | Pubmed| CoraFull | Computer | Photo  | CS | Physics\n- | - | - | - | - | - | -\nBernNet | 89.12±0.33        | 67.88±0.29          | 88.8±0.34           | 94.21±0.16       | 94.64±0.10    | 96.23±0.11\nSTAGNN  | 90.46±0.22        | 72.65±0.36          | 91.72±0.30          | 95.64±0.27       | 95.77±0.16    | 97.09±0.18\n\n\nWe can observe that STAGNN consistently outperforms BernNet. This underscores the exceptional performance of STA.\n\n> Q2. \"The kernelized attention is a method from Transformer variants. Need citation.\"\n\nA2. Indeed, we have cited several kernelized-related papers (line 104, 106, 149). We will take the reviewer's suggestion and offer a clearer introduction to kernelized methods.\n\nAdditionally, we kindly invite the reviewer to take a glance at **the first Q&A in the Author Rebuttal** for a more detailed explanation of the role kernelized method plays in STA.\n\n\n> Q3. \"Datasets are not large, should include OGB-datasets, at least ogbn-arxiv\"\n\nA3. We appreciate the suggestion from the reviewer. As recommended, we conduct an additional experiment on the ogbn-arxiv dataset. We treat ogbn-arxiv as an undirected graph. The results of the experiment are as follows:\n\nMethod        | ogbn-arxiv\n-|-\nMLP           | 55.50±0.23\nGCN           | 71.74±0.29\nJKNet         | 72.19±0.21\nUniMP         | 73.11±0.20\nSTAGNN        | 75.42±0.35\n\n\nWe can observe that even on a larger-scale dataset, STAGNN still maintains highly competitive performance.\n\n> Q4. \"How is Graph Transformer applied to node classification? Is it the pipeline described? Because graph for node-classification is larger than graphs for graph-level tasks? Maybe subgraph sampling is needed?\"\n\nA4. Indeed, datasets used for node classification are often larger than those for graph classification. There exist methods using graph transformers in conjunction with subgraph sampling for node classification tasks [1, 2]. In our experiments, however, we aim for a fair comparison between STA and vanilla self-attention in the context of topological data structures. Therefore, our Graph Transformer baselines do not employ these sampling techniques.\n\nWhile different graph transformers in our baselines have their unique designs, such as learnable positional encodings [3] and GNNs as auxiliary [4], these models employ a straightforward pipeline where each node is treated as a distinct token. They are then fed into the transformer model, yielding a final representation for each node.\n\n> Q5. \"Two questions about novelty. The STAGNN is an attention-based method to combine multi-hop information of graphs? The difference of using vanilla transformer and STA is that STA has kernalized attention? If not, can you briefly describe the novelty here?\"\n\nA5. This is a valuable question. We would like to highlight that STA is very different from other graph attention approaches, and **this differentiation goes beyond its use of kernelized attention**. Given that this is a critical issue raised by multiple reviewers, we have provided a detailed response in the Author Rebuttal. We kindly invite the reviewer to refer to **the first Q&A in the Author Rebuttal** for a thorough explanation.\n\nFurthermore, we kindly invite you to review the PDF attached to the Author Rebuttal above. In that pdf, we visually illustrate the novel parts of STA.\n\n> Q6. \"Is the multi-head attention applied in STAGNN?\"\n\nA6. Yes, in STAGNN, we employ a carefully crafted multi-head attention mechanism (section 3.3). We have also conducted an ablation study on multi-head attention within STAGNN in Appendix G.1 and demonstrated its efficacy.\n\n> Q7. \"If the attention will converge to degree-based global attention, why will STA work?\"\n\nA7. We fully understand the reviewer's concern. We would like to clarify that **it's not** that STA converges to degree-based global attention, but rather, $\\text{STA}_k$ converges to degree-based global attention as $ k $ increases (section 3.4). And STA contains $\\text{STA}_0$, $\\text{STA}_1$, $\\text{STA}_2$ ... And STA can finally employ various hop-aggregation strategies to adaptively adjust the weights of different hops (e.g. GPR-like aggregation used in STAGNN).\n\nWe highlight that $\\text{STA}_1$ functions as local attention, while $\\text{STA}_k$ converges to degree-masked global attention. Thus **STA theoretically includes every intermediate step between local and global attention** (line 207).\n\n> Q8. \"Is the convergence achieved in training?\"\n\nA8. In fact, this convergence is intrinsic and remains unaffected by the training process. Consider both $\\text{STA}_k (Q,K,V)$ and $\\text{SA} (Q,K,V)$ as functions of $Q, K, V$. As illustrated in section 3.4, $\\text{STA}_k (Q,K,V)$ converges to $\\text{SA} (Q,K,V)$ as $k$ increases. This convergence is solely determined by $k$ and is unaffected by the values of $Q, K, V$. Conversely, the training process only modifies $Q, K, V$.\n\nTheoretically, this convergence stems from the properties of random walks on graphs. If a random walk repeats infinitely, then the probability of landing on a specific node depends on the degree of that node. This explains why  $\\text{STA}_k$ converges to degree-based global attention.\n\n\n[1]. Q. Dai, et al. A Self-Attention Network based Node Embedding Model. ECML\n 2020.\n\n[2]. Z. Zhang, et al. Hierarchical Graph Transformer with Adaptive Node Sampling. NeurIPS 2022.\n\n[3]. D. Kreuzer, et al. Rethinking Graph Transformers with Spectral Attention. NeurIPS 2021.\n\n[4]. L. Rampášek, et al. Recipe for a General, Powerful, Scalable Graph Transformer. NeurIPS 2022."}, {"review_id": "U6jo9Qbk3Y", "reviewer": "Reviewer_ds4S", "summary": "The paper introduces Subtree Attention (STA), a new graph attention mechanism that overcomes limitations of existing mechanisms in graph learning. STA combines fully-attentional structure with rooted subtrees, approximating masked global attention under extreme settings. By computing attention weights among multi-hop neighbors directly, STA addresses previous mechanism problems. The authors present STAGNN, a graph neural network that utilizes STA and a hop-aware attention strategy. Extensive evaluations on ten node classification datasets show that STA-based models outperform existing graph transformers and mainstream GNNs.", "questions": "Why do you only evaluate STA on the node classification task? What about Graph Classification?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1, This paper proposes a novel multi-hop graph attention mechanism, Subtree Attention (STA), which bridges the fully-attentional structure and the rooted subtree. \n2, By incorporating kernelized softmax, they develop an optimized approach for STA, resulting in a linear time complexity.\n3, Experiments have shown that STAGNN achieves good performance on Node classification tasks.", "weaknesses": "The performance improvement of NAGphormer is limited, as demonstrated in Table 1. For instance, when evaluating photos, NAGphormer achieved a score of 95.49±0.11, while STAGNN achieved 95.64±0.27. The difference between 95.49 and 95.64 is only 0.15, which is smaller than 0.27. A similar trend is observed in the case of CS. Furthermore, in Physics, STAGNN performs worse than NAGphormer. Additionally, it is worth noting that the variance of STAGNN appears to be much larger than that of NAGphormer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback on our manuscript. We provide the following detailed responses to your major concerns.\n\n> Q1. \"The performance improvement of NAGphormer is limited, as demonstrated in Table 1. For instance, when evaluating photos, NAGphormer achieved a score of 95.49±0.11, while STAGNN achieved 95.64±0.27. The difference between 95.49 and 95.64 is only 0.15, which is smaller than 0.27. A similar trend is observed in the case of CS. Furthermore, in Physics, STAGNN performs worse than NAGphormer. Additionally, it is worth noting that the variance of STAGNN appears to be much larger than that of NAGphormer.\"\n\nA1. Thank you for bringing up this point.  Firstly, of the ten node classification datasets we tested, our model achieved the highest average scores on nine. This underscores the competitive performance of STAGNN.\n\n| Method     | Pubmed      | CoraFull   | Computer   | Photo      | CS         | Physics    |\n|------------|------------:|-----------:|-----------:|-----------:|-----------:|-----------:|\n| NAGphormer | 89.70±0.19  | 71.51±0.13 | 91.22±0.14 | 95.49±0.11 | 95.75±0.09 | 97.34±0.03 |\n| STAGNN     | 90.46±0.22  | 72.65±0.36 | 91.72±0.30 | 95.64±0.27 | 95.77±0.16 | 97.09±0.18 |\n\nAdditionally, the performance of STAGNN and NAGphormer is tabulated above. From the table, STAGNN and NAGphormer have comparable scores on Photo, CS, and Physics, where both models already achieve very high scores (above 95%). In reality, due to inherent noise in graph datasets, pushing for even higher accuracy in such scenarios can be challenging (To the best of our knowledge, we have not observed higher scores on these three datasets in other literature). A more apt observation might be that both STAGNN and NAGphormer perform exceedingly well on these datasets.\n\nFurthermore, we would like to note that STAGNN is merely a basic application of STA. As pointed out in line 249 in our manuscript, STAGNN employs a rather simple structure. Despite this simplicity, the fact that STAGNN can surpass NAGphormer in performance further showcases the superiority of the STA mechanism.\n\nLastly, we would like to highlight the theoretical differences between STA and NAGphormer. In brief:\n- Hop2Token initially aggregates the node representations. Subsequently, it leverages these representations to derive keys, queries, and values.\n- STA, on the contrary, computes the keys, queries, and values for each node initially and subsequently propagates the keys and values using a message-passing mechanism.\n\nThe approach employed by Hop2Token, which involves message propagation on node representations, remains susceptible to challenges like over-smoothing and over-squashing. In contrast, STA's methodology of adapting graph attention through the message-passing paradigm remains immune to such issues. This ensures STA with superior theoretical properties (and, correspondingly, empirical results).\n\nWe have attached a PDF in the Author Rebuttal which clearly illustrates the distinctions between STAGNN and NAGphormer. We kindly invite you to take a glance at the attached PDF.\n\n> Q2. \"Why do you only evaluate STA on the node classification task? What about Graph Classification?\"\n\nA2. Thanks for the reviewer's suggestion.  STAGNN is mainly designed for node classification tasks, hence we do not evaluate it on graph classification. However, STA, as a promising alternative to self-attention in the graph domain, can naturally be applied to graph classification tasks. We have conducted an additional experiment. We replace the self-attention module in the vanilla transformer with our STA module and apply this new STA+Transformer model for graph classification tasks. \n\n|Long Range Graph Benchmark (LRGB) | | Peptides-func (graph classification)| Peptides-struct (graph regression) | \n|--------------|------------:|------------:|-----------:|\n|              | #Params. |  AP ↑  |  MAE ↓ | \n| GCN|508k| 59.30±0.23| 34.96±0.13 | \n| GCNII|505k| 55.43±0.78 | 34.71±0.10 | \n| GINE |476k| 54.98±0.79| 35.47±0.45 | \n| GatedGCN |509k| 58.64±0.77|34.20±0.13 | \n| Transformer+LapPE |488k| 63.26±1.26| 25.29±0.16 | \n| **STA**+Transformer+LapPE |488k|**65.83**±**0.94**| **24.16**±**0.21**| \n\nWe notice that by solely substituting the self-attention module with STA, we achieve noticeable improvements in performance. This demonstrates STA's capability in handling graph-level tasks.\n\nA more detailed analysis of this experiment can be found at the third Q&A in the Author Rebuttal.\n\n"}]

Author rebuttal:
We extend our sincere gratitude to the reviewers for their invaluable feedback on our work.

We are delighted to see comments such as "**The Subtree attention is a novel and interesting idea**" (Reviewer smZ8), "**Good theoretical analysis is provided.**" (Reviewer K8yB), and "**can be very useful for treating over-smoothing in MPNNs**" (Reviewer smZ8).

We have carefully considered each comment. Below, we address three common questions raised by some reviewers:

> Q1. How does STA differ from the vanilla graph transformer and NAGphormer? Where is its novelty?

A1. The novelty of STA proposed in this paper can be summarized in two main points.

Firstly, STA presents a novel method for graph attention calculation.  We demonstrate that **STA theoretically includes both local and global attention, as well as every intermediate step between local and global attention** (line 207).  $\text{STA}_1$ functions as local attention, while $\text{STA}_k$ converges to degree-based global attention as $ k $ increases (section 3.4). And STA contains $\text{STA}_0$, $\text{STA}_1$ ...  $\text{STA}_k$ and thus can better capture the hierarchical neighborhood structure than existing graph attention methods.

We provide an intuitive comparison of STA, vanilla graph transformer, and NAGphormer in the attached PDF (You can view it at the bottom of this Author Rebuttal). **We kindly invite you to take a minute to review the attached PDF**. 

Secondly, while the idea behind STA is straightforward, its computation is more intricate compared to other vanilla graph attention. We innovatively use kernelized methods to address this issue. To be clear, we **are not** the pioneers in applying kernelized methods in the graph attention domain, but **we are the first to fuse message-passing frameworks with graph attention computations using kernelized methods**. In simpler terms, we initially compute the key, value, and query for each node, followed by the propagation of the key and value across the graph. This idea of computing graph attention is novel and, in our belief, holds vast potential for extensions, such as integration with edge sampling methods.

> Q2. STAGNN doesn't seem to have achieved significant improvements over NAGphormer?

A2. Firstly, of the ten node classification datasets we tested, our model achieved the highest average scores on nine. This underscores the competitive performance of STAGNN.
| Method     | Pubmed      | CoraFull   | Computer   | Photo      | CS         | Physics    |
|------------|------------:|-----------:|-----------:|-----------:|-----------:|-----------:|
| NAGphormer | 89.70±0.19  | 71.51±0.13 | 91.22±0.14 | 95.49±0.11 | 95.75±0.09 | 97.34±0.03 |
| STAGNN     | 90.46±0.22  | 72.65±0.36 | 91.72±0.30 | 95.64±0.27 | 95.77±0.16 | 97.09±0.18 |

Additionally, the performance of STAGNN and NAGphormer is tabulated above. From the table, STAGNN and NAGphormer have comparable scores on Photo, CS, and Physics, where both models already achieve very high scores (above 95%). In reality, due to inherent noise in graph datasets, pushing for even higher accuracy in such scenarios can be challenging (To the best of our knowledge, we have not observed higher scores on these three datasets in other literature). A more apt observation might be that both STAGNN and NAGphormer perform exceedingly well on these datasets.

Lastly, we would like to note that STAGNN is merely a basic application of STA. As pointed out in line 249 in our manuscript, STAGNN employs a rather simple structure. Despite this simplicity, the fact that STAGNN can surpass NAGphormer in performance further showcases the superiority of the STA mechanism.

> Q3. The main contribution of this paper is to accelerate Graph Attention with kernelized methods?

A3. The answer is **NO**. We emphasize that our primary contribution is **SubTree Attention**. The motivation behind STA is its superior capability and robust theoretical properties. Beyond this main objective, we employ a kernelized approach to devise a linear complexity algorithm for computing STA and introduce STAGNN. Compared to STA, both the efficient computation of STA and STAGNN are secondary contributions.

STA can do more. STA can be employed to enhance existing attention-based graph methods. This potential is also echoed by "Ablation studies show that STA is beneficial to global attention in graph learning." (Reviewer sk7A). 

To better showcase the potential of STA, we conduct an additional experiment. We replace the self-attention module in the vanilla transformer with our STA module and apply this new **STA+Transformer** model for graph classification tasks. The experimental results are as follows:

|Long Range Graph Benchmark (LRGB) | | Peptides-func (graph classification)| Peptides-struct (graph regression) | 
|--------------|------------:|------------:|-----------:|
|              | #Params. |  AP ↑  |  MAE ↓ | 
| GCN|508k| 59.30±0.23| 34.96±0.13 | 
| GCNII|505k| 55.43±0.78 | 34.71±0.10 | 
| GINE |476k| 54.98±0.79| 35.47±0.45 | 
| GatedGCN |509k| 58.64±0.77|34.20±0.13 | 
| Transformer+LapPE |488k| 63.26±1.26| 25.29±0.16 | 
| **STA**+Transformer+LapPE |488k|**65.83**±**0.94**| **24.16**±**0.21**|

By solely replacing the self-attention module with STA, we achieve noticeable improvements in performance. This experiment shows us the capabilities of STA in three different dimensions:
- STA can capture **long-range relationships** in these Long Range Graph Benchmarks. (Reviewer smZ8)
- STA can be applied to **graph-level** tasks and achieve competitive performance. (Reviewer ds4S)
- STA, as **a strong alternative to self-attention in the graph domain**, has the potential to replace and enhance existing attention-based graph learning methods.

---

We conclude by kindly inviting you to take a glance at this attached pdf. It intuitively illustrates how STA differs from vanilla graph attention methods.

Meta-review: The authors design a novel Graph Transformer called STA-GNN (SubTree-attention-GNN) to address the over-smoothing and over-squashing in the message-passing scheme. The newly proposed Subtree Attention constructs for each node the similarity pairs of key and query matrices from its k-hop neighborhood. Experimental evaluations on node classification datasets confirm the effectiveness of STA and the competitive performance of STAGNN.
The reviewers agree that the paper is well-written and easy to follow. The idea of subtree attention is quite interesting and novel. The theoretical study shows that subtree attention can approximate global attention, which can be very useful for treating over-smoothing in MPNNs. Some concerns regarding the evaluation came up in the reviews but the additional experiments provided could address most of these concerns. After the discussions, the reviewers reached a consensus to accept this paper.
Decision: Accept (poster)