SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {v} _ {1} = \frac {\mathbf {D} ^ {\frac {1}{2}} \overrightarrow {\mathbf {1}}}{\| \mathbf {D} ^ {\frac {1}{2}} \overrightarrow {\mathbf {1}} \| _ {2}} \tag {15}
$$
Notice that  $\| \mathbf{D}^{\frac{1}{2}}\overrightarrow{\mathbf{1}}\|_2 = \sqrt{\sum_{i=1}^{N}d(i)}$ . Using the fact that  $\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_n$  form an orthonormal basis and  $\mathbf{D}^{-\frac{1}{2}}\mathbf{p}_i = \sum_{i=1}^{N}c_i\mathbf{v}_i$ , we then have:
$$
\begin{array}{l} c _ {1} = \left(\mathbf {D} ^ {- \frac {1}{2}} \mathbf {p} _ {\mathbf {i}}\right) ^ {T} v _ {1} \\ = \mathbf {p} _ {\mathbf {i}} ^ {T} \mathbf {D} ^ {- \frac {1}{2}} \frac {\mathbf {D} ^ {\frac {1}{2}} \overrightarrow {\mathbf {1}}}{\| \mathbf {D} ^ {\frac {1}{2}} \overrightarrow {\mathbf {1}} \| _ {2}} \\ = \frac {\mathbf {p} _ {\mathbf {i}} ^ {T} \overrightarrow {\mathbf {1}}}{\left\| \mathbf {D} ^ {\frac {1}{2}} \overrightarrow {\mathbf {1}} ^ {\prime} \right\| _ {2}} \tag {16} \\ = \frac {1}{\| \mathbf {D} ^ {\frac {1}{2}} \overrightarrow {\mathbf {T}} \| _ {2}} \\ = \frac {1}{\sqrt {\sum_ {i = 1} ^ {N} d (i)}} \\ \end{array}
$$
Notice that  $\mathbf{A}_{\mathrm{sym}}^k\mathbf{v_1} = \lambda_1^k\mathbf{v_1}$  and  $\lambda_{1} = 1$  . Thus we have:
$$
\begin{array}{l} c _ {1} \mathbf {D} ^ {\frac {1}{2}} \mathbf {A} _ {\mathrm {s y m}} ^ {k} \mathbf {v _ {1}} = c _ {1} \mathbf {D} ^ {\frac {1}{2}} \lambda_ {1} ^ {k} \mathbf {v _ {1}} \\ = c _ {1} \mathbf {D} ^ {\frac {1}{2}} \mathbf {v} _ {1} \\ = \frac {1}{\sqrt {\sum_ {i = 1} ^ {N} d (i)}} \mathbf {D} ^ {\frac {1}{2}} \frac {\mathbf {D} ^ {\frac {1}{2}} \overrightarrow {\mathbf {1}}}{\| \mathbf {D} ^ {\frac {1}{2}} \overrightarrow {\mathbf {1}} \| _ {2}} \tag {17} \\ = \frac {\mathbf {D} \overrightarrow {\mathbf {1}}}{\sum_ {i = 1} ^ {N} d (i)} \\ = \pi \\ \end{array}
$$
Considering Equation 14 and Equation 17, we have:
$$
\hat {\mathbf {A}} ^ {k} \mathbf {p} _ {\mathbf {i}} = \boldsymbol {\pi} + \sum_ {i = 2} ^ {N} c _ {i} \mathbf {D} ^ {\frac {1}{2}} \mathbf {A} _ {\mathrm {s y m}} ^ {k} \mathbf {v} _ {\mathbf {i}} \tag {18}
$$
and immediately:
$$
\begin{array}{l} \| \hat {\mathbf {A}} ^ {k} \mathbf {p _ {i}} - \boldsymbol {\pi} \| _ {2} ^ {2} = \| \sum_ {i = 2} ^ {N} c _ {i} \mathbf {D} ^ {\frac {1}{2}} \mathbf {A} _ {\mathrm {s y m}} ^ {k} \mathbf {v _ {i}} \| _ {2} ^ {2} \\ = \left\| \mathbf {D} ^ {\frac {1}{2}} \sum_ {i = 2} ^ {N} c _ {i} \mathbf {A} _ {\text {s y m}} ^ {k} \mathbf {v} _ {\mathrm {i}} \right\| _ {2} ^ {2} \tag {19} \\ \leq | | | \mathbf {D} ^ {\frac {1}{2}} | | | _ {p} ^ {2} \| \sum_ {i = 2} ^ {N} c _ {i} \mathbf {A} _ {\mathrm {s y m}} ^ {k} \mathbf {v} _ {\mathbf {i}} \| _ {2} ^ {2} \\ \end{array}
$$
where  $||\mathbf{D}^{\frac{1}{2}}||_p = \sup_{x\in \mathbb{R}^N}\frac{\|\mathbf{D}^{\frac{1}{2}}x\|_2}{\|x\|_2} = \sqrt{d_{\max}}$ . Thus we have:
$$
\left\| \hat {\mathbf {A}} ^ {k} \mathbf {p} _ {\mathbf {i}} - \boldsymbol {\pi} \right\| _ {2} ^ {2} \leq d _ {\max } \left\| \sum_ {i = 2} ^ {N} c _ {i} \mathbf {A} _ {\text {s y m}} ^ {k} \mathbf {v} _ {\mathbf {i}} \right\| _ {2} ^ {2} \tag {20}
$$
And using the fact that  $\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_n$  are orthonormal and  $1 - \hat{\lambda} = \max \left\{\lambda_2, |\lambda_N|\right\} = \max \left\{|\lambda_2|, |\lambda_3|, \ldots, |\lambda_N|\right\}$ , we then have:
$$
\begin{array}{l} \| \hat {\mathbf {A}} ^ {k} \mathbf {p _ {i}} - \boldsymbol {\pi} \| _ {2} ^ {2} \leq d _ {\max } \| \sum_ {i = 2} ^ {N} c _ {i} \lambda_ {i} ^ {k} \mathbf {v _ {i}} \| _ {2} ^ {2} \\ = d _ {\max } \sum_ {i = 2} ^ {N} c _ {i} ^ {2} \lambda_ {i} ^ {2 k} \tag {21} \\ \leq d _ {\max } (1 - \hat {\lambda}) ^ {2 k} \sum_ {i = 2} ^ {N} c _ {i} ^ {2} \\ = d _ {\max } (1 - \hat {\lambda}) ^ {2 k} \| \mathbf {D} ^ {- \frac {1}{2}} \mathbf {p} _ {\mathrm {i}} \| _ {2} ^ {2} \\ \end{array}
$$
Notice that  $\| \mathbf{D}^{-\frac{1}{2}}\mathbf{p_i}\| _2^2\leq ||\mathbf{D}^{-\frac{1}{2}}||_p^2\| \mathbf{p_i}\| _2^2 = \frac{1}{d_{\min}}$  . Therefore:
$$
\left\| \hat {\mathbf {A}} ^ {k} \mathbf {p} _ {\mathbf {i}} - \boldsymbol {\pi} \right\| _ {2} ^ {2} \leq \frac {d _ {\operatorname* {m a x}}}{d _ {\operatorname* {m i n}}} (1 - \hat {\lambda}) ^ {2 k} \tag {22}
$$
$$
\begin{array}{l} \| \hat {\mathbf {A}} ^ {k} \mathbf {p} _ {i} - \boldsymbol {\pi} \| _ {2} \leq \frac {d _ {\max}}{d _ {\min}} (1 - \hat {\lambda}) ^ {k} \\ \leq \sqrt {N} (1 - \hat {\lambda}) ^ {k} \tag {23} \\ \leq \sqrt {N} e ^ {- k \hat {\lambda}} \\ \end{array}
$$
Using Cauchy-Schwarz, we then have:
$$
\left\| \hat {\mathbf {A}} ^ {k} \mathbf {p} _ {\mathbf {i}} - \boldsymbol {\pi} \right\| _ {1} \leq \sqrt {N} \| \hat {\mathbf {A}} ^ {k} \mathbf {p} _ {\mathbf {i}} - \boldsymbol {\pi} \| _ {2} \leq N e ^ {- k \hat {\lambda}} \tag {24}
$$
In conclusion, given an arbitrarily small positive number  $\epsilon$ , for all  $k_{0}$  greater than or equal to  $\frac{1}{\lambda} \log \frac{N}{\epsilon}$ , the L1 norm of the difference between  $\hat{\mathbf{A}}^k \mathbf{p_i}$  and the vector  $\pi$  is less than or equal to  $\epsilon$ . This result establishes that  $\frac{1}{\lambda} \log \frac{N}{\epsilon}$  indeed serves as an upper bound.
Notice that the vector  $\mathbf{p_i}$  is an arbitrary distribution. Thus, we may consider  $\mathbf{p_i}$  to be one of the  $i^{\mathrm{th}}$  unit basis vector in the  $N$ -dimensional space:  $\{\mathbf{p_1},\mathbf{p_2},\dots ,\mathbf{p_N}\}$ , where each vector has only one element equal to 1 (the  $i^{\mathrm{th}}$  element) and all other elements equal to 0. Thus we have  $\| \hat{\mathbf{A}}^k\mathbf{p_j} - \boldsymbol {\pi}\| _1 = \sum_{i = 1}^N |\hat{\mathbf{A}}_{ij}^k -\boldsymbol {\pi}_i|$ . Then given  $\epsilon >0$ , we have that:
$$
\forall j \in [   [ 1, N ]   ], \forall k \geq \frac {1}{\lambda} \log \frac {N}{\epsilon}, \sum_ {i = 1} ^ {N} | \hat {\mathbf {A}} _ {i j} ^ {k} - \boldsymbol {\pi} _ {i} | \leq \epsilon \tag {25}
$$
which demonstrates immediately the first part of Theorem 1:
$$
\forall i, j \in [ 1, N ] ^ {2}, \forall k \geq \frac {1}{\hat {\lambda}} \log \frac {N}{\epsilon}, | \hat {\mathbf {A}} _ {i j} ^ {k} - \boldsymbol {\pi} _ {i} | \leq \epsilon \tag {26}
$$
B.2 Proof for Equation 11
In this subsection, we prove the following results: if  $\mathbf{V}$  is computed by  $\mathbf{V} = \sigma (\mathbf{X}\mathbf{W}_V)$  where  $\sigma$  is a non-negative activation function, then:
$$
\forall i, j \in [ 1, N ] ^ {2}, \forall \eta \in ] 0, 1 [, \exists K _ {1} \in \mathbb {N}, \forall k > K _ {1}, \frac {1 - \eta}{1 + \eta} \leq \frac {\mathrm {S T A} _ {k} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) _ {i j}}{\mathrm {S A} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) _ {i j}} \leq \frac {1 + \eta}{1 - \eta}
$$
This result can, indeed, be viewed as a straightforward corollary of Equation (10). The crucial prerequisite is that all elements of the vector  $\mathbf{V}$  must be positive. Importantly, there are no specific
requirements imposed on the non-negative activation function  $\sigma$ , meaning it can be any function that ensures non-negativity.
$\mathrm{STA}_k$  and SA are defined as follows:
$$
\operatorname {S T A} _ {k} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) _ {i:} = \frac {\sum_ {j = 1} ^ {N} \hat {\mathbf {A}} _ {i j} ^ {k} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {j :}\right) \mathbf {V} _ {j :}}{\sum_ {j = 1} ^ {N} \hat {\mathbf {A}} _ {i j} ^ {k} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {j :}\right)} \tag {27}
$$
$$
\operatorname {S A} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) _ {i:} = \frac {\sum_ {j = 1} ^ {N} \pi_ {i} \operatorname {s i m} (\mathbf {Q} _ {i :} , \mathbf {K} _ {j :}) \mathbf {V} _ {j :}}{\sum_ {j = 1} ^ {N} \pi_ {i} \operatorname {s i m} (\mathbf {Q} _ {i :} , \mathbf {K} _ {j :})}
$$
Equation 27 shows their form as row vectors. Their  $j^{\mathrm{th}}$  elements are:
$$
\operatorname {S T A} _ {k} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) _ {i j} = \frac {\sum_ {t = 1} ^ {N} \hat {\mathbf {A}} _ {i t} ^ {k} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j}}{\sum_ {t = 1} ^ {N} \hat {\mathbf {A}} _ {i t} ^ {k} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right)} \tag {28}
$$
$$
\operatorname {S A} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) _ {i j} = \frac {\sum_ {t = 1} ^ {N} \pi_ {i} \operatorname {s i m} (\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}) \mathbf {V} _ {t j}}{\sum_ {t = 1} ^ {N} \pi_ {i} \operatorname {s i m} (\mathbf {Q} _ {i :} , \mathbf {K} _ {t :})}
$$
$$
\frac {\operatorname {S T A} _ {k} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) _ {i j}}{\operatorname {S A} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) _ {i j}} = \frac {\sum_ {t = 1} ^ {N} \hat {\mathbf {A}} _ {i t} ^ {k} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j}}{\sum_ {t = 1} ^ {N} \pi_ {i} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j}} \times \frac {\sum_ {t = 1} ^ {N} \pi_ {i} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right)}{\sum_ {t = 1} ^ {N} \hat {\mathbf {A}} _ {i t} ^ {k} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right)} \tag {29}
$$
For clarity, we proceed under the assumption that none of the denominators equal zero, which is reasonable considering the context. Let  $\delta_{it}$  represent the difference between  $\hat{\mathbf{A}}_it^k$  and  $\pi_i$ :  $\hat{\mathbf{A}}_it^k = \pi_i + \delta_{it}$ . Given  $\eta \in ]0,1[$ , we aim to determine an upper bound of the convergence rate between  $\mathrm{STA}_k$  and SA.
Using Equation 10, we take  $\epsilon = \frac{\eta}{N^2}$  and we have immediately:
$$
\forall i, t \in [ 1, N ] ^ {2}, \forall k \geq \frac {2 \log \frac {N}{\eta}}{1 - \hat {\lambda}}, | \hat {\mathbf {A}} _ {i t} ^ {k} - \boldsymbol {\pi} _ {i} | = | \delta_ {i t} | \leq \epsilon = \frac {\eta}{N ^ {2}} \tag {30}
$$
We can rewrite Equation 29 as:
$$
\frac {\operatorname {S T A} _ {k} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) _ {i j}}{\operatorname {S A} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) _ {i j}} = \frac {\sum_ {t = 1} ^ {N} \left(\pi_ {i} + \delta_ {i t}\right) \sin \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j}}{\sum_ {t = 1} ^ {N} \pi_ {i} \sin \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j}} \times \frac {\sum_ {t = 1} ^ {N} \pi_ {i} \sin \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right)}{\sum_ {t = 1} ^ {N} \left(\pi_ {i} + \delta_ {i t}\right) \sin \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right)} \tag {31}
$$
Assuming that  $k \geq \frac{2\log\frac{N}{\eta}}{1 - \hat{\lambda}}$ . Considering the fraction  $\frac{\sum_{i = 1}^{N}\delta_{it}\sin(\mathbf{Q}_{i,:},\mathbf{K}_{t:})\mathbf{V}_{tj}}{\sum_{i = 1}^{N}\pi_{i}\sin(\mathbf{Q}_{i,:},\mathbf{K}_{t:})\mathbf{V}_{tj}}$  in the first part of Equation 31. Using Equation 30 and the fact that  $\sin (\mathbf{Q}_{i,:},\mathbf{K}_{t:})$  and  $\mathbf{V}_{tj}$  are all positive, we have:
$$
\begin{array}{l} \left| \sum_ {t = 1} ^ {N} \delta_ {i t} \operatorname {s i m} \left(\mathbf {Q} _ {i:}, \mathbf {K} _ {t:}\right) \mathbf {V} _ {t j} \right| = \sum_ {t = 1} ^ {N} \left| \delta_ {i t} \right| \operatorname {s i m} \left(\mathbf {Q} _ {i:}, \mathbf {K} _ {t:}\right) \mathbf {V} _ {t j} \tag {32} \\ \leq \sum_ {t = 1} ^ {N} \frac {\eta}{N ^ {2}} \operatorname {s i m} \left(\mathbf {Q} _ {i:}, \mathbf {K} _ {t:}\right) \mathbf {V} _ {t j} \\ \end{array}
$$
Notice that  $\forall t\in [[1,N]],\pi_i\geq \frac{1}{N^2}$ . Hence, we have:
$$
\left| \sum_ {t = 1} ^ {N} \boldsymbol {\pi} _ {i} \operatorname {s i m} \left(\mathbf {Q} _ {i:}, \mathbf {K} _ {t:}\right) \mathbf {V} _ {t j} \right| \geq \sum_ {t = 1} ^ {N} \frac {1}{N ^ {2}} \operatorname {s i m} \left(\mathbf {Q} _ {i:}, \mathbf {K} _ {t:}\right) \mathbf {V} _ {t j} \tag {33}
$$
$$
\left| \frac {\sum_ {t = 1} ^ {N} \delta_ {i t} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j}}{\sum_ {t = 1} ^ {N} \boldsymbol {\pi} _ {i} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j}} \right| = \frac {\left| \sum_ {t = 1} ^ {N} \delta_ {i t} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j} \right|}{\left| \sum_ {t = 1} ^ {N} \boldsymbol {\pi} _ {i} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j} \right|} \leq \eta <   1 \tag {34}
$$
$$
1 - \eta \leq \frac {\sum_ {t = 1} ^ {N} \left(\boldsymbol {\pi} _ {i} + \delta_ {i t}\right) \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j}}{\sum_ {t = 1} ^ {N} \boldsymbol {\pi} _ {i} \operatorname {s i m} \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right) \mathbf {V} _ {t j}} \leq 1 + \eta \tag {35}
$$
Table 3: Statistics on datasets
Dataset	Context	# Nodes	# Edges	# Features	# Classes
Cora	Citation	2,708	5,429	1,433	7
Citeseer	Citation	3,327	4,732	3,703	6
Deezer	Social Connection	28,281	92,752	31,241	2
Actor	Co-occurrence	7,600	29,926	931	5
Pubmed	Citation	19,717	44,324	500	3
CoraFull	Citation	19,793	126,842	8,710	70
Computer	Co-purchasing	13,752	491,722	767	10
Photo	Co-purchasing	7,650	238,163	745	8
CS	Co-authorship	18,333	163,788	6,805	15
Physics	Co-authorship	34,493	495,924	8,415	5
Considering the second part  $\frac{\sum_{t=1}^{N}\pi_{i}\sin(\mathbf{Q}_{i:t},\mathbf{K}_{t:t})}{\sum_{t=1}^{N}(\pi_{i} + \delta_{it})\sin(\mathbf{Q}_{i:t},\mathbf{K}_{t:t})}$  of Equation 31. Utilizing the same line of reasoning, we can obtain:
$$
\left| \frac {\sum_ {t = 1} ^ {N} \delta_ {i t} \sin \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right)}{\sum_ {t = 1} ^ {N} \pi_ {i} \sin \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right)} \right| \leq \eta <   1 \tag {36}
$$
$$
\frac {1}{1 + \eta} \leq \frac {\sum_ {t = 1} ^ {N} \pi_ {i} \sin \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right)}{\sum_ {t = 1} ^ {N} \left(\pi_ {i} + \delta_ {i t}\right) \sin \left(\mathbf {Q} _ {i :} , \mathbf {K} _ {t :}\right)} \leq \frac {1}{1 - \eta} \tag {37}
$$
Considering Equation 35, Equation 37 and Equation 31, we finally prove that:
$$
\frac {1 - \eta}{1 + \eta} \leq \frac {\operatorname {S T A} _ {k} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) _ {i j}}{\operatorname {S A} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) _ {i j}} \leq \frac {1 + \eta}{1 - \eta} \tag {38}
$$
which proves the second part of Theorem 1.
C Dataset Information
In this section, we present the datasets used in our experiments. These different types of data provide a robust platform to evaluate the performance of our methods.
The detailed information for each dataset is presented in Table 3. These datasets are drawn from the areas of citation networks, co-purchasing networks, co-authorship networks, and social networks:  $\bullet$  Citation Networks: The citation networks datasets include Cora, Citeseer, Pubmed, and CoraFull. Nodes in these networks correspond to scientific publications, while the edges represent citations between these documents. In addition to the topological structure, each node carries a binary attribute vector, encoding the presence or absence of specific words from a pre-determined dictionary. The dimensionality of these attribute vectors varies from 1,433 in Cora to 8,710 in CoraFull. Moreover, each document node is associated with a unique class label, signifying the document's overarching scientific discipline.  $\bullet$  Co-authorship Networks: We utilize the CoauthorCSDataset and CoauthorPhysicsDataset that capture co-authorship relationships in Computer Science and Physics domains, respectively. Nodes represent individual authors and edges encode co-authorship relations, thus creating an undirected graph.  $\bullet$  Co-purchasing Networks: We utilize the AmazonCoBuyComputerDataset and AmazonCoBuyPhotoDataset, derived from Amazon's co-purchasing network. Nodes denote products and edges symbolize frequent co-purchase incidents. Moreover, the nodes can carry diverse product-specific information.  $\bullet$  Social Networks: The Deezer-Europe dataset is a dataset representing a social network of Deezer users collected via the public API in March 2020. The nodes in this network symbolize Deezer users hailing from various European countries, while the edges represent reciprocal follower relationships between these users. The features of each node are derived from the preferences of the users, specifically, the artists they have expressed an interest in. The task associated with this graph involves binary node classification, wherein the objective is to predict the user's gender.  $\bullet$  Co-occurrence Networks: We utilize the Actor dataset, a type of co-occurrence network based on the Microsoft Academic Graph. Nodes represent actors, and an edge signifies their co-appearance on the same Wikipedia page.
Positional Encoding We use Laplacian positional encoding to capture the structural information. As positional encoding is not the focus of our work, we use a simple approach to combine positional encoding with the original features of the nodes, which is also applied by [6]. Formally, we first calculate the eigenvectors corresponding to the smallest  $m$  eigenvalues of the Laplace matrix to construct the matrix  $\mathbf{P} \in \mathbb{R}^{n \times m}$ . Then we take  $\mathbf{X}' = [\mathbf{X}, \mathbf{P}]$  as the new input, where [] denotes row-wise concatenation. For all the datasets, we set  $m = 3$ .
D.1 Node Classification
Training Details We choose two recent studies [43, 6] and we adhere to their experimental configurations. The metrics for the baselines are also derived from these works [43, 6]. For Cora, Citeuser, Deezer and Actor, we apply the same random splits with train-valid/test ratios of  $50\% /25\% /25\%$  as [43]. We conduct 5 runs with different splits and take the mean accuracy and standard deviation for comparison. For Pubmed, Corafull, Computer, Photo, CS and Physics, we apply the same random splits with train-valid/test ratios of  $60\% /20\% /20\%$  as [6]. We conduct 10 runs with different splits and take the mean accuracy and standard deviation for comparison. Specifically, we utilize the ROC-AUC measure for binary classification on the Deezer dataset. For other datasets containing more than two classes, we opt for Accuracy as the metric. We employ the Adam optimizer for gradient-based optimization. The training procedure can at most repeat until a given budget of 3000 epochs and we set the patience of early stop to 200 epochs. We report the test accuracy of the epoch which has the highest accuracy on the validation set.
Hyperparameters For the model configuration of STAGNN, we fix the number of hidden channels at 64. We use grid search for hyper-parameter settings. The learning rate is searched within  $\{0.001,0.01\}$ , dropout probability searched within  $\{0.0,0.2,0.4,0.6\}$ , weight decay searched within  $\{0.0001,0.0005,0.001,0.005\}$ , height of the rooted subtree  $K$  searched within  $\{3,5,10\}$ , number of attention heads searched within  $\{1,2,4,6,8\}$ . The best hyper-parameters are provided in supplementary materials.
D.2 Study on the Necessity of SubTree Attention in the Presence of Global Attention
In this experiment, we extend STAGNN by replacing the STA module with global attention enhanced by 0, 1, 2, or 3 hop/hops subtree attention. We now present a detailed mathematical description of the experimental configurations. Formally, we compare the performance of the STAGNN-based model equipped with four different attention strategies: Global Attn Only, 1-hop STA + GA, 2-hops STA + GA and 3-hops STA + GA on six datasets: Pubmed, Corafull, Computer, Photo, CS and Physics, with the same experiment setting described in subsection D.1.
First, we calculate keys, queries and values.
$$
\mathbf {Q} = \mathbf {H} \mathbf {W} _ {Q}, \mathbf {K} = \mathbf {H} \mathbf {W} _ {K}, \mathbf {V} = \mathbf {H} \mathbf {W} _ {V}, \mathbf {H} = \operatorname {M L P} (\mathbf {X}) \tag {39}
$$
Next, the output of the four different models (equipped with global attention enhanced by subtree attention of different heights) can be described as:
- Global Attn Only:
$$
\mathbf {O} = \operatorname {S A} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \tag {40}
$$
- 1-hop STA + GA:
$$
\mathbf {O} = \alpha_ {T} \operatorname {S A} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) + \sum_ {k = 0} ^ {1} \alpha_ {k} \operatorname {S T A} _ {k} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \tag {41}
$$
- 2-hops STA + GA:
$$
\mathbf {O} = \alpha_ {T} \operatorname {S A} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) + \sum_ {k = 0} ^ {2} \alpha_ {k} \operatorname {S T A} _ {k} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \tag {42}
$$
- 3-hops STA + GA:
$$
\mathbf {O} = \alpha_ {T} \operatorname {S A} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) + \sum_ {k = 0} ^ {3} \alpha_ {k} \operatorname {S T A} _ {k} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \tag {43}
$$
$\alpha_{T}$  here represents the coefficient of teleportation, because we can regard the global attention enhanced by subtree attention here as the random walk with teleportation. The only difference between these models is that they use subtree attention of different heights as an auxiliary to global attention. As shown in Table 2, we can observe that 2-hops  $STA + GA$  and 3-hops  $STA + GA$  outperform Global Attn Only by a large margin.
D.3 Study on HopAggregation Methods
In this experiment, we investigate different choices of the HopAggregation functions within the STA module. We compare GPR-like aggregation with sum, concat [18], and attention-based readout [6]. We now present a detailed mathematical description of the experimental configurations. Formally, we compare the performance of the following four models: STAGNN-GPR (origin STAGNN), STAGNN-SUM, STAGNN-CONCAT and STAGNN-ATTN on four datasets: Cora, Citeseer, Deezer-Europe and Actor, with the same experiment setting described in subsection D.1.
$$
\mathbf {Q} = \mathbf {H} \mathbf {W} _ {Q}, \mathbf {K} = \mathbf {H} \mathbf {W} _ {K}, \mathbf {V} = \mathbf {H} \mathbf {W} _ {V}, \mathbf {H} = \operatorname {M L P} (\mathbf {X}) \tag {44}
$$
Next, the output of the four different models (STAGNN with different HopAggregation methods) can be described as:
- STAGNN-GPR (origin STAGNN):
$$
\mathbf {O} = \sum_ {k = 0} ^ {K} \alpha_ {k} \mathrm {S T A} _ {k} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \tag {45}
$$
STAGNN-SUM:
$$
\mathbf {O} = \sum_ {k = 0} ^ {K} \operatorname {S T A} _ {k} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \tag {46}
$$
- STAGNN-CONCAT:
$$
\mathbf {O} = \left[ \operatorname {S T A} _ {0} (\mathbf {Q}, \mathbf {K}, \mathbf {V}), \operatorname {S T A} _ {1} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \dots , \operatorname {S T A} _ {K} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \right] \mathbf {W} _ {O} \tag {47}
$$
where  $\mathbf{W}_O$  is a linear projection matrix.
- STAGNN-ATTN:
$$
\mathbf {O} = \operatorname {S T A} _ {0} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) + \sum_ {k = 1} ^ {K} \beta_ {k} \operatorname {S T A} _ {k} (\mathbf {Q}, \mathbf {K}, \mathbf {V}), \tag {48}
$$
$$
\beta_ {k} = \frac {\exp \left([ \mathrm {S T A} _ {0} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) , \mathrm {S T A} _ {k} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) ] \mathbf {W} _ {a} ^ {\top}\right)}{\sum_ {i = 1} ^ {K} \exp \left([ \mathrm {S T A} _ {0} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) , \mathrm {S T A} _ {i} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) ] \mathbf {W} _ {a} ^ {\top}\right)}
$$
where  $\mathbf{W}_a$  is a linear projection matrix and  $[\cdot]$  denotes row-wise concatenation.
E More Visualizations of GPR Weights
We conduct more visualizations of the GPR weights on Cora and Actor, with heights  $K$  of the rooted subtrees ranging from 3 to 75. The results are shown in Figure 7.
In the case of Cora, we observe that as the depth  $K$  of the rooted subtree increases, STA keeps increasing the GPR weights of the local neighborhood in order to preserve the local information from being covered up by the global information.
Figure 7: GPR weights of STAGNN when the heights  $K$  of the subtree ranging from 3 to 75.