SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 2: Overall architecture of STAGNN. STAGNN can be decomposed into two parts: Transformation and STA-based Propagation, the latter of which can be seen as an STA module using GPR-like aggregation as HopAggregation function. According to the algorithm introduced in subsection 3.2, employing STA here is equivalent to letting keys and values propagate on the graph.
Comparing Equation 5 and Equation 9, we find that the only difference between SA and  $\mathrm{STA}_k$  lies in the mask for the similarity matrix.  $\mathrm{STA}_k$  employs  $\hat{\mathbf{A}}_{ij}^k$  as the mask, whereas SA employs  $\pi_i$ . We now demonstrate that when the height  $K$  of the rooted subtree takes the same order of magnitude as  $\log (N)$ ,  $\mathrm{STA}_k$  can be approximately considered as SA.
Theorem 1 Let  $\hat{\mathbf{A}}\in \mathbb{R}^{N\times N}$  denote the random walk matrix of a connected and non-bipartite graph, i.e.,  $\hat{\mathbf{A}} = \mathbf{AD}^{-1}$ . Let  $1 = \lambda_{1}\geq \dots \geq \lambda_{N}$  be the eigenvalues of  $\hat{\mathbf{A}}$ . Then we have the following results:
$$
\forall i, j \in [   [ 1, N ]   ] ^ {2}, \forall \epsilon > 0, \exists K _ {0} \in \mathbb {N}, \forall k > K _ {0}, | \hat {\mathbf {A}} _ {i j} ^ {k} - \boldsymbol {\pi} _ {i} | \leq \epsilon \tag {10}
$$
And for a given  $\epsilon$ , the smallest  $K_{0}$  that satisfies the condition shown in Equation 10 is at most  $\mathcal{O}\left(\frac{\log\frac{N}{\epsilon}}{1 - \max\{\lambda_2,|\lambda_n|\}}\right)$ . If we had more information about  $\mathbf{V}$ , we could specify the convergence rate of  $\mathrm{STA}_k$ . e.g., if  $\mathbf{V}$  is computed by  $\mathbf{V} = \sigma (\mathbf{XW}_V)$  where  $\sigma$  is a non-negative activation function, then:
$$
\forall i, j \in [ 1, N ] ^ {2}, \forall \eta \in ] 0, 1 [, \exists K _ {1} \in \mathbb {N}, \forall k > K _ {1}, \frac {1 - \eta}{1 + \eta} \leq \frac {\operatorname {S T A} _ {k} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) _ {i j}}{\operatorname {S A} (\mathbf {Q} , \mathbf {K} , \mathbf {V}) _ {i j}} \leq \frac {1 + \eta}{1 - \eta} \tag {11}
$$
holds true when none of the denominators is equal to zero. And for a given  $\eta$ , the smallest  $K_{1}$  that satisfies the condition shown in Equation 11 is at most  $\mathcal{O}\left(\frac{\log\frac{N}{\eta}}{1 - \max\{\lambda_2,|\lambda_n|\}}\right)$ .
Equation 10 demonstrates that  $\hat{\mathbf{A}}_{ij}^{k}$  converges to  $\pi_i$  with logarithmic complexity, indicating that under general conditions,  $\mathrm{STA}_k$  quickly tends to SA. Notice that although  $\mathrm{STA}_k$  is a multi-hop graph attention module implemented using the message-passing mechanism, it displays the characteristics of global attention when the height of the subtree is  $\mathcal{O}(\log N)$ . This property prevents STA from the issues associated with the message-passing scheme such as over-smoothing and over-squashing. From this perspective, subtree attention serves as a bridge connecting local and global attention.  $\mathrm{STA}_1$  plays the role of local attention, while  $\mathrm{STA}_{\mathcal{O}(\log N)}$  acts as the global self-attention. Subtree attention achieves a hierarchical attention computation by a hop-wise aggregation of  $\{\mathrm{STA}_k\}_{k\in [[1,\mathcal{O}(\log N)]}$ . A detailed proof of Theorem 1 is provided in Appendix B.
4 The Proposed Multi-Hop Graph Attention Network: STAGNN
In this section, we present a simple yet effective multi-hop graph attention network, named STAGNN. This model is built upon decoupled GCN, but employs STA as the message-passing module instead. STAGNN can be divided into two steps: first, we apply MLP to compute queries, keys, and values for each node; then, we use STA to propagate information. Formally, it can be described as:
$$
\mathbf {O} = \sum_ {k = 0} ^ {K} \alpha_ {k} \operatorname {S T A} _ {k} (\mathbf {Q}, \mathbf {K}, \mathbf {V}), \quad \mathbf {Q} = \mathbf {H} \mathbf {W} _ {Q}, \quad \mathbf {K} = \mathbf {H} \mathbf {W} _ {K}, \quad \mathbf {V} = \mathbf {H} \mathbf {W} _ {V}, \quad \mathbf {H} = \operatorname {M L P} (\mathbf {X}) \tag {12}
$$
Figure 3: Comparison of four GNN baselines, three graph structure learning baselines and STAGNN on four common node classification datasets. The missing result of Deezer is due to out-of-memory.
Table 1: Comparison of four GNN baselines, five Graph Transformer baselines and STAGNN on six common node classification datasets. The best results appear in **bold**.
Method	Pubmed	CoraFull	Computer	Photo	CS	Physics
GCN	86.54±0.12	61.76±0.14	89.65±0.52	92.70±0.20	92.92±0.12	96.18±0.07
GAT	86.32±0.16	64.47±0.18	90.78±0.13	93.87±0.11	93.61±0.14	96.17±0.08
APPNP	88.43±0.15	65.16±0.28	90.18±0.17	94.32±0.14	94.49±0.07	96.54±0.07
GPRGNN	89.34±0.25	67.12±0.31	89.32±0.29	94.49±0.14	95.13±0.09	96.85±0.08
GT	88.79±0.12	61.05±0.38	91.18±0.17	94.74±0.13	94.64±0.13	97.05±0.05
Graphormer	OOM	OOM	OOM	92.74±0.14	OOM	OOM
SAN	88.22±0.15	59.01±0.34	89.83±0.16	94.86±0.10	94.51±0.15	OOM
GraphGPS	88.94±0.16	55.76±0.23	OOM	95.06±0.13	93.93±0.12	OOM
NAGphormer	89.70±0.19	71.51±0.13	91.22±0.14	95.49±0.11	95.75±0.09	97.34±0.03
STAGNN	90.46±0.22	72.65±0.36	91.72±0.30	95.64±0.27	95.77±0.16	97.09±0.18
where  $\mathbf{X}$  is the input node feature and  $\mathbf{O}$  is the learned representation for each node. We adopt the GPR-like aggregation [8] for HopAggregation in STA. To be precise, we assign learnable parameters  $\{\alpha_{k}\}_{k\in [0,K]}$  to each hop (initialized simply to 1), and then the nodes aggregate information from each hop based on these learned weights. Figure 2 shows the overall architecture of STAGNN.
Comparing Equation 1 and Equation 12, the only difference between decoupled GCN and STAGNN lies in the propagation method. Decoupled GCN relies on high powers of the normalized adjacency matrix to capture long-range information, which inevitably results in over-smoothing. In contrast, STAGNN utilizes subtree attention for message propagation, effectively learning more informative representations from multi-hop neighbors without suffering from the inherent problems associated with the message-passing scheme.
Wu et al. (2022) [43] have drawn attention to an issue named over-normalization: In the context of graphs with a large volume of nodes, the use of the global attention module may lead to a situation where the attention scores for the majority of nodes are nearly zero and thus resulting in gradient vanishing. Subtree attention can alleviate this problem by providing a hierarchical calculation focusing on each level of the rooted subtree instead of the whole graph.
We evaluate the performance of STAGNN on ten common node classification datasets, with detailed dataset information provided in Appendix C. We then verify the performance of STAGNN under extreme settings, empirically showing its capacity to tackle over-smoothing. Furthermore, We conduct an experiment that confirms the necessity of subtree attention even in the presence of global attention. Additional ablation studies are conducted for further discussions. For implementation, we fix the number of hidden channels at 64. More implementation details are presented in Appendix D. All experiments are conducted on an NVIDIA RTX4090 with 24 GB memory.
Figure 4: (Left and Middle): Performance of STAGNN across different heights of the rooted subtree. (Right): GPR weights of STAGNN when the height of the subtree  $K = 100$ .
5.1 Experiments on Node Classification
To compare STAGNN with a wide variety of baselines, we select two recent works [43, 6] and follow their experiment settings and their choice of baselines. The metrics of the baselines are adopted from these works [43, 6]. The code for reproduction can be found in supplementary materials.
Comparison with Multi-Hop GNNs and Structure Learning Methods For a fair comparison, we strictly follow the experiment settings from Wu et al. (2022) [43]. We test the performance of STAGNN on four common datasets: Cora, Citeseer, Deezer, and Actor. The first two are homogeneous graphs, while the latter two are heterophilic [24, 52, 34]. We compare STAGNN with two mainstream GNNs: GCN [21], GAT [39], two multi-hop GNNs: JKNet [47], MixHop [1], and three graph structure learning methods: LDS [14], IDGL [7], NodeFormer [43]. We apply the same random splits with train/valid/test ratios of  $50\% / 25\% / 25\%$  as [43]. Further details can be found in Appendix D.1. Figure 3 displays the experimental results, showing that STAGNN, which features a relatively simple model architecture, outperforms all the baselines on all four datasets. This result highlights the effectiveness of STAGNN in managing both homogeneous and heterophilic graphs.
Comparison with Decoupled GCNs and Graph Transformers For a fair comparison, we strictly follow the experiment settings from Chen et al. (2022) [6]. We test the performance of STAGNN on six common datasets: Pubmed, Corafull, Computer, Photo, CS and Physics. More information about these datasets can be found in Appendix C. We compare STAGNN with two mainstream GNNs: GCN [21], GAT [39], two decoupled GCNs: APPNP [22], GPRGNN [8], and five graph transformers: GT [12], Graphormer [48], SAN [23], GraphGPS [32], NAGphormer [6]. We apply the same random splits with train/valid/test ratios of  $60\% / 20\% / 20\%$  as [6]. The experimental results are shown in Table 1. STAGNN shows comparable or superior performance compared to all the baselines, which highlights the competitiveness of STAGNN when compared to existing GNNs and graph transformers.
5.2 Experiments on Deep STAGNN
We evaluate the performance of deep STAGNN on Cora and Actor, with the height of the subtree ranging from 3 to 100. The experimental results are presented in Figure 4. In contrast to MP-GNNs, STAGNN maintains robust performance even when the height of the subtree reaches 100. For Cora, the accuracy of STAGNN peaks at  $K = 10$ , demonstrating its ability to effectively collect information from a large receptive field. We further visualize the GPR weights of STAGNN when the height of the subtree is set to  $K = 100$ , and we observe distinct characteristics for Cora and Actor. In the case of Cora, the GPR weights exhibit a monotonic decrease, which aligns with the witnessed performance drop on Cora as the height of the subtree increases. Therefore, for Cora, we may consider keeping the height of the subtree within a reasonable range. In the case of Actor, the GPR weights eventually stabilize at a value close to 1. This finding suggests that the limiting state of  $\mathrm{STA}_k$ , i.e.,  $\mathrm{SA}_{\pi}$ , is suitable for Actor, which is confirmed by the robust performance of deep STAGNN on Actor. More visualizations of GPR weights can be found in Appendix E. In summary, STAGNN can achieve impressive results even with an extremely deep architecture.
Table 2: Necessity of subtree attention in the presence of global attention. We compare two scenarios: (i) Only Global attention and (ii) Global attention supplemented by subtree attention.
Method	Pubmed	CoraFull	Computer	Photo	CS	Physics
GAT	86.32±0.16	64.47±0.18	90.78±0.13	93.87±0.11	93.61±0.14	96.17±0.08
Global Attn (GA)	88.87±0.61	62.34±0.95	85.7±0.52	92.92±0.32	94.74±0.37	96.47±0.24
1-hop STA + GA	90.16±0.51	70.65±0.71	91.52±0.23	95.42±0.47	95.49±0.26	97.09±0.22
2-hops STA + GA	90.56±0.49	72.24±0.38	91.93±0.35	95.75±0.36	95.70±0.29	97.17±0.20
3-hops STA + GA	90.66±0.24	72.36±0.37	91.89±0.28	95.88±0.31	95.81±0.15	97.15±0.23
Figure 5: Comparison of different HopAggregation methods
Study on the Necessity of SubTree Attention in the Presence of Global Attention We demonstrate that  $\mathrm{STA}_k$  converges to the global self-attention, which naturally leads to a critical question:
- Is it still necessary to employ subtree attention when global attention is already present?
In this experiment, we seek to answer the question above by exploring two scenarios: (i) applying global attention independently and (ii) combining global attention with subtree attention. Formally, we extend STAGNN by replacing the STA module with global attention enhanced by 0, 1, 2, or 3 hop/hops of subtree attention. More details on the experiment settings can be found in Appendix D.2.
The experimental results are presented in Table 2. We find that incorporating subtree attention as an auxiliary to global attention significantly improves overall performance, implying that it is still necessary to employ subtree attention to capture the neighborhood structure even in the presence of global attention. This observation also inspires us that leveraging subtree attention to augment global attention could be an interesting approach for enhancing Graph Transformers.
Study on HopAggregation methods In this experiment, we investigate the influence of different HopAggregation functions within STAGNN. We compare GPR-like aggregation with sum, concat [18], and attention-based readout [6]. Details of the experiment settings can be found in Appendix D.3. The experimental results are shown in Figure 5. GPR-like aggregation outperforms the alternatives on all four datasets. This observation highlights the importance of enabling nodes to adaptively learn the weight of each hop, which is the main advantage of GPR-like aggregation.
We propose a novel multi-hop graph attention mechanism called Subtree Attention (STA), which effectively addresses the limitations of local and global attention in graph learning. The proposed STA mechanism bridges the gap between local and global attention, hierarchically capturing neighborhood structures while addressing issues associated with the message-passing scheme. The key algorithm for computing subtree attention, utilizing kernelized softmax and the message-passing scheme, reduces the computational cost from  $\mathcal{O}(N^2)$  to  $\mathcal{O}(|\mathcal{E}|)$  while avoiding the need to store high powers of the adjacency matrix. This process can be approximately viewed as keys and values performing a random walk on the graph. We further prove theoretically that under extreme settings, STA approximates the global self-attention. Based on the STA module, we propose a simple yet performant multi-hop graph attention network, coined STAGNN. Comprehensive evaluations on various node classification datasets demonstrate that STAGNN outperforms mainstream GNNs and graph transformers. Ablation studies further verify the effectiveness of subtree attention, even in the presence of global attention.
Current Limitations, Potential Impacts and Further Discussions In the present work, we mainly evaluate STAGNN, a novel multi-hop graph attention network incorporating STA. However, there are many other potential applications of STA, including combining STA with other graph learning methods that utilize self-attention mechanisms or supplementing global attention with subtree attention to enhance graph transformers. Furthermore, evaluating the robustness of STA and digging into its interpretability can also be part of future works. We provide a detailed discussion of potential impacts in Appendix G. And further analysis of the gate mechanism within the mixture of attention heads can be found in Appendix F.
This work was sponsored by the National Natural Science Foundation of China (NSFC) grant (No. 62106143), and Shanghai Pujiang Program (No. 21PJ1405700).
[47] K. Xu, C. Li, Y. Tian, T. Sonobe, K. Kawarabayashi, and S. Jegelka. Representation learning on graphs with jumping knowledge networks. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 5449-5458. PMLR, 2018.
[48] C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T. Liu. Do transformers really perform badly for graph representation? In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 28877-28888, 2021.
[49] A. W. Yu, D. Dohan, M. Luong, R. Zhao, K. Chen, M. Norouzi, and Q. V. Le. Quantitative learning of local convolution with global self-attention for reading comprehension. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[50] J. Zhang, H. Zhang, C. Xia, and L. Sun. Graph-bert: Only attention is needed for learning graph representations. CoRR, abs/2001.05140, 2020.
[51] J. Zhao, Y. Dong, M. Ding, E. Kharlamov, and J. Tang. Adaptive diffusion in graph neural networks. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 23321-23333, 2021.
[52] J. Zhu, Y. Yan, L. Zhao, M. Heimann, L. Akoglu, and D. Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
A Complexity Analysis of STA
In this subsection, we analyze the time complexity of SubTree Attention (STA). STA has two key components: the feature map and the HopAggregation function. Both of these components offer a wide range of potential options for consideration. Different options will affect the time complexity of STA. In the following analysis, we will adopt the configuration used by STAGNN, i.e., we choose  $\phi(x) = elu(x) + 1$  as the feature map and use GPR-like aggregation as the HopAggregation function.
The computation of STA can be seen as an aggregation of  $\{\mathrm{STA}_i\}_{i\in [1,K]}$ , which refer to the attention-based aggregation of each level of the rooted subtree. Therefore, we can start by analyzing the time complexity of  $\mathrm{STA}_k$ .
Figure 6: Efficient algorithm of SubTree Attention
The calculation of  $\mathrm{STA}_k$  can be divided into three steps. In the first step, we compute  $\phi(\mathbf{K}_{i:})$  and  $\phi(\mathbf{K}_{i:})^T\mathbf{V}_{i:}$  for each node. The time complexity of this step depends on the feature map. In our model, we chose  $\phi(x) = elu(x) + 1$  as the feature map. Thus, the time complexity of computing  $\phi(\mathbf{K}_{i:})$  is  $\mathcal{O}(Nd_k)$ . We also need to compute  $\phi(\mathbf{K}_{i:})^T\mathbf{V}_{i:}$  for each node, the time complexity of this part is  $\mathcal{O}(Nd_k d_v)$ . Therefore, the overall time complexity of the first step is  $\mathcal{O}(Nd_k + Nd_k d_v)$ .
In the second step, we let  $\phi(\mathbf{K}_{i:})$  and  $\phi(\mathbf{K}_{i:})^T\mathbf{V}_i$ : propagate on the graph. For  $\mathrm{STA}_k$ , we need to propagate  $k$  times. The time complexity of propagating  $\phi(\mathbf{K}_{i:})$  once is  $\mathcal{O}(d_k)$ , and the time complexity of  $\phi(\mathbf{K}_i)^T\mathbf{V}_i$ : propagating once is  $\mathcal{O}(d_k d_v)$ . The message propagation occurs on each edge. Considering that there are in total  $|\mathcal{E}|$  edges and  $k$  times propagation, the overall time complexity of this step is  $\mathcal{O}(k|\mathcal{E}|d_k + k|\mathcal{E}|d_k d_v)$ .
In the third step, we use the information  $\sum_{j=1}^{N} \hat{\mathbf{A}}_{ij}^k \phi(\mathbf{K}_{j:})^T \mathbf{V}_{j}$ ; and  $\sum_{j=1}^{N} \hat{\mathbf{A}}_{ij}^k \phi(\mathbf{K}_{j:})^T$  aggregated by each node, along with the node's own query  $\phi(\mathbf{Q}_{i:})$ , to complete the computation of STA. For each node, we need to calculate  $\phi(\mathbf{Q}_{i:}) \sum_{j=1}^{N} \hat{\mathbf{A}}_{ij}^k \phi(\mathbf{K}_{j:})^T \mathbf{V}_{j}$ , the time complexity of this part is  $\mathcal{O}(Nd_k d_v)$ . At the same time, for each node, we need to calculate  $\phi(\mathbf{Q}_{i:}) \sum_{j=1}^{N} \hat{\mathbf{A}}_{ij}^k \phi(\mathbf{K}_{j:})^T$ , the time complexity of this part is  $\mathcal{O}(Nd_k)$ . So the total time complexity of this step is  $\mathcal{O}(Nd_k + Nd_k d_v)$ . In summary, the total time complexity of  $\mathrm{STA}_k$  is  $\mathcal{O}(2Nd_k + 2Nd_k d_v + k|\mathcal{E}|d_k + k|\mathcal{E}|d_k d_v)$ .
Next, we analyze the time complexity of STA when the height of the rooted subtree is  $K$ . It should be noted that  $\{\mathrm{STA}_i\}_{i\in [1,K]}$  can be viewed as a nested process, calculated one after another. Therefore, the first two steps of the above-mentioned calculation of  $\mathrm{STA}_k$  do not need to be repeated. We only need to complete the full calculation of  $\mathrm{STA}_K$  and perform the third step mentioned above  $K$  times. Therefore, the time complexity of calculating STA is  $\mathcal{O}((K + 1)Nd_k + (K + 1)Nd_kd_v + K|\mathcal{E}|d_k + K|\mathcal{E}|d_kd_v)$ . In general, we can think of the time complexity of STA as  $\mathcal{O}(K|\mathcal{E}|d_kd_v)$ .
B.1 Proof for Equation 10
Let  $\hat{\mathbf{A}}$  denote the random walk matrix of a connected and non-bipartite graph, and let  $\mathbf{A}_{\mathrm{sym}}$  denote the symmetric normalized adjacency matrix. Let  $1 = \lambda_{1} \geq \dots \geq \lambda_{N}$  be the eigenvalues of  $\hat{\mathbf{A}}$ , which are also the eigenvalues of  $\mathbf{A}_{\mathrm{sym}}$  [26]. Let  $\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_n$  be the corresponding orthonormal eigenvectors  $(\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_n$  here are column vectors). Let  $\pi_j = \frac{d(j)}{\sum_{i=1}^N d(i)}$  and  $d(i)$  denotes the degree of the  $i^{\mathrm{th}}$  node.  $\hat{\lambda} = 1 - \max\{\lambda_2, |\lambda_n|\}$  denotes the corresponding spectral gap, and let  $\mathbf{D}$  be the diagonal degree matrix.  $\overrightarrow{1}$  denotes an all-ones column vector.
In this subsection, we prove the following results:
$$
\forall i, j \in [ [ 1, N ] ] ^ {2}, \forall \epsilon > 0, \exists K _ {0} \in \mathbb {N}, \forall k > K _ {0}, | \hat {\mathbf {A}} _ {i j} ^ {k} - \pmb {\pi} _ {i} | \leq \epsilon
$$
And for a given  $\epsilon$ , the smallest  $K_{0}$  that satisfies the condition above is at most  $\mathcal{O}\left(\frac{\log\frac{N}{\epsilon}}{1 - \max\{\lambda_{2},|\lambda_{n}|\}}\right)$ . We begin by considering an arbitrary distribution  $\mathbf{p_i}\in \mathbb{R}^N$ , which is a column vector and  $\| \mathbf{p_i}\| _2 = 1$ . Notice that  $\mathbf{v_1},\mathbf{v_2},\ldots ,\mathbf{v_n}$  form an orthonormal basis, we can rewrite  $\mathbf{D}^{-\frac{1}{2}}\mathbf{p_i}$  as:
$$
\mathbf {D} ^ {- \frac {1}{2}} \mathbf {p} _ {\mathbf {i}} = \sum_ {i = 1} ^ {N} c _ {i} \mathbf {v} _ {\mathbf {i}} \tag {13}
$$
We next consider the new distribution obtained when  $\mathbf{p_i}$  undergoes  $k$ -step random walk. Notice that  $\hat{\mathbf{A}} = \mathbf{AD}^{-1} = \mathbf{D}^{\frac{1}{2}}\left(\mathbf{D}^{-\frac{1}{2}}\mathbf{A}\mathbf{D}^{-\frac{1}{2}}\right)\mathbf{D}^{-\frac{1}{2}} = \mathbf{D}^{\frac{1}{2}}\mathbf{A}_{\mathrm{sym}}\mathbf{D}^{-\frac{1}{2}}$ . Thus we have:
$$
\begin{array}{l} \hat {\mathbf {A}} ^ {k} \mathbf {p} _ {\mathbf {i}} = \left(\mathbf {D} ^ {\frac {1}{2}} \mathbf {A} _ {\mathrm {s y m}} \mathbf {D} ^ {- \frac {1}{2}}\right) ^ {k} \mathbf {p} _ {\mathbf {i}} \\ = \mathbf {D} ^ {\frac {1}{2}} \mathbf {A} _ {\text {s y m}} ^ {k} \mathbf {D} ^ {- \frac {1}{2}} \mathbf {p} _ {\mathbf {i}} \\ = \mathbf {D} ^ {\frac {1}{2}} \mathbf {A} _ {\text {s y m}} ^ {k} \sum_ {i = 1} ^ {N} c _ {i} \mathbf {v} _ {\mathbf {i}} \\ = \sum_ {i = 1} ^ {N} c _ {i} \mathbf {D} ^ {\frac {1}{2}} \mathbf {A} _ {\text {s y m}} ^ {k} \mathbf {v} _ {\mathbf {i}} \tag {14} \\ = c _ {1} \mathbf {D} ^ {\frac {1}{2}} \mathbf {A} _ {\mathrm {s y m}} ^ {k} \mathbf {v _ {1}} + \sum_ {i = 2} ^ {N} c _ {i} \mathbf {D} ^ {\frac {1}{2}} \mathbf {A} _ {\mathrm {s y m}} ^ {k} \mathbf {v _ {i}} \\ \end{array}
$$
We now consider  $c_{1} \mathbf{D}^{\frac{1}{2}} \mathbf{A}_{\mathrm{sym}}^{k} \mathbf{v}_{1}$ . As we know that  $\mathbf{D}^{\frac{1}{2}} \overrightarrow{\mathbf{1}}$  is an eigenvector of  $\mathbf{A}_{\mathrm{sym}}$  with eigenvalue 1. We then have: