SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rattan and Seppelt [39] show that the  $L_{0}$  isospectra is strictly bounded by 2-FWL. The  $L_{1}$  isospectra, through the introduction of 2-simplices (triangles), can distinguish some non-isomorphic graph pairs that are indistinguishable by 2-FWL. See Appendix C for detailed examples.
The zero eigenvalues of  $L_{1}$  have some more important properties. Its multiplicity is the 1-th Betti number  $\beta_{1}$ , which is exactly the number of cycles (except triangles) in the graph. We further consider the eigenvectors of  $L_{1}$ , each eigenvector  $\mathbf{u}_i$  of the eigenvalues  $\lambda_{i}$  has a length  $m$ , and each element  $\mathbf{u}_{ij}$  in it reflects the weight of the corresponding edge  $e_j$  at this frequency  $\lambda_{i}$ . The absolute values of elements corresponding to the edges in cycles are non-zero, while the edges not in cycles have zero weights in the eigenvectors. In other words, the eigenvectors of zero eigenvalues can efficiently mark the edges that are in a cycle. More intuitive illustration and theoretical proof are given in Appendix C.2.2.
Hodge1Lap: sign and basis invariant edge PE. In this section, we propose Hodge1Lap, a novel edge-level positional encoding method based on the spectral analysis of Hodge 1-Laplacian. To the best of our knowledge, this is the first sign and basis invariant edge-level PE based on Hodge 1-Laplacian  $L_{1}$ .
Recall the geometric meaning of the Hodge 1-Laplacian spectra in Section 5.2. Zero eigenvalues and eigenvectors reflect the cycles in the graph. These insights of Hodge 1-Laplacian spectra shed light on our design for edge-level positional encoding. Denote the eigenvalues  $\lambda_{i}$  with multiplicity  $m(i)$  as  $\lambda_{i(1)},\lambda_{i(2)},\ldots ,\lambda_{i(m_i)}$ , respectively. The corresponding eigenvectors are  $\mathbf{u}_{i(1)},\dots,\mathbf{u}_{i(m_i)}$  but note that these eigenvectors are: (i) not sign invariant, since if  $L_{1}\mathbf{u}_{i(j)} = 0,j = 1,\dots,m_{i}$ , then  $L_{1}(-\mathbf{u}_{i(j)}) = 0$ ; (ii) not basis invariant if  $m_{i} > 1$ , since any  $m_{i}$  linearly independent basis of the kernel space are also eigenvectors, and the subspace they span is identical to the kernel space. This is analogous to the  $L_0$  eigenvectors: they are not sign and basis invariant, which makes it difficult for us to design sign and basis invariant positional encodings. Therefore, we propose a novel projection-based method to build Hodge1Lap, a sign and basis invariant edge-level positional encoding.
Formally, Hodge1Lap processes the eigenvalues  $\lambda_{i}$  with multiplicity  $m_{i}$  and relevant eigenvectors as follows. Recall the projection matrix
$$
P _ {p r o j, i} = \mathbf {U} \mathbf {U} ^ {T} = \sum_ {j = 1} ^ {m _ {i}} \mathbf {u} _ {i (j)} \mathbf {u} _ {i (j)} ^ {T} \tag {4}
$$
where the subscript  $proj$  is used to distinguish the projection matrix from probability transition matrix  $P$ , and  $\mathbf{U} = [\mathbf{u}_{i(1)},\dots,\mathbf{u}_{i(m_i)}]$ . For any vector  $\mathbf{v} \in \mathbb{R}^m$ ,  $P_{proj,i}\mathbf{v}$  projects it into the subspace spanned by the eigenvectors  $u_{i(j)}, j = 1,\ldots,m_i$ . It is straightforward to verify that the projection in the subspace is independent of the choice of basis  $u_{i(j)}$  as long as they are linearly independent and hence is both sign and basis invariant. As long as the preimage  $\mathbf{v}$  is well defined (e.g., permutation equivariant to edge index), the projection can satisfy permutation equivariance as well as sign and basis invariance. In Hodge1Lap, we propose to use two different forms of preimages: a unit vector  $\mathbf{e} \in \mathbb{R}^m$  with each element  $\mathbf{e}_j = \frac{1}{\sqrt[m]{m}}$ , and the original edge feature  $\mathbf{X}(E) \in \mathbb{R}^{m\times d}$ . The first variant considers puer structure information, while the second variant jointly encodes structure and feature information. Taking the first variant as an example, Hodge1Lap implemented by projection can be formulated as
$$
\operatorname {H o d g e 1 L a p} _ {\text {p r o j}} (E) = \sum_ {i} \phi_ {i} \left(P _ {\text {p r o j}, i} \mathbf {e}\right) \tag {5}
$$
where  $\phi_{i}$  are injective functions and can be replaced by MLP layers, and the summation is performed over the interested eigen-subspaces.
In addition to the projection-based implementation of Hodge1Lap, we also implement other variants (analogously to the implementation of LapPE [30]): (i) We use a shared MLP  $\phi$  to directly embed the  $n_{\text{eigen}}$  eigenvectors corresponding to the smallest  $n_{\text{eigen}}$  eigenvalues, where  $n_{\text{eigen}}$  is a hyper-parameter shared for all graphs. We refer this implementation as Hodge1Lapsim  $(E) = \sum_{i=1}^{n_{\text{eigen}}} \phi(\mathbf{u}_i)$ . (ii) We take the absolute value of each element in eigenvectors before passing them to the MLP, which we denote as Hodge1Lapabs  $(E) = \sum_{i=1}^{n_{\text{eigen}}} \phi(|\mathbf{u}_i|)$ , where  $|\cdot|$  means taking element-wise absolute value. It is remarkable that, while Hodge1Lapproj is sign-invariant and basis-invariant, Hodge1Lapsim is not invariant to both sign and basis, and Hodge1Lapabs is sign-invariant yet not basis-invariant. We also allow combination of the above implementations; see Appendix E for more implementation details.
Our Hodge1Lap has elegant geometric meanings thanks to the spectral properties of  $L_{1}$ . For example, the kernel space of  $L_{1}$  related to the zero eigenvalues is fully capable of detecting cycles and rings in graphs [23], which can play a significant role in many domains. In molecular graphs, for example, cycle structures such as benzene rings have crucial effects on molecular properties. Hodge1Lap is able to extract such rings in a natural way rather than manually listing them, and Hodge1Lapabs is able to differentiate edges from distinct cycles. Intuitively, according to the Hodge decomposition theorem, any vector field defined on edges  $\mathcal{C}^1$  can be decomposed into three orthogonal components: a solenoidal component, a gradient component and a harmonic (both divergence-free and curl-free) component; see Appendix A.  $\ker(\mathbf{L}_1)$  is the harmonic component, and since divergence-free and curl-free edge flows can only appear on cycles, the eigenvectors corresponding to  $\ker(\mathbf{L}_1)$  therefore mark out the cycles in the graph; see Appendix C.2.2 for more technical details and illustrative examples. Moreover, taking into account more subspaces other than the kernel space of  $L_{1}$ , Hodge1Lap contains other structure information since the eigenvectors are real and continuous vectors. Ideally, one can apply any sign and basis invariant functions to obtain a universal approximator [32] for functions on 1-faces besides projections, see Section 6 for general conclusions.
6 Random walk on higher-order and inter-order simplices
In Section 4 and Section 5, we systematically analyze the random walk and Hodge Laplacian-based PE and SE on 0-simplices (node level) and 1-simplices (edge level), respectively. As we have shown, introducing higher-order simplices into random walk benefits their theoretical expressivity. In this section, we formally introduce random walks on higher-order simplices and analyze their expressivity. We will also investigate the spectral analysis of Hodge  $k$  Laplacians, whose normalization forms are closely related to random walks on  $k$ -simplices. Besides random walk within same-order simplices, we define a novel inter-order random walk that is able to transmit within different orders of simplices. This random walk scheme incorporates and unifies a wide range of simplicial networks [12, 8, 14].
6.1 Higher-order Hodge Laplacians and random walk
The  $k$ -th order Hodge Laplacian is defined as  $\mathbf{L}_k = \mathbf{B}_k^*\mathbf{B}_k + \mathbf{B}_{k+1}\mathbf{B}_{k+1}^* = \mathbf{L}_{k,down} + \mathbf{L}_{k,up}$ . Analogous to  $\mathbf{L}_1$ , a properly normalized Hodge  $k$  Laplacian  $\tilde{\mathbf{L}}_{\mathbf{k}}$  corresponds to a  $k$ -th order random walk on  $k$ -simplices in the lifted space. The matrix representation for the lifting is  $\mathbf{V}_k = [+\mathbf{I}_{n_k} - \mathbf{I}_{n_k}]^T \in \mathbb{R}^{2n_k \times n_k}$ , where  $n_k = |S_k|$  is the number of  $k$ -simplices in the simplicial complex  $\mathcal{K}$ . For undirected graphs, one only needs to sum over different orientations to get the cochain group  $\mathcal{C}^k$  from  $\mathcal{D}^k$ , where  $|\mathcal{D}^k| = 2|\mathcal{C}^k|$  is the cochain group in the lifted space. The transition matrix  $\hat{\mathbf{P}}_k$  for  $k$ -th order random walk is defined through  $-\frac{1}{2}\tilde{\mathbf{L}}_{\mathbf{k}}\mathbf{V}_k^T = \mathbf{V}_k^T\hat{\mathbf{P}}_k$ .
Similarly to the edge-level random walk in the lifted space, the transition matrix  $\hat{\mathbf{P}}_k$  describes that each step of  $k$ -th order random walk move towards either  $k$ -down neighbors or  $k$ -up neighbors. When going through the upper adjacent  $k + 1$  faces, the walk uniformly transits to an upper adjacent  $k$ -simplex with different orientation relative to the shared  $k + 1$  face, unless it has no upper adjacent faces. If the step is taken towards a lower-adjacent  $k - 1$  face, the walk transits along or against the original direction to one of its  $k$ -down neighbors.
Based on  $\hat{\mathbf{P}}_k$ , we can design  $k$ -th order RWSE for  $k$ -simplicial data according to the  $k$ -th order random walk,  $k - \mathrm{RWSE} = \psi_k(\hat{\mathbf{P}}_k)$ , where  $\psi_k$  is an injective function that acts on either  $\hat{\mathbf{P}}_k$  or its polynomials. If we maintain all  $k$ -RWSE for  $k = 0,1,\ldots ,K$  in a simplicial complex  $\mathcal{K}$  with
dimension larger than  $K$ , then we can get a more powerful algorithm by adding  $K + 1$ -RWSE to the  $K + 1$ -simplices in  $\mathcal{K}$ .
In addition to directly making use of the random walk on the  $k$ -simplices, spectral analysis of  $\mathbf{L}_k$  also sheds light on PE designs for higher-order simplicial data. Based on the eigenvalues and eigenvectors of  $\mathbf{L}_k$ , we can build permutation equivariant and basis invariant functions defined on  $\mathcal{K}_{k+1}$  that can simulate arbitrary  $k$ -cochain or  $k$ -form. Concretely, if we use the normalized version of  $k$ -th Hodge Laplacian  $\Delta_k$  as in [24], the eigenvalues of  $\Delta_k$  will be compact  $0 \leq \lambda \leq k+2$ . Then applying a permutation equivariant and basis-invariant function such as Unconstrained BasisNet [32] on the eigenvalues and eigenvectors, we are able to approximate any  $k$ -form which is basis-invariant. We refer interested readers to Appendix C.3 for more details.
6.2 Inter-order random walk
The concept of random walk can be even generalized to a more universal version, which we denote as inter-order random walk. In each step, the inter-order random walk at a  $k$ -simplex can transit not only to the  $k$ -down neighbors and  $k$ -up neighbors (they are all  $k$ -simplices as well), but also to lower adjacent  $k-1$ -simplices and upper adjacent  $k+1$ -simplices. Here we denote the (unnormized) adjacent matrix for the inter-order random walk on a  $K$ -order simplicial complex  $\mathcal{K}$  as  $\mathcal{A}_K(\mathcal{K})$  which is defined as
$$
\mathcal {A} _ {K} (\mathcal {K}) = \left[ \begin{array}{c c c c c c} \mathbf {L} _ {0} & \mathbf {B} _ {1} & & & & \\ \mathbf {B} _ {1} ^ {T} & \mathbf {L} _ {1} & \mathbf {B} _ {2} & & & \\ & \dots & \dots & \dots & & \\ & & \dots & \dots & \dots & \\ & & & \mathbf {B} _ {K - 1} ^ {T} & \mathbf {L} _ {K - 1} & \mathbf {B} _ {K} \\ & & & & \mathbf {B} _ {K} ^ {T} & \mathbf {L} _ {K} \end{array} \right] \tag {6}
$$
which is a block matrix with  $\mathbf{L}_k$  in the  $k$ -th diagonal block,  $\mathbf{B}_k^T$  and  $\mathbf{B}_{k+1}$  in the offset  $\pm 1$  diagonal blocks, while all other blocks are zeros. Although Chen et al. [14] also mentioned a similar block matrix, they do not pose a concrete form of the off-diagonal blocks. The inter-order adjacent matrix we define has a clear physical interpretation that one can only transform to simplices with different orders that are boundaries and co-boundaries of current simplex. A properly normalized version  $\tilde{\mathcal{A}}_K$  can describe the inter-order random walk with a certain rule. Here, we give a property of the power of  $\mathcal{A}_K$  which still holds in normalized versions.
$$
\mathcal {A} _ {K} ^ {r} = \left[ \begin{array}{c c c c c} p _ {r} (\mathbf {L} _ {0}) & q _ {r - 1} (\mathbf {L} _ {0, u p}) \mathbf {B} _ {1} & & & \\ q _ {r - 1} (\mathbf {L} _ {1, d o w n}) \mathbf {B} _ {1} ^ {T} & p _ {r} (\mathbf {L} _ {1}) & q _ {r - 1} (\mathbf {L} _ {1, u p}) \mathbf {B} _ {2} & & \\ & \dots & \dots & \dots & \\ & & \dots & \dots & \dots \\ & & & q _ {r - 1} (\mathbf {L} _ {K, d o w n}) \mathbf {B} _ {K} ^ {T} & p _ {r} (\mathbf {L} _ {K}) \end{array} \right] \tag {7}
$$
where  $p_r(\cdot)$  and  $q_{r}(\cdot)$  are polynomials with maximum order  $r$ . The above equation states that simplices with differences of order larger than one cannot directly exchange information even after infinite rounds, but they can affect each other through the coefficients in  $p_r$  and  $q_{r - 1}$  in the blocks on the offset  $\pm 1$ -diagonal blocks.
Several previous works such as [8] can be unified by  $\mathcal{A}_K$ . Additionally, we can make use of  $\mathcal{A}_K^r$  to build random walk-based positional encoding for all simplices in the  $K$ -dimensional simplicial complex that contains rich information.
In this section, we present a comprehensive ablation study on Zinc-12k to investigate the effectiveness of our proposed methods. We also verify the performance on graph-level OGB benchmarks. Due to the limited space, experiments on synthetic datasets and more real-world datasets as well as experimental details are presented in Appendix E.
Ablation study on Zink-12k. Zinc-12k [17] is a popular real-world dataset containing 12k molecules. The task is the graph-level molecular property (constrained solubility) regression. In our ablation study, we use GINE [25], GAT [45], PNA [15], SSWL+ [50], GPS [38] and GRIT [33] as our base models, where the first three are message-passing based GNNs, SSWL+ is an instance of subgraph GNN, while GPS and GRIT are recent SOTA graph transformers. Four different factors are studied: (1) the node-level PE or SE, where RWSE refers to [18], LapPE refers to [30] and
"-" suggests no node-level PE/SE; (2) EdgeRWSE, the edge-level SE based on spatial domain of 1-down random walk, where "directed" and "undirected" are used to distinguish the two types of simplified version of 1-down random walk; (3) Hodge1Lap, the edge-level PE based on spectra of  $\mathbf{L}_1$ , where "abs" refers to the sign-invariant method (summing over absolute values of eigenvectors, or Hodge1Lapabs), and "project" refers to the sign and basis invariant method (project the unit vector into interested subspace, or Hodge1Lapproj); (4) RWMP, a novel Random Walk Message Passing scheme we propose, which performs message passing based on probability calculated by a distance metric; see Appendix D for details of RWMP.
The full results of performance on the Zinc dataset are reported in Table 1. Note that all our base models are improved when augmented with our EdgeRWSE or Hodge1Lap: both GAT and PNA reduce by over  $50\%$  MAE. In particular, a simple GINE without using any transformer or subgraph GNN variations is able to surpass GPS with our PE/SE, verifying the impressive effectiveness of our proposed methods. Applying EdgeRWSE and Hodge1Lap to GRIT results in new State-of-the-Art performance. Regarding ablation, all variants of our EdgeRWSE and Hodge1Lap can improve performance of base models, see Appendix E for more implementation details of these variants. One may observe that RWSE is significantly beneficial in this task, and combining node-level RWSE and our edge-level PE/SE methods would lead to a further performance gain. In general, Hodge1Lap shows better performance than EdgeRWSE, indicating the effectiveness of embedding structures such as rings through spectral analysis. The effect of whether EdgeRWSE is directed or the implementation method in Hodge1Lap is rather small. We also observe that Full-EdgeRWSE, Inter-RWSE, and CellularRWSE are beneficial, see Appendix E for more details. Additionally, the RWMP mechanism is also capable of improving performance, which we will analyze in Appendix D.
Experiments on OGB benchmarks. We also verify the performance of EdgeRWSE and Hodge1Lap on graph-level OBG benchmarks, including the ogbg-molhiv and ogbg-molpcba datasets. The results are shown in Table 2. We apply our Hodge1Lap and EdgeRWSE to both GatedGCN and GPS(consists of GatedGCN and Transformer) and show that our methods can improve both architectures. In general, both two edge-level PE/SE are able to achieve comparable performance as the SOTA models, though EdgeRWSE suffers from overfitting on ogbg-molhiv. It should be noted that SOTA results on ogbg-molhiv typically involve manually crafted structures, including GSN [9] and CIN [7]. Natural methods and complex models usually suffer from overfitting and cannot generalize well in the test set.
In this paper, we propose to facilitate graph neural networks through the lens of random walk on simplicial complexes. The random walk on  $k$ -th order simplices is closely related to Hodge  $k$  Laplacian  $\mathbf{L}_k$ , and we emphasize that both spatial analysis of random walk and spectra of  $\mathbf{L}_k$  can improve the theoretical expressive power and performance of GNNs. For 0-simplices, we connect a number of exciting PE and SE methods (such as RWSE) via node-level random walk, and further provide a theoretical expressivity bound. For 1-simplices, we propose two novel edge-level PE and SE methods, namely EdgeRWSE and Hodge1Lap. EdgeRWSE directly encodes information based on edge-level random walk, while Hodge1Lap is the first sign and basis invariant edge-level PE based on Hodge-1 Laplacian spectra. We also generalize our theory to arbitrary-order simplices, showing how  $k$ -order and inter-order random walk as well as spectral analysis of Hodge Laplacians can facilitate graph and simplicial learning. Besides analyzing theoretical expressive power and physical meanings of these random walk-based methods, we also verify the effectiveness of our methods, which achieve SOTA or highly competitive performance on several datasets.
[1] Ralph Abboud, Ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. CoRR, abs/2010.01179, 2020. URL https://arxiv.org/abs/2010.01179.
[2] Chandra A.K., Raghavan P., Ruzzo W.L., and et al. The electrical resistance of a graph captures its commute and cover times. Comput Complexity 6, 312-340, 1996.
[3] M. Balcilar, Pierre Héroux, Benoit Gaüzère, Pascal Vasseur, Sébastien Adam, and Paul Honeine. Breaking the limits of message passing graph neural networks. In International Conference on Machine Learning, 2021.
[4] D. Beaini, Saro Passaro, Vincent L'etourneau, William L. Hamilton, Gabriele Corso, and Pietro Lio'. Directional graph networks. In International Conference on Machine Learning, 2020.
[5] Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, G. Balamurugan, Michael M. Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. ArXiv, abs/2110.02910, 2021.
[6] Deyu Bo, Chuan Shi, Lele Wang, and Renjie Liao. Specformer: Spectral graph neural networks meet transformers. ArXiv, abs/2303.01028, 2023.
[7] Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yu Guang Wang, Pietro Lio', Guido Montufar, and Michael M. Bronstein. Weisfeiler and lehman go cellular: Cw networks. In Neural Information Processing Systems, 2021.
[8] Cristian Bodnar, Fabrizio Frasca, Yu Guang Wang, Nina Otter, Guido Montúfar, Pietro Lio', and Michael M. Bronstein. Weisfeiler and lehman go topological: Message passing simplicial networks. ArXiv, abs/2103.03212, 2021.
[9] Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael M. Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):657-668, 2023. doi: 10.1109/TPAMI.2022.3154319.
[10] Xavier Bresson and Thomas Laurent. Residual gated graph convnets. ArXiv, abs/1711.07553, 2017.
[11] Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. CoRR, abs/1611.08097, 2016. URL http://arxiv.org/abs/1611.08097.
[12] Eric Bunch, Qian You, Glenn Moo Fung, and Vikas Singh. Simplicial 2-complex convolutional neural nets. arXiv: Algebraic Topology, 2020.
[13] Jin-Yi Cai, Martin Fürer, and Neil Immerman. An optimal lower bound on the number of variables for graph identification. Combinatorica, 12:389-410, 1989.
[14] Yuzhou Chen, Yulia R. Gel, and H. Vincent Poor. Bscnets: Block simplicial complex neural networks. In AAAI Conference on Artificial Intelligence, 2021.
[15] Gabriele Corso, Luca Cavalleri, D. Beaini, Pietro Lio', and Petar Velickovic. Principal neighbourhood aggregation for graph nets. ArXiv, abs/2004.05718, 2020.
[16] Hanjun Dai, Elias B. Khalil, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. CoRR, abs/1704.01665, 2017. URL http://arxiv.org/abs/1704.01665.