SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cai Zhou
zhouc20@mails.tsinghua.edu.cn
Xiyuan Wang
wangxiyuan@pku.edu.cn
Muhan Zhang
muhan@pku.edu.cn
Node-level random walk has been widely used to improve Graph Neural Networks. However, there is limited attention to random walk on edges and, more generally, on  $k$ -simplices. This paper systematically analyzes how random walk on different orders of simplicial complexes (SC) facilitates GNNs in their theoretical expressivity. First, on 0-simplices or node level, we establish a connection between existing positional encoding (PE) and structure encoding (SE) methods through the bridge of random walk. Second, on 1-simplices or edge level, we bridge edge-level random walk and Hodge 1-Laplacians and design corresponding edge PE respectively. In the spatial domain, we directly make use of edge level random walk to construct EdgeRWSE. Based on the spectral analysis of Hodge 1-Laplacians, we propose Hodge1Lap, a permutation equivariant and expressive edge-level positional encoding. Third, we generalize our theory to random walk on higher-order simplices and propose the general principle to design PE on simplices based on random walk and Hodge Laplacians. Inter-level random walk is also introduced to unify a wide range of simplicial networks. Extensive experiments verify the effectiveness of our random walk-based methods.
Graph neural networks (GNNs) have recently achieved great success in tasks with graph-structured data, benefiting many theoretical application areas, including combinatorial optimization, bioinformatics, social-network analysis, etc. [11, 29, 16]. Two important aspects to evaluate GNN models are their theoretical expressivity in distinguishing non-isomorphic graphs, and their performance on real-world tasks. Positional encoding (PE) and structure encoding (SE) are widely adopted methods to enhance both theoretical expressivity and real-world performance of GNNs. Generally, PE encodes the information of the nodes' local or global positions, while SE provides information about local or global structures in the graph. For example, Kreuzer et al. [30] uses eigenvectors of the graph Laplacian, Dwivedi et al. [18] proposes to use diagonal elements of the  $t$ -step random walk matrix, and Bouritsas et al. [9] manually count some predefined structures. There are also some methods based on pair-wise node distances, such as the shortest path distance [31], the heat kernel [20], and the graph geodesic [35]. Although some work theoretically analyzes some of these methods [51], there are still some left-out methods, and people lack a unified perspective to view all these PE and SE designs. Moreover, most existing methods focus only on node data, while PE and SE on edge data as well as some higher-order topological structures are waited to be studied.
In addition to PE and SE, geometric deep learning has recently become a central topic. Researchers are inspired by concepts of differential geometry and algebraic topology, which resulted in many works on simplices and simplicial complexes [8, 7, 47]. Despite their capability to deal with higher-order structures, these simplicial networks should follow orientation symmetry, which brings difficulties in their applications in undirected graphs. This work connects these two separate areas via a central
concept: random walk on simplicial complexes. On the one hand, by introducing concepts of higher-order simplicial complexes, we can design more PE and SE methods that are both theoretically and practically powerful. On the other hand, PE and SE greatly facilitate simplicial data and benefit graph learning.
In summary, we first connect a number of existing PE and SE methods through the bridge of node-level random walk on 0-simplices. Then, for 1-simplices or edges, we design two novel sign and basis invariant edge-level PE and SE, namely EdgeRWSE and Hodge1Lap. EdgeRWSE uses an edge-level random walk directly to capture structure information, while Hodge1Lap is based on spectral analysis of Hodge 1 Laplacian, which is closely related to random walk on edges. We further generalize our theory to random walk on higher-order and inter-order simplices to facilitate graph and simplicial learning. Our methods achieve State-Of-The-Art or highly competitive performance on several datasets and benchmarks. Code is available at https://github.com/zhouc20/HodgeRandomWalk.
Theoretical expressivity and Weisfeiler-Lehman test. Weisfeiler-Lehman tests are a classical family of algorithms to distinguish non-isomorphic graphs. Previous work has built connections between the expressivity of GNNs and the WL hierarchy. Some classical conclusions include that for  $k \geq 2$ ,  $k + 1$ -dimensional WL is more powerful than  $k$ -WL. [46] proves that traditional message-passing neural networks (MPNN) are not more powerful than 1-WL. There is another variation of the WL test called the Folklore Weisfeiler-Lehman (FWL) test, and  $k$ -FWL is equivalent to  $k$ -WL in expressivity for  $k \geq 1$ .
Symmetry in graph and simplicial learning. Symmetry is a central topic in graph and simplicial learning. In graph learning, node features and edge features need to be permutation (i.e., relabeling of nodes or edges) equivariant, while the graph features should be permutation invariant. In simplicial learning, one needs to further orientation symmetry [47] in an oriented simplicial complex (SC). The incidence relations and the simplicial adjacencies in an oriented SC are altered when the orientations are reversed. The  $k$ -form remains invariant to this transformation, while the features of  $k$ -simplices are equivariant in terms of the basis. [32] also state the standard that the graph-level functions (and in the context of SC,  $k$ -forms) should be invariant to both sign and basis (either of orientation or of space), which is a basic rule for our PE and SE designs.
Graphs. We denote a graph as  $G(V,E,A)$ , where  $V, E$  is the set of nodes and the set of edges, respectively, and  $A$  is the adjacency matrix for the nodes. For convenience, we use  $n = |V|$  and  $m = |E|$  to represent the number of nodes and edges in the graph  $G(V,E,A)$ . In an undirected graph, for any  $u, v \in V$ , we have  $(u,v) \in E \Leftrightarrow (v,u) \in E$ . Let  $\mathcal{N}(v,G) = \{u \in V | (u,v) \in E\}$  denote the set of neighbors of node  $v$  in graph  $G$ . Let diagonal matrix  $D = \text{diag}(d_1, \ldots, d_n)$ , where  $d_i$  is the degree of node  $v_i$ .
The transition matrix of a typical random walk at node level is  $P = D^{-1}A$ , which indicates that in each step the walk moves from the current node  $v$  to one of its neighboring nodes  $u \in \mathcal{N}(v, G)$  with equal probabilities. Consequently, a  $t$  step of the aforementioned random walk corresponds to a transition matrix  $P^t$ .
Discrete Hodge Laplacian of abstract simplicial complex. An abstract simplicial complex  $\mathcal{K}$  on a finite set  $V$  is a collection of subsets of  $V$  that is closed under inclusion. In our paper,  $V$  will be a vertex set  $[n] = \{1,2,\dots,n\}$  if without special statement. An element of cardinality  $k + 1$  is called a  $k$ -face or  $k$ -simplex of  $\mathcal{K}$ . For instance, 0-faces are usually called vertices, 1-faces are directed edges, and 2-faces are 3-cliques (triangles) with an orientation. We denote the collection of all  $k$ -faces of  $\mathcal{K}$  as  $S_{k}(\mathcal{K})$ . The dimension of a  $k$ -face is  $k$ , and the dimension of a complex  $\mathcal{K}$  is defined as the maximum dimension of the faces in  $\mathcal{K}$ .
The definition of neighbors of simplices is crucial in this paper. Two  $k + 1$ -simplices sharing a collective  $k$ -face are called  $k$ -down neighbors, and two  $k$ -simplices sharing a collective  $k + 1$ -simplex are called  $k + 1$ -up neighbors. Generally, a face  $F$  is chosen as an ordering on its vertices and is said
to be oriented, denoted by  $[F]$ . For any permutation element  $\sigma \in \mathcal{G}_{k + 1}$  where  $\mathcal{G}_{k + 1}$  is the symmetric group of permutations on  $\{0,\dots,k\}$ , two orders of vertices transformed by  $\sigma$  are said to determine the same orientation if  $\sigma$  is an even permutation and opposite if  $\sigma$  is odd.
In the Hilbert space, the matrix representations of boundary and coboundary operators are adjacency matrices of order  $k$  and  $k + 1$  simplices. In order to keep coordinate with most existing literature, we write the adjacent matrix of  $k$ -th and  $k + 1$ -th simplices as  $\mathbf{B}_{k + 1} \in \mathbb{R}^{|S_k| \times |S_{k + 1}|}$ .  $\mathbf{B}_{k + 1}[i,j] = 1$  if the  $i$ -th  $k$ -simplex and  $j$ -th  $k + 1$ -simplex are adjacent and share the same direction,  $\mathbf{B}_{k + 1}[i,j] = -1$  if adjacent with opposite directions, and 0 if they are not adjacent. For example,  $\mathbf{B}_1$  is the node-to-edge incidence matrix.
In discrete Hodge-deRham theory, the  $k$ -th order Hodge Laplacian is defined as
$$
\mathbf {L} _ {k} = \mathbf {B} _ {k} ^ {*} \mathbf {B} _ {k} + \mathbf {B} _ {k + 1} \mathbf {B} _ {k + 1} ^ {*} \tag {1}
$$
where  $\mathbf{B}_k^* = \mathbf{B}_k^T$  is the adjoint of  $\mathbf{B}_k$  and is equivalent to the transpose of  $\mathbf{B}_k$  in Hilbert space. A special case is that when  $k = 0$ ,  $\mathbf{B}_0$  is not defined and  $\mathbf{L}_0 = \mathbf{B}_1\mathbf{B}_1^* = \mathbf{D} - \mathbf{A}$  is exactly the graph Laplacian. We refer readers to Appendix C.2.2 for an illustrative calculation example of Hodge Laplacians. In our following texts, we will make use of higher-order Hodge Laplacians such as  $\mathbf{L}_1$  rather than previously used  $\mathbf{L}_0$  alone.
The kernel space of  $\mathbf{L}_k$  is called the  $k$ -th cohomology group:  $\tilde{\mathcal{H}}^k(\mathcal{K},\mathbb{R}) \coloneqq \ker(\mathbf{B}_{k+1}^*) / \operatorname{im}(\mathbf{B}_k^*) \cong \ker(\mathbf{B}_{k+1}^*) \cap \ker(\mathbf{B}_k) = \ker(\mathbf{L}_k)$ . We will write  $\tilde{\mathcal{H}}^k(\mathcal{K},\mathbb{R})$  simply as  $\tilde{\mathcal{H}}^k$  without causing confusion. The kernel spaces of Hodge Laplacians are closely associated with harmonic functions and will play an important role in our following analysis. Particularly, the multiplicity of zero eigenvalues of  $\mathbf{L}_k$ , or the dimension of null space of Hodge  $k$ -Laplacian  $\ker(\mathbf{L}_k)$ , is called the  $k$ -th Betti number  $\beta_k$  [23]. This is exactly the number of cycles composed of  $k$ -simplicials that are not induced by a  $k$ -boundary, or intuitively,  $k$ -dimensional "holes" in the simplicial complex  $\mathcal{K}$ . For example, zero eigenvalues and their eigenvectors of  $\mathbf{L}_0$  are associated with the 0-th cohomology group of the graph, corresponding to the connected components of the graph. The zero eigenvalues and eigenvectors of  $\mathbf{L}_1$  are associated with cycles (in the usual sense), and those of  $\mathbf{L}_2$  correspond to cavities. We refer readers to Appendix C.2.2 for detailed explanations and illustrative examples of cohomology groups.
4 Random walk on 0-simplices
Random walk on 0-simplices or at node level has been studied systematically. Previous work has established comprehensive analysis on the theoretical properties of node-level random walk, which provide theoretical insights into the design of random walk-based methods. However, there is still limited research on the theoretical expressivity of random walk-based positional encoding (PE) and structure encoding (SE) methods. In this section, we establish connections between several PE and SE with node-level random walk, and provide theoretical expressive power bounds for them.
RWSE. [52] and Dwivedi et al. [18] propose a structure encoding method based on node-level random walk, which we denote as RWSE. Concretely, RWSE considers  $K$  steps of random walk at the node level of the graph, obtaining  $\mathbf{P}, \mathbf{P}^2, \dots, \mathbf{P}^K$ . Then the method only takes into account each node's return probabilities to itself, i.e. the diagonal elements of  $\mathbf{P}^k$ ,  $k = 1, 2, \dots, K$ . For each node  $v_i$ , the RWSE feature is  $h_i^{RWSE} = [\mathbf{P}_{ii}, \mathbf{P}_{ii}^2, \dots, \mathbf{P}_{ii}^K]$ . Compared with encoding methods based on graph Laplacian eigenvalues and eigenvectors, this method is sign and basis invariant. It internally captures some structure information within  $K$ -hops and achieves impressive results in experiments [38]. However, there are limited investigations on the theoretical expressivity of RWSE and its extensions. Here, we provide a theoretical bound of positional and structure encoding methods based on random walk transition matrix  $\mathbf{P}$ .
Theorem 4.1. RWSE is strictly less powerful than 2-FWL, i.e. RWSE  $\prec$  2-FWL.
The above expressivity bound holds because 2-FWL can simulate the multiplication and injective transformations of a matrix, including the adjacency matrix  $\mathbf{A}$ . Therefore, 2-FWL is capable of obtaining  $\mathbf{P}^k$ ,  $k \in \mathbb{N}$ . Specifically, a block of PPGN [34] can simulate one time of matrix multiplication. Moreover, RWSE is strictly less expressive than 2-FWL, since it loses much structure information when taking the diagonal elements of  $\mathbf{P}^k$  only. In other words, RWSE is a summary of full random walk transition probabilities (on spatial domain), which accelerates calculation at the cost of losing expressivity.
Resistance distance and random walk. In addition to RWSE, there are a number of positional encoding methods closely related to the node-level random walk. A.K. et al. [2], Zhang et al. [51] connect commute time in random walks with resistance in electrical networks, which can be used as a PE method called resistance distance (RD). Zhang et al. [51] prove that RD and shortest path distance (SPD) [31] are both upper-bounded by 2-FWL in expressive power.
Positive definite kernels based on graph Laplacian spectrum. Graph Laplacian, or Hodge 0-Laplacian as we refer to later, is closely connected with random walk on graph. The definition of graph Laplacian is  $\mathbf{L}_0 = \mathbf{D} - \mathbf{A} = \delta_0^*\delta_0 = \Delta_0$ . Through the spectrum of  $\mathbf{L}_0$ , we are able to define a family of positive definite kernels on graphs [42] by applying a regularization function  $r$  to the spectrum of  $\mathcal{L}_0$ :  $K_{r} = \sum_{i = 1}^{m}r(\lambda_{i})\mathbf{u}_{i}\mathbf{u}_{i}^{T}$ , where  $\mathbf{L}_0 = \sum_i\lambda_i\mathbf{u}_i\mathbf{u}_i^T$  is the eigenvalue decomposition. For example, the heat kernel or the diffusion kernel [20] can be incorporated if  $r(\lambda_i) = e^{-\beta \lambda_i}$ . Other methods directly use eigenvectors as PE [30]. These results imply that spectral analysis of graph Laplacians can also inspire more powerful PE and SE, and we will generalize graph Laplacian  $\mathbf{L}_0$  to arbitrary order of Hodge  $k$  Laplacians in the following section to facilitate graph learning.
5 Random walk on 1-simplices
While node-level random walk has been widely studied, edge-level random walk is still limited. In this section, we will first introduce Hodge 1 Laplacian  $\mathbf{L}_1$ , as well as its connection with random walk on 1-simplices (in the lifted space) and thus edges of undirected graph. Analogous to node-level RWSE, we introduce EdgeRWSE, a more theoretically powerful PE for edges. Furthermore, we systematically analyze the spectra of  $\mathbf{L}_1$  and propose a novel Hodge1Lap PE, the first sign and basis invariant edge-level positional encoding that makes use of the spectra of  $\mathbf{L}_1$  instead of the previously adopted  $\mathbf{L}_0$  only.
5.1 Normalized Hodge-1 Laplacian and edge-level random walk
Theoretical analysis of edge-level random walk. The standard Hodge  $k$ -Laplacian is  $\mathbf{L}_k = \mathbf{B}_k^*\mathbf{B}_k + \mathbf{B}_{k+1}\mathbf{B}_{k+1}^*$ , and there are a number of normalized Hodge Laplacian because the normalization is rather flexible. Schaub et al. [41] propose a normalized form for Hodge 1-Laplacian  $\mathbf{L}_1$  with a clear interpretation of a random walk in the lifted edge space. Concretely,
$$
\tilde {\mathbf {L}} _ {1} = \mathbf {D} _ {2} \mathbf {B} _ {1} ^ {*} \mathbf {D} _ {1} ^ {- 1} \mathbf {B} _ {1} + \mathbf {B} _ {2} \mathbf {D} _ {3} \mathbf {B} _ {2} ^ {*} \mathbf {D} _ {2} ^ {- 1} \tag {2}
$$
where  $\mathbf{D}_2$  is the diagonal matrix with adjusted degrees of each edge  $\mathbf{D}_2 = \max (diag(|\mathbf{B}_2|\mathbf{1}),I)$ ,  $\mathbf{D}_1$  is the diagonal matrix of weighted degree of nodes  $\mathbf{D}_1 = 2\cdot diag(|\mathbf{B}_1|\mathbf{D}_2\mathbf{1})$ , and  $\mathbf{D}_3 = \frac{1}{3}\mathbf{I}$ .
To interpret this normalized Hodge 1-Laplacian  $\tilde{\mathbf{L}}_1$ , Schaub et al. [41] introduce a lifted space of edges, where the original  $m = |S_1|$  directed edges are lifted to  $2m$  directed edges. For example, if  $(i,j)\in S_{1}$ , then we add  $(j,i)$  to the lifted space. Consequently, the edge flow  $\mathbf{f}\in \mathcal{C}^1$  expands to a larger space  $\mathcal{D}^1$  where there are two orientations for each edge,  $|\mathcal{D}^1| = 2|\mathcal{C}^1|$ . The matrix representation for this lifting procedure is  $\mathbf{V} = \left[+{\bf I}_m - {\bf I}_m\right]^T\in \mathbb{R}^{2m\times m}$ . Then the probability transition matrix for this lifted random walk corresponding to  $\tilde{L}_{1}$  is  $\hat{\mathbf{P}}: - \frac{1}{2}\tilde{\mathbf{L}}_{1}\mathbf{V}^{T} = \mathbf{V}^{T}\hat{\mathbf{P}}$ . In practice, we also perform a simpler row-wise normalization over  $\mathbf{L}_1$  to obtain another form of probability transition matrix.
Using  $\hat{\mathbf{P}}$ , we can construct an edge-level random walk-based PE method to enrich edge data by encoding structure information, analogous to node-level RWSE. We will also discuss some variations and simplified versions of the aforementioned random walk on 1-simplices and theoretically analyze their expressivity.
EdgeRWSE. Similar to node-level random walk, a well-defined edge-level random walk contains some structure information and can be used to facilitate edge data, namely edge-level positional encoding. While node-level positional encodings have been widely studied, the edge-level positional encoding is a nearly blank field.
Inspired by (node-level) RWSE, EdgeRWSE is based on edge-level random walk. A full version of EdgeRWSE is based on the full edge-level random walk as we have stated above and in [41]. For undirected graphs, two edges with opposite directions  $(i,j)$  and  $(j,i)$  are again merged by summing
the two probabilities, that is, the lifted space  $\mathcal{D}^1$  is mapped back to  $\mathcal{C}^1$ . Generally speaking, PE can be based on any injection functions  $\psi$  in  $\hat{\mathbf{P}}$  and its powers.
$$
\operatorname {E d g e R W S E} (\hat {\mathbf {P}}) _ {i} = \psi ([ \hat {\mathbf {P}} ^ {k} ]), k = 1, 2, \dots K \tag {3}
$$
where  $K$  is the maximum steps we consider. One possible example is to encode the return probability of each edge, which is written  $\mathrm{EdgeRWSE}_{\mathrm{ret}}(\hat{\mathbf{P}})_i = \psi ([\hat{\mathbf{P}}_{ii}^k ]), k = 1,2,\ldots K$ . If  $\psi$  is well defined, the theoretical expressivity of the full EdgeRWSE above is able to break the 2-FWL bottleneck of node-level RWSE. In practice, we can apply neural networks like MLP or Transformer to encode  $\hat{\mathbf{P}}^k$  and concatenate them with the original edge features. Then any standard GNN is applicable for downstream tasks. If the GNN is at least as powerful as 1-FWL, then the GNN with EdgeRWSE is strictly more powerful than 1-FWL and can distinguish some non-isomorphic graph pairs in which 2-FWL fails.
In addition to the edge-level random walk in the lifted space of 1-simplicials in [41], we further define two simplified versions of the edge-level random walk only through lower adjacency. We neglect the 2-simplices or the triangles in our simplified version random walk, i.e. we only consider the 1-down neighbors that share a 0-simplices (node). In this way,  $\hat{\mathbf{P}}$  becomes  $\mathbf{P}_{\text{down}}$ . This simplification will lead to a theoretically weaker expressivity than using full  $\hat{\mathbf{P}}$ , which will be bounded by 2-FWL. However, this simplification is appropriate and beneficial for real-world data that contain a small number of triangles. We illustrate these two variations temporarily on undirected connected graphs without multiple edges and self-loops for simplicity.
The two variations of edge-level random walk via down-neighbors differ in whether two lower adjacent nodes of the edge have the same status. Concretely, the first type of edge-level random walk based on  $\mathbf{P}_{\text{down}}$ , which we define as directed 1-down random walk follows a two-stage procedure at every step. The walk first selects one of the two lower-adjacent nodes with equal probability 0.5 each, then moves towards the neighboring edges connected with the selected node with equal probabilities. If there are no other edges connected to the selected node, the walk returns to the original edge. On the other hand, the second type, which we denote as undirected 1-down random walk, chooses the two nodes  $u, v$  with probabilities proportional to their degrees minus one (since we want to exclude the case of returning to  $e$  itself). Consequently, the walk transits to all 1-down neighbors of the source edge with equal probabilities.
In a similar way as the full EdgeRWSE, we propose two simplified versions of EdgeRWSE based on directed 1-down and undirected 1-down random walk, both can be implemented in a rather flexible way. As a special case, the return probabilities of each edge after  $k = 1,\dots ,K$  steps are encoded, but notice again that it is not the only implementation choice.
We conclude by summarizing the expressivity of EdgeRWSE.
Theorem 5.1. Full EdgeRWSE can distinguish some non-isomorphic graphs that are indistinguish by 2-FWL. EdgeRWSE based on directed and undirected 1-down random walk are not more powerful than 2-FWL.
5.2 Sign and basis invariant edge-level positional encoding
Theoretical analysis of Hodge 1-Laplacian spectrum. Recall that the unnormalized Hodge 1-Laplacian is  $\mathbf{L}_1 = \mathbf{B}_1^T\mathbf{B}_1 + \mathbf{B}_2\mathbf{B}_2^T = \mathbf{L}_{1,down} + \mathbf{L}_{up}$ . Here, we analyze the theoretical properties of Hodge 1-Laplacian including its spectrum, which provides solid insights into our following designs.
Note that previous simplicial networks [12, 47, 8, 7] are orientation equivariant and permutation equivariant; thus, they can only be applied to simplicial complexes where all edges are directed. This is frustrating if we want to boost general learning on graphs rather than simplicial complexes alone. However, the spectral analysis of Hodge 1-Laplacian is applicable to undirected graphs. An important property of Hodge Laplacians is that their eigenvalues are invariant to permutation and orientation (if the simplices are oriented), thus they could be directly applied to analyze undirected graphs. Hence in this section, we temporarily omit discussion on permutation and orientation invariance since they naturally hold. Instead, we care more about the sign and basis invariance in the field of spectral analysis [32].
We can show that the nonzero eigenvalues of  $\mathbf{L}_{1,down}$  are the same as  $\mathbf{L}_{0,up}$  and hence  $\mathbf{L}_0$ . This implies that if there are no 2-simplicials (triangles), Hodge 1-Laplacian has the same nonzero
eigenvalues as Hodge 0-Laplacian. However, the corresponding eigenvectors still provide different information about the nodes and edges, respectively.
Theorem 5.2. The number of non-zero eigenvalues of Hodge 1-Laplacian  $L_{1}$  is not less than the number of non-zero eigenvalues of Hodge 0-Laplacian  $L_{0}$ .
One direct conclusion is that graph isomorphism based on Hodge 1-Laplacian isospectra is strictly more powerful than Hodge 0-Laplacian. Here we draw a conclusion on the theoretical expressivity of the  $L_{1}$  isospectra:
Theorem 5.3.  $L_{1}$  isospectra is incomparable with 1-FWL and 2-FWL.