SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Experiments on synthetic datasets. To verify the theoretical expressive power of Hodge1Lap and EdgeRWSE, we carried out experiments on two classical synthetic datasets: EXP and SR25. EXP [1] contains 600 pairs of non-isomorphic graphs that 1-WL and 2-WL fail to distinguish. SR25 [3] contains 15 non-isomorphic strongly regular graphs (i.e., 105 non-isomorphic pairs) that 3-WL fails to distinguish. An accuracy of  $50\%$  on EXP and  $6.67\%$  on SR25 suggests the model fails to distinguish any non-isomorphic graphs in the dataset. The results are shown in Table 3. The 1-WL equivalent GINE back-end model can fully distinguish EXP and SR25 when augmented by Hodge1Lap and Full-EdgeRWSE, indicating that both methods can distinguish non-isomorphic graph pairs that 3-WL (and 2-FWL) fails, which is consistent with our theoretical analysis.
Experiments on MNIST and CIFAR10. Since the graphs are all directed, only the directed version of EdgeRWSE is applicable since a random walk can only go along the edge, while Hodge1Lap is also applicable if we follow the strict directions of edges and simplices. Here, we evaluate the effectiveness of EdgeRWSE acting on GatedGCN and GPS, see Table 4. With EdgeRWSE, the GPS model (which consists of a GatedGCN and a Transformer in each layer) achieves the best on MNIST and second best on CIFAR10. It is remarkable that a simple GatedGCN is greatly enhanced by EdgeRWSE, achieving highly competitive performance on both datasets. This verifies the effectiveness of EdgeRWSE in models involving edge features.
Experiments on LRGB. The results are shown in Table 5; the baseline results are adopted from [38]. On both Peptides-func and PCQM-Contact, GPS model with EdgeRWSE (we use undirected version) and Hodge1Lap (we use projection method) achieve the best and second best performance, respectively. On Peptides-struct, the improvement of edge-level PE/SE is not significant for GPS, but is obvious for the GatedGCN base model. Admittedly, as pointed out by [44], the baseline models need some reassessing including hyper-parameter searching, and there are more SOTA results. However, we emphasize that our results are obtained without hyperparameter searching - our main goal is to verify the enhancement brought by EdgeRWSE and Hodge1Lap to the baseline models. SOTA methods can always be facilitated with our PE/SE to achieve better performance.
Trajectory prediction. Since our methods originate from simplicial complexes, we also verify the effectiveness of our Hodge1Lap on simplicial data. We adopt the trajectory prediction task (edge flow classification) in [8]. The edge flows are represented as signals on oriented simplicial complexes. We use the same synthetic dataset of trajectories on the simplicial complex in [8], where each triangle is treated as a 2-simplex. There are two holes in the complex, and all trajectories pass one of the holes, thus giving rise to two different classes to be distinguished. Due to the presence of the two holes, the trajectories of the two classes approximately correspond to orthogonal directions in the space of harmonic eigenfunctions of the  $\mathbf{L}_1$  Hodge-Laplacian [41]. The dataset contains
1000 train trajectories and 200 test trajectories. Following [8], to make the task more challenging for non-orientation-invariant models, all training complexes use the same orientation for the edges, while the test trajectories use random orientations. We adopt orientation-invariant message-passing simplicial networks (MPSN) [8] with orientation equivariant layers as the base models.
Note that since we aim to distinguish two orthogonal directions in the harmonic kernel space of  $\mathbf{L}_1$  Hodge-Laplacian, we do not use the projection implementation of Hodge1Lap; instead, we use the simple implementation Hodge1Lapsim (described in the main text) which embeds two eigenvectors respectively with a shared MLP. Theoretically, we can distinguish the two classes of flows with the help of eigenvectors since they correspond to two distinct cycles (i.e., two orthogonal directions in the kernel space). As shown in Table 6, all variants of MPSN achieve better performance (almost perfect test accuracy) when they are augmented with our Hodge1Lap. This is consistent with our theory, which experimentally verified the capability of Hodge1Lap to distinguish cycles.
E.3 Experiment details
Dataset split and repetitiveness. In all experiments, the datasets follow the common train/validation/test split as we stated in datasets description. The results of the baseline models are reported from [38]. For our results, we report the test results according to the best validation results. All experiments are run under 5 different random seeds, with mean and variance reported.
Hyperparameters. To verify that our methods are capable of improving the performance of the base models, all hyperparameters including training configuration and model hyperparameters are set the same as in [38]. For the edge PE/SE, we keep the embedding dimensions the same as the node PE/SE in GPS models. Hence, our experimental results strongly confirmed that our methods are beneficial for both naive and complex base models.
E.4 Implementation details
In this subsection, we provide more implementation details of our methods and more discussion on the corresponding variants. Our code is based on GPS [38], which enable us to integrate our methods into a comprehensive graph learning framework.
EdgeRWSE. In real-world datasets, we use both directed 1-down RWSE and undirected 1-down RWSE in our experiments, except in MNIST and CIFAR10 where only a directed walk is appropriate on the directed graphs. We also integrate variance of each row in the transition matrix corresponding to the target edge in each step by an MLP. Generally, directed and undirected versions reveal similar performance, and we report the performance of undirected EdgeRWSE if not specific.
For Full EdgeRWSE, we verify its theoretical expressive power through experiments on synthetic datasets; see Table 3. A GINE enhanced with Full EdgeRWSE achieves  $100\%$  performance on the synthetic datasets EXP and SR25. To fully distinguish EXP and SR25, the model needs to be more expressive than 1-WL and 2-FWL, respectively. This shows that Full-EdgeRWSE is (partly) more powerful than 2-FWL (3-WL), which is consistent with our theory. In comparison, 1-down EdgeRWSE that does not consider 2-simplices fails  $(0\%)$  to distinguish strongly regular graphs in SR25. However, we experimentally find that the performance of Full-EdgeRWSE on zinc dataset is similar to 1-down EdgeRWSE with a GINE base model. This is because 2-simplices (or triangles)
are actually rare in molecules, so Full-EdgeRWSE makes no significant difference than 1-down EdgeRWSE.
We also implement the interorder random walk among 0, 1, 2-simplices. The random walk probability matrix is the normalized  $\mathcal{A}_2$  in our main text. We compute  $\mathcal{A}_2, \ldots, \mathcal{A}_2^T (T = 20$  for Zinc) and embed the diagonal elements for both node and edge features. The performance of GINE augmented with Inter-RWSE is significantly better than pure GINE but is slightly weaker than simultaneously applying NodeRWSE and EdgeRWSE (which are separately computed without inter-order communications). We attribute it to: (i) lack of tuning hyper-parameters such as  $T$ , and (ii) the return probabilities are much less in inter-order random walk, as there are totally  $n + m$  possible targets, instead of  $n$  for O-random walk and  $m$  for 1-random walk. This makes it harder for the model to learn meaningful structure information and may require larger  $T$ .
CellularRWSE The concept of CellularRWSE is explained in Appendix C.3, where we treat the edges as 1-cells and extract all the rings (cycles) as 2-cells. In this CellularRWSE, a random walk is able to transit from one source edge to (i) another target edge sharing one node, or (ii) another target edge that is in the same 2-cell as the source edge. The performance is slightly better than EdgeRWSE (see Table 1), which may be due to the ability to distinguish higher-order structures (rings and cycles).
Hodge1Lap. We already include a detailed discussion in the main text and in Appendix C.2.2 on two different implementations: projection-based methods and sign-invariant methods. As we have shown, the projection method Hodge1Lapproj is both sign and basis invariant, and we consider the subspace spanned by eigenvectors of zero eigenvalues. The sign-invariant method Hodge1Lapabs simply takes absolute values of eigenvectors element-wise, and uses a MLP to encode eigenvectors and eigenvalues. We additionally apply a MLP to learn eigenvalue-eigenvector pairs of Hodge-1 Laplacians in our Hodge1Lap implementation. In practice, we adopt the combination of these two methods; if not specific, we report the results of the combined variant.
RWMP. We use the cosine similarities of the node features as our distance metric. The drop probability follows Equation (76). In each layer, we use a normal GINE layer without RWMP and a GINE layer with RWMP and add their features as the final representation of the layer. The GINE [25] model with Hodge1Lap and RWMP achieves highly competitive performance on Zinc, outperforming GPS [38] and Specformer [6].
Other implementation details. Some other implementation details and discussions are summarized below.
- For SSWL+, we add edge PE/SE features embedded by a unique linear projection to the original edge features in every layer.
- For GRIT, the PE is embedded by a linear layer and then directly added to the edge features. For virtual edges, the PE values are padded as zeros.
- Our method is can be applied to higher-order GNNs, including simplicial networks [8, 7] and 2- $\delta$ -FWL equivalent classes [36], while others (SPD, RD) cannot. Our method is also more sparse.
- Our method can be applied to any base model without increasing much computation cost and can be applied together with other levels of PE/SE.
- There are more than one way to integrate PE/SE into the models, while we choose the most simple ones: we treat PE/SE as initialization of edge features by concatenating or adding them to the original edge features; they can also be used as in graphormer (as attention biases), or any convolution/attention computation in simplicial networks.
E.5 Complexity and applicability analysis
Theoretically, the complexity to generate Hodge1Lap and EdgeRWSE are both  $O(m^2)$ , where  $m$  is the number of edges. For small or sparse graphs, they share similar complexity with node-level PE such as LapPE and RWSE. For example, the average generation times on Zinc computed by RTX3090 are: RWSE (23s), EdgeRWSE (32s), Hodge1Lap (28s). Even for extremely large graphs with millions of nodes, we can handle by sampling subgraphs as we are only interested in the local
structure information for some PE/SE. For example, for a  $K$ -step EdgeRWSE, we can sample a  $K$ -hop subgraph rooted at the target edge and continue to process, without having to deal with large matrix multiplication. It is noticeable that we only have to preprocess once for every dataset, so the precompute time is negligible compared to experiment time.
For the computation complexity in the forward pass of models, since all our PE/SE are embedded by light-weight MLPs, the extra computation cost is completely ignorable compared with the whole model. In summary, our methods are able to significantly enhance model performance with extremely small additional computation cost.
Regarding applicability, on the base model side, our Hodge1Lap and EdgeRWSE are universally applicable to any base model as long as edge features are considered. Our methods are orthogonal to other PE/SE and can be used together. SOTA methods can always be facilitated with our methods to achieve better performance. On the data type side, as we have emphasized, our methods are applicable to (both directed and undirected) graphs, simplicial complexes, and other discrete topological data structures, e.g. cellular complexes.