SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Rook and Shrikhande graph cannot be distinguished by 2-WL [9, 52]. However, OR curvature is sensitive to differences in their first-hop peripheral subgraphs [22], thus distinguishing them. This result shows the limitations of Forman-Ricci and Resistance curvature, as well as the benefits of using Ollivier-Ricci curvature. We show further experiments with curvature notions on strongly-regular graphs in the experimental section, observing improvements whenever we use them as filtrations.
Counting Substructures. Evaluating the ability of curvature to encode structural information is a crucial aspect for understanding its expressivity and validating its overall utility in graph learning. It has previously been shown that incorporating structural information of graphs can extend the expressive power of message-passing neural networks [11, 25, 42]. Additionally, Bouritsas et al. [9] show how GNNs can be (i) strictly more expressive than 1-WL by counting local isomorphic substructures (e.g. cliques), and (ii) exhibit predictive performance improvements when adding such substructure counts to the node features. For instance, some strongly-regular graphs can be distinguished by counting 4-cliques. Discrete curvature measures are informed by these local substructures and have been shown to improve expressivity beyond 1-WL [65] when included as a node feature. Nevertheless, there is limited work exploring what substructure information curvature carries, which would allow us to describe the expressivity of the measure. Moreover, persistent homology can track the number of cycles over the filtration of interest, allowing additional structural information to be encoded at multiple scales. Thus, we will explore the extent to which substructures can be counted for different curvatures (with and without a topological component) in a subsequent experiment, providing evidence on the expressive power of curvature filtrations. See Table 4 for our experimental results, and Appendix E for a more detailed discussion on the tendencies of each curvature notion when counting substructures.
3.3 Choosing a Curvature Notion in Practice
In Section 3.1, we showed that all three of our prototypical curvature notions exhibit advantageous stability properties, thus implying that they may all be used reliably within our method to measure
the distance between two sets of graphs. However, which curvature should be chosen in practice? In general, we find that the answer to this question lies at the intersection of expressivity and scalability, but depends ultimately on the nature of the experiment at hand. Nevertheless, we aim to provide an intuition for all three notions, along with a general recommendation. We hope this, in conjunction with the experimental and computational complexity results in Section 4 and Appendix G will help practitioners in making an optimal choice.
Comparison. Forman-Ricci is arguably the simplest and most local notion of curvature. Though limited in expressivity when compared to Ollivier-Ricci curvature, it is by far the most efficient to compute. Resistance curvature, by contrast, is the most global notion, making it sensitive to large substructures. However, computing the effective resistance metric on a graph requires inverting the Laplacian, making it far less efficient than Forman and Ollivier-Ricci, especially for large graphs. Finally, we have Ollivier-Ricci curvature, which we have found to be the most expressive based on its ability to (i) distinguish strongly-regular graphs, and (ii) count substructures. It is also the most versatile, given the option to adapt the underlying probability measure; this comes at the cost of lower computational performance in comparison to Forman—Ricci curvature, though.
Given its high expressivity, as well as its overall experimental and computational performance, we recommend using Ollivier-Ricci curvature whenever feasible.
We have proven the theoretical stability of discrete curvature notions under certain graph perturbations. We also illustrated their ability to distinguish distance-regular and strongly-regular graphs. Subsequently, we will discuss empirical experiments to evaluate these claims and to further test the utility of our methods.
4.1 Distinguishing Strongly-Regular Graphs
In addition to the theoretical arguments outlined, we explore the ability of our method to distinguish strongly-regular graphs in a subset of data sets, i.e. sr16622, sr261034, sr281264, and sr401224. These data sets are known to be challenging to classify since they cannot be described in terms of the 1-WL test [7]. Our main goal is not to obtain the best accuracy, but to show how the discriminative power of discrete curvature can be improved by using it in a ultra
Table 1: Success rate  $(\uparrow)$  of distinguishing pairs of strongly-regular graphs when using either raw curvature values or a curvature filtration.
Method	sr16622	sr261034	sr281264	sr401224
\( \kappa_{\text{FR}} \)	0.00	0.00	0.00	0.00
\( \kappa_{\text{OR}} \)	1.00	0.78	1.00	0.00
\( \kappa_{\text{R}} \)	0.00	0.00	0.00	0.00
Filtration (\( \kappa_{\text{FR}} \))	1.00	0.20	0.00	0.93
Filtration (\( \kappa_{\text{OR}} \))	1.00	0.89	1.00	0.93
Filtration (\( \kappa_{\text{R}} \))	1.00	0.20	0.00	0.93
tion context. Table 1 depicts the results of our classification experiment. We perform a pairwise analysis of all graphs in the data set, calculating distances based on histograms of discrete curvature measurements, or based on the bottleneck distance between persistence diagrams ('Filtrations'). Subsequently, we count all non-zero distances  $(>1\times 10^{-8}$  to correct for precision errors). Our main observation is that combining TDA with curvature is always better than or equal to curvature without TDA. Similar to our theoretical predictions, both resistance curvature and Forman curvature fail to distinguish any of the graphs without TDA. We therefore show the benefits from an expressivity point of view for using discrete curvature as a filtration. Notably, we achieve the best results with OR curvature, which is particularly flexible since it permits changing the underlying probability measure. Using a probability measure based on random walks (see Appendix F) takes into account higher-order neighbourhoods and improves discriminative power (on sr261034, the pairwise success rate drops to  $0.0 / 0.2$  with raw/TDA values, respectively, if the uniform probability measure is used).
4.2 Expressivity experiments with the BREC dataset
We evaluate discrete curvatures and their filtrations on the BREC data set, which was recently introduced to evaluate GNN expressiveness [66]. The data set consists of different categories of graph pairs (Basic, Regular, and Extension), which are distinguishable by 3-WL but not by 1-WL, as well as Strongly-Regular (STR) and CFI graph pairs, which are indistinguishable using 3-WL. We explore the ability of curvature filtrations to distinguish these graph pairs and compare them
Table 2: Success rate (↑) of distinguishing pairs of graphs in the BREC dataset when using different discrete curvatures and their filtrations.
Method	Basic (56)	Regular (50)	STR (50)	Extension (97)	CFI (97)
1-WL	0.00	0.00	0.00	0.00	0.00
3-WL	1.00	1.00	0.00	1.00	0.59
S3	0.86	0.96	0.00	0.05	0.00
S4	1.00	0.98	1.00	0.84	0.00
κOR	1.00	0.96	0.06	0.93	0.00
κFR	0.96	0.92	0.00	0.52	0.00
κR	1.00	1.00	0.00	1.00	0.04
Filtration (κOR)	1.00	1.00	0.08	0.95	0.00
Filtration (κFR)	0.98	0.96	0.04	0.61	0.00
Filtration (κR)	1.00	1.00	0.04	1.00	0.04
to substructure counting,  $S_{3}$  and  $S_{4}$ , which involves enumerating all 3-node/4-node substructures around nodes in combination with the WL algorithm. These approaches, unlike discrete curvature, have limited practical applications due to their high computational complexity. Similar to our experiments on strongly-regular graphs, we calculate Wasserstein distances based on histograms of OR curvature measurements between the pairs of graphs. Subsequently, we count all non-zero distances ( $>1 \times 10^{-8}$  to correct for precision errors). Our main observations from Table 2 are that curvature can distinguish graphs which are 3-WL indistinguishable. Additionally, we observe improvements in success rate using the filtration on the Basic, Regular, STR and Extension graph pairs. Moreover, our curvature-based approach performs competitively and sometimes even better than  $S_{4}$ , which has been shown to be extremely effective in graph learning tasks [9]. Despite its empirical prowess,  $S_{4}$  is computationally expensive, making it an infeasible measure in many applications. Discrete curvature and its filtrations, by contrast, scale significantly better.
4.3 Behaviour with Respect to Perturbations
To explore the behaviour of curvature descriptors under perturbations, we analyse the correlation of our metric when adding and removing edges in the 'Community Graph' data set: we increase the fraction of edges added or removed from 0.0 to 0.95, measuring the distance between the perturbed graphs and the original graphs for each perturbation level. Following O'Bray et al. [54], we require a suitable distance measure to be highly correlated with increasing amounts of perturbation. We compare to current approaches that use descriptor functions with MMD. As Table 3 shows, a curvature filtration yields a higher correlation than curvature in
Table 3: Pearson correlation (↑) of measures when adding/removing edges.
Measure	Adding Edges	Removing Edges
Laplacian	0.457 ± 0.013	0.420 ± 0.000
Clust. Coeff.	0.480 ± 0.012	0.504 ± 0.020
Degrees	0.761 ± 0.003	0.995 ± 0.000
κFR	0.420 ± 0.000	0.432 ± 0.003
κOR	0.903 ± 0.005	0.910 ± 0.002
κR	0.420 ± 0.004	0.441 ± 0.005
Filtration (κFR)	0.571 ± 0.006	0.996 ± 0.006
Filtration (κOR)	0.997 ± 0.000	0.970 ± 0.005
Filtration (κR)	0.730 ± 0.005	0.954 ± 0.008
combination with MMD, showing the benefits of employing TDA from a stability perspective. Additionally, curvature filtrations improve upon the normalized Laplacian and clustering coefficient (again, we used MMD for the comparison of these distributions). OR curvature exhibits particularly strong results when adding/removing edges, even surpassing the local degrees of a graph (which, while being well-aligned with perturbations of the graph structure, suffer in terms of overall expressivity).
4.4 Counting Substructures
The ability of a descriptor to count local substructures is important for evaluating its expressive power [3]. We follow Chen et al. [12], who assess the ability of GNNs to count substructures such as triangles, chordal cycles, stars and tailed triangles. This is achieved by generating regular graphs with random edges removed, and counting the number of occurrences of each substructure in a given graph. To assess the power of curvature to capture such local structural features, we use the same experimental setup and pass the edge-based curvatures through a simple 1-layer MLP to output the substructure count. Additionally, we evaluate the effect of using curvature as a filtration.
(a) Curvature distinguishes graphon data sets
(b) Curvature distinguishes bioinformatics data sets
Figure 2: (a) Adjusted Rand Index  $(\uparrow)$  for clustering sets of four graphons. We compare our curvature filtrations  $(\mathbb{E})$  to kernel-based methods. (b) Permutation testing values  $(\downarrow)$  for distinguishing different bioinformatics data sets. Position  $(i,j)$  in each matrix denotes a permutation test between data set  $i$  and data set  $j$ .
Table 4: MAE  $(\downarrow)$  for counting substructures based on raw curvature values and curvature filtrations. The 'Trivial Predictor' always outputs the mean training target.
Method	Triangle	Tailed Tri.	Star	4-Cycle
Trivial Predictor	0.88	0.90	0.81	0.93
GCN	0.42	0.32	0.18	0.28
κFR	0.54	0.56	0.72	0.53
κOR	0.33	0.31	0.40	0.31
κR	0.59	0.50	0.72	0.47
Filtration (κFR)	0.45	0.52	0.49	0.60
Filtration (κOR)	0.23	0.24	0.34	0.31
Filtration (κR)	0.47	0.48	0.36	0.42
Table 4 shows our experimental results, reported on the pre-defined test split defined by Chen et al. [12]. In comparison to Graph Convolutional Networks [40], we find that OR curvature exhibits improved performance in counting small local structures such as triangles and tailed triangles; this is a surprising finding given the smaller computational footprint of this curvature formulation (in comparison to a GCN). Moreover, we find that the OR curvature performs better than both the Forman curvature and the resistance curvature for the different substructures in the data. We also observe that combining curvature with TDA almost always improves upon
using the curvature alone, the only exception being Forman curvature for counting 4-cycles. We leave a more detailed investigation of these phenomena for future work.
4.5 Synthetic Graph Generative Model Evaluation
To have a ground truth for a graph distribution, we tested our metric's ability to distinguish graphons. Following the approach suggested by Sabanayagam et al. [60], we generate four graphons,  $W_{1}(u,v) = uv$ ,  $W_{2}(u,v) = \exp\{-\max(u,v)^{0.75}\}$ ,  $W_{3}(u,v) = \exp\{-0.5*(\min(u,v) + u^{0.5} + v^{0.5})\}$  and  $W_{4}(u,v) = \|u - v\|$ . Sampling from these graphons produces dense graphs, and we control their size to be between 9 and 37 nodes, thus ensuring that we match the sizes of molecular graphs in the ZINC data set [32], an important application for generative models.
We perform experiments by considering all combinations of three and four graphons. We generate distances between graphs with our method as well as other kernel-based approaches, and use spectral clustering to separate the distributions. We measure the performance of the algorithms using the Adjusted Rand Index (ARI) of the predicted clusters, comparing to three state of the art, kernel-based approaches: (i) Wasserstein Weisfeiler-Le(h)man graph kernels [63], (ii) graph neural tangent kernels [20], and (iii) Tree Mover's Distance [13]. From Figure 2a, we find that a filtration based on OR curvature is better able to distinguish and cluster graphons than the previously-described approaches based on kernels and it performs best for all sets of graphons. We also observe that OR curvature performs better than other discrete curvatures, with resistance curvature achieving higher ARI than Forman curvature. Notice that unlike these kernel approaches, our method can be easily extended to provide a proper metric between distributions of graphs.
4.6 Real-World Graph Generative Model Evaluation
By converting the generated persistence diagrams into a persistence landscape, we can generate an average topological descriptor for all graphs in a distribution [10]. This allows us to calculate norms between graph distributions, making it possible to perform two-sample and permutation testing, unlike a majority of kernel-based approaches, which provide a distance between all individual graphs, or would require MMD to assess mean similarities. We randomly sample ten graphs from four different bioinformatics data sets, i.e. KKI, PROTEINS, Peking, and ENZYMES [51]. We measure the distance between two data sets using the  $L^P$  norm between their average persistence landscapes. We then permute graphs from both samples, randomly selecting sets of equal size, measure their respective distances, and finally aggregate the fraction of distances which are higher than the original. A low fraction indicates that distances are lower for permutations than between the original sets of graphs, suggesting that the metric can distinguish between the two distributions. We compare our approach to previous methods that combine graph statistics with MMD. Using a significance threshold of  $p < 0.05$ , we see in Figure 2b that both Forman and the OR curvature are able to distinguish all but one pair of data sets, an improvement over all the other approaches. In general, we find that fractions are lower using curvature filtrations than graph statistic based approaches, demonstrating the utility of our approach.
We have described the first thorough analysis of both stability and expressivity of discrete curvature notions and their filtration formulations on graphs. We believe this to be important for the community in multiple contexts, ranging from improving expressivity of GNNs to understanding the robustness of curvature on graphs. Using curvature filtrations and their topological descriptors (here: persistence landscapes), we develop a new metric to measure distances between graph distributions. Our metric can be used for evaluating graph generative models, is robust and expressive, and improves upon current approaches based on graph descriptor and evaluator functions. We have also demonstrated clear advantages over state-of-the-art methods that combine graph statistics with MMD, providing instead a metric with (i) well-understood parameter and function choices, (ii) stability guarantees, (iii) added expressivity, and (iv) improved computational performance. Most notably, we scale significantly better for large populations of molecular-sized graphs (see Appendix G for more details), which we consider crucial for current graph generative model applications. We hope that our pipeline will provide a principled, interpretable, and scalable method for practitioners to use when evaluating the performance of graph generative models.
Future work could explore other representations [1, 58], focus on different filtration constructions [15], new curvature measures, or further extend our stability and expressivity results (for instance to the setting of weighted graphs). Given the beneficial performance and flexibility of Ollivier-Ricci curvature in our experiments, we believe that changing—or learning—the probability measure used in its calculation could lead to further improvements in terms of expressivity, for example. Another relevant direction involves incorporating node and edge features into the distance measure and applying the model to specific use cases such as evaluating molecule generation. We envision that this could be done using bi-filtrations or by considering node features for the curvature calculations.
The authors want to thank the anonymous reviewers for their comments, which helped us improve the paper. We are also grateful for the area chair who believed in this work. The authors are grateful to Fabrizio Frasca and Leslie O'Bray for valuable feedback on early versions of the manuscript. J.S. is supported by the UKRI CDT in AI for Healthcare http://ai4health.io (Grant No. P/S023283/1). M.B. is supported in part by the ERC Consolidator grant No. 724228 (LEMAN) and the EPSRC Turing AI World-Leading Researcher Fellowship. B.R. is supported by the Bavarian State government with funds from the Hightech Agenda Bavaria. He wishes to dedicate this paper to his son Andrin.
[64] Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=7UmjRGzp-A.
[65] Ameya Velingker, Ali Kemal Sinop, Ira Ktena, Petar Velicković, and Sreenivas Gollapudi. Affinity-aware graph networks, 2022. URL https://arxiv.org/abs/2206.11941.
[66] Yanbo Wang and Muhan Zhang. Towards better evaluation of gnn expressiveness with BREC dataset, 2023. URL https://arxiv.org/abs/2304.07702.
[67] Duncan J. Watts and Steven H. Strogatz. Collective dynamics of 'small-world' networks. Nature, 393(6684):440-442, 1998. doi: 10.1038/30918.
[68] JunJie Wee and Kelin Xia. Forman persistent Ricci curvature (FPRC)-based machine learning models for protein-ligand binding affinity prediction. Briefings in Bioinformatics, 22(6): bbab136, 2021. doi: 10.1093/bib/bbab136.
[69] JunJie Wee and Kelin Xia. Ollivier persistent Ricci curvature-based machine learning for the protein-ligand binding affinity prediction. Journal of Chemical Information and Modeling, 61 (4):1617-1626, 2021. doi: 10.1021/acs.jcim.0c01415. URL https://doi.org/10.1021/acs.jcim.0c01415.
[70] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=ryGs6iA5Km.
[71] Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. GraphRNN: Generating realistic graphs with deep auto-regressive models. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages 5708-5717. PMLR, 2018.
A Pseudocode
Here we give pseudocode for various parts of the method, highlighting the most relevant aspects of using curvature filtrations to evaluate graph generative models. First, we outline a crucial part of our evaluation framework in Algorithm 1: how we compute summary topological descriptors for sets of graphs. This algorithm, taken from Bubenik [10], assumes a list of precomputed persistence landscapes, one for each graph. Algorithm 2, on the other hand, outlines our procedure for generating a distance between two sets of graphs using their summary topological descriptors.
Algorithm 1 Compute Average of Persistence Landscapes
Require:  $\Lambda$  is a list of persistence landscapes  
Ensure:  $\bar{\Lambda}$  is the average persistence landscape  
function COMPUTE AVERAGE LANDSCAPE( $\Lambda$ )  
 $n \gets$  length of  $\Lambda$ $D \gets$  maximum homology degree occurring in  $\Lambda$ $\bar{\Lambda} \gets$  empty persistence landscape with  $D$  homology dimensions.  
 $\lambda^i(k, t) \gets$  the piecewise-linear function encoding hom-deg  $k$  contained in  $\lambda^i \in \Lambda$ .  
 $R \gets$  maximal domain for each function contained in  $\Lambda$ .  
 $k \gets 0$   
while  $k \leq D$  do  
for  $t \in R$  do  
 $\bar{\Lambda}(k, t) \gets \frac{1}{n} \sum_{i}^{n} \lambda^i(k, t)$   
end for  
 $k \gets k + 1$   
end while  
return  $\bar{\Lambda}$   
end function
Recall that persistence landscapes are a collection of piecewise linear functions that encode the homological information tracked by a specified filtration. Thus, to compute the average landscape  $\bar{\Lambda}$  at each dimension, we simply sum the piecewise linear functions, and divide by the total number of landscapes in consideration. To understand our method for comparing sets of graphs, and thus evaluating graph generative models, based on average persistence landscapes see Algorithm 2.
B Additional Proofs on Stability and Expressivity
B.1 Stability Proofs
Proof. Considering the calculation of persistence diagrams based on scalar-valued filtrations functions, every point in the persistence diagram  $D_{f}$  can be written as a tuple of the form  $(f(e_{F}), f(e_{F}^{\prime}))$ , with  $e_{F}, e_{F}^{\prime} \in E_{F}$ ; the sample applies for  $D_{g}$ . The inner distance between such tuples that occur in the bottleneck distance calculation can thus be written as
$$
\left\| \left(f \left(e _ {F}\right), f \left(e _ {F} ^ {\prime}\right)\right) - \left(g \left(e _ {G}\right), g \left(e _ {G} ^ {\prime}\right)\right) \right\| _ {\infty}. \tag {6}
$$
The maximum distance that can be achieved using this expression is determined by the maximum variation of the functions, expressed via  $\mathrm{dis}(f,g)$  and  $\mathrm{dis}(g,f)$ , respectively.
Edge Filtrations versus Vertex Filtrations Our results are structured to address filtrations built by a function on the edges of a graph  $G = (V, E)$ ,  $f \colon E \to \mathbb{R}$ . This matches our notions of discrete curvature, which are also defined edge-wise.  $f$  gives an explicit ordering on  $E$  and thus an induced ordering on  $V$  given by:
$$
v \leq v ^ {\prime} \iff \sum_ {e \in E _ {v}} f (e) \leq \sum_ {e ^ {\prime} \in E _ {v ^ {\prime}}} f \left(e ^ {\prime}\right)
$$