SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joshua Southern*
jks17@ic.ac.uk
Jeremy Wayland*
Helmholtz Munich & Technical University of Munich
jeremy.wayland@tum.de
Michael Bronstein†
michael.bronstein@cs.ox.ac.uk
Bastian Rieck†
bastian.rieck@tum.de
Graph generative model evaluation necessitates understanding differences between graphs on the distributional level. This entails being able to harness salient attributes of graphs in an efficient manner. Curvature constitutes one such property that has recently proved its utility in characterising graphs. Its expressive properties, stability, and practical utility in model evaluation remain largely unexplored, however. We combine graph curvature descriptors with emerging methods from topological data analysis to obtain robust, expressive descriptors for evaluating graph generative models.
Graph-structured data are prevalent in a number of different domains, including social networks [48], bioinformatics [38, 59], and transportation [33]. The ability to generate new graphs from a distribution is a burgeoning technology with promising applications in molecule or protein design [34, 35], as well as circuit design [47]. To compare graph generative models and advance research in this area, it is essential to have a metric that can measure the distance between a set of generated and reference graphs, with enough expressivity to critically evaluate model performance. Typically, this is done by using a set of descriptor functions, which map a graph to a high-dimensional representation in  $\mathbb{R}^d$ . An evaluator function, such as the maximum mean discrepancy [27, MMD], may then be used to get a distance between two distributions of graphs by comparing their vectorial representations [44, 53]. This state of the art was recently critiqued by O'Bray et al. [54] since it (i) requires numerous parameter and function choices, (ii) is limited by the expressivity of the descriptor function, and (iii) does not come equipped with stability guarantees.
We propose to overcome these issues through topological data analysis (TDA), which is capable of capturing multi-scale features of graphs while being more expressive than simple descriptor functions. TDA is built on the existence of a function of the form  $f\colon V\to \mathbb{R}$ , or  $f\colon E\to \mathbb{R}$  on a graph  $G = (V,E)$ . This is used to obtain a filtration, i.e. an ordering of subgraphs, resulting in a set of topological descriptors, the persistence diagrams. Motivated by their expressive power and computational efficiency, we use recent notions of discrete curvature [19, 24, 55] to define filtrations and calculate persistence landscapes [10] from the persistence diagrams, thus obtaining a descriptor whose Banach space formulation permits statistical calculations. Our proposed method comes equipped with stability guarantees, can count certain substructures and measure important graph characteristics, and can be used to evaluate a variety of statistical tests since it permits computing distances between graph distributions.
These authors jointly directed the work.
- We provide a thorough theoretical analysis of the stability and expressivity of recent notions of discrete curvature, showing their fundamental utility for graph learning applications.
- Using discrete curvature and TDA, we develop a new metric for graph generative model evaluation.
- Our experiments reveal our metric is robust and expressive, thus improving upon current approaches that use simple graph descriptor and evaluator functions.
The topological descriptors used in this paper are based on computational topology and discrete curvature. We give an overview of these areas and briefly comment on previous work that makes use of graph statistics in combination with MMD. In the following, we consider undirected graphs, denoted by  $G = (V,E)$ , with a set of vertices  $V$  and a set of edges  $E \subseteq V \times V$ .
2.1 MMD and Metrics Based On Graph Statistics
MMD is a powerful method for comparing different distributions of structured objects. It employs kernel functions, i.e. positive definite similarity functions, and can thus be directly combined with standard graph kernels [8] for graph distribution comparison. However, there are subtle issues when calculating kernels in  $\mathbb{R}^d$ : Gaussian kernels on geodesic metric spaces, for instance, have limited applicability when spaces have non-zero curvature [23], and certain kernels in the literature are indefinite, thus violating one of the tenets of MMD [54]. Despite these shortcomings, MMD is commonly used to evaluate graph generative models [17, 71]. This is accomplished by extracting a feature vector from each graph, such as the clustering coefficient or node degree, and subsequently calculating empirical MMD values between generated and reference samples. Some works [49] combine multiple structural properties of graphs into a single metric through the Kolmogorov-Smirnov (KS) multidimensional distance [39]. Combining graph statistics into a single measure has also led to metrics between molecular graphs, such as the quantitative estimate of drug-likeness (QED), which is a common measure in drug discovery [5]. However, these simple statistics, even when considered jointly, often lack expressivity, have no stability guarantees, and their use with MMD requires numerous parameter and function choices [54].
2.2 Computational Topology
Computational topology assigns invariants—characteristic properties that remain unchanged under certain transformations—to topological spaces. For graphs, the simplest invariants are given by the 0-dimensional  $(\beta_0)$  and 1-dimensional  $(\beta_{1})$  Betti numbers. These correspond to the number of connected components and number of cycles, respectively, and can be computed efficiently. Their limited expressivity can be substantially increased when paired with a scalar-valued filtration function  $f\colon E\to \mathbb{R}$ . Since  $f$  can only attain a finite number of values  $a_0,a_1,a_2,\ldots$  on the graph, this permits calculating a graph filtration  $\emptyset \subseteq G_0\subseteq G_1\dots \subseteq G_{k - 1}\subseteq G_k = G$ , where each  $G_{i}\coloneqq (V_{i},E_{i})$  with  $E_{i}\coloneqq \{e\in E\mid f(e)\leq a_{i}\}$  and  $V_{i}\coloneqq \{v\in V\mid \exists e\in E_{i}$  s.t.  $v\in e\}$ . This sublevel set filtration permits tracking topological features, such as cycles, via persistent homology [21]. If a topological feature appears for the first time in  $G_{i}$  and disappears in  $G_{j}$ , we represent the feature as a tuple  $(a_i,a_j)$ , which we can collect in a persistence diagram  $D$ . Persistent homology thus tracks changes in connected components and cycles over the complete filtration, measured using a filtration function  $f$ . Persistence diagrams form a metric space, with the distance between them given by the bottleneck distance, defined as  $d_B(D,D')\coloneqq (\inf_{\eta :D\to D'}\sup_{x\in D}\| x - \eta (x)\|_{\infty})$ , where  $\eta$  ranges over all bijections between the two diagrams. A seminal stability theorem [14] states that the bottleneck distance between persistence diagrams  $D_{f},D_{g}$ , generated from two functions  $f$  and  $g$  on the same graph, is upper-bounded by  $d_B(D_f,D_g)\leq ||f - g||_\infty$ . The infinity norm of the functions, a geometrical quantity, hence limits the topological variation. In practice, we convert persistence diagrams to an equivalent representation, the persistence landscape [10], which is more amenable to statistical analyses and the integration into machine learning pipelines.
One can also consider functions over vertices,  $f \colon V \to \mathbb{R}$ , when building filtrations. See Appendix B.1 for a remark on the equivalence of these viewpoints.
4Swapping  $\stackrel {\cdot}{\leq}$  for  $\geq$  and max for min results in the superlevel set filtration of equal expressivity.
Advantageous Properties. Persistent homology satisfies expressivity and stability properties. The choice of filtration  $f$  affects expressivity: with the right filtration, persistent homology can be more expressive than the 1-WL test [31, 57], which is commonly used to bound the expressivity of graph neural networks [50, 52, 70]. Thus, given a suitable filtration, we can create a robust and expressive metric for comparing graphs. Moreover, as we will later see, TDA improves the expressivity of any  $f$ , meaning that even if the function on its own is not capable of distinguishing between different graphs, using it in a filtration context can overcome these deficiencies.
2.3 Discrete Curvature
While filtrations can be learnt [29, 31], in the absence of a well-defined learning task for graph generative model evaluation, we opt instead to employ existing functions that exhibit suitable expressivity properties. Of particular interest are functions based on discrete curvature, which was shown to be an expressive feature for graph and molecular learning tasks [65, 68, 69]. Curvature is a fundamental concept in differential geometry and topology, making it possible to distinguish between different types of manifolds. There are a variety of different curvature formulations with varying properties, with Ricci curvature being one of the most prominent. Roughly speaking, Ricci curvature is based on measuring the differences in the growth of volumes in a space as compared to a model Euclidean space. While originally requiring a smooth setting, recent work started exploring how to formulate a theory of Ricci curvature in the discrete setting [16, 19, 24, 45, 55, 62]. Intuitively, discrete curvature measures quantify a notion of similarity between node neighbourhoods, the discrete concept corresponding to 'volume.' They tend to be larger for structures where there are overlapping neighbourhoods such as cliques, smaller (or zero) for grids and lowest (or negative) for tree-like structures. Ricci curvature for graphs provides us with sophisticated tools to analyse the neighbourhood of an edge and recent works have shown the benefits of using some of these curvature-based methods in combination with Graph Neural Networks (GNNs) to boost performance [65], assess differences between real-world networks [61], or enable graph rewiring to reduce over-squashing in GNNs [64]. However, the representational power and stability properties of these measures remain largely unexplored. Subsequently, we will focus on three different types of curvature, (i) Forman-Ricci curvature [24], (ii) Ollivier-Ricci curvature [55], and (iii) Resistance curvature [19]. We find these three notions to be prototypical of discrete curvature measures, increasing in complexity and in their ability to capture global properties of a graph.
Forman-Ricci Curvature. The Forman(-Ricci) curvature for an edge  $(i,j)\in E$  is defined as
$$
\kappa_ {\mathrm {F R}} (i, j) := 4 - d _ {i} - d _ {j} + 3 | \# _ {\Delta} |, \tag {1}
$$
where  $d_{i}$  is the degree of node  $i$  and  $|\#_{\Delta}|$  is the number of 3-cycles (i.e. triangles) containing the adjacent nodes.
Ollivier-Ricci Curvature. Ollivier introduced a notion of curvature for metric spaces that measures the Wasserstein distance between Markov chains, i.e. random walks, defined on two nodes [55]. Let  $G$  be a graph with some metric  $d_G$ , and  $\mu_v$  be a probability measure on  $G$  for node  $v \in V$ . The Ollivier-Ricci curvature of any pair  $i, j \in V \times V$  with  $i \neq j$  is then defined as
$$
\kappa_ {\mathrm {O R}} (i, j) := 1 - \frac {1}{d _ {G} (i , j)} W _ {1} \left(\mu_ {i}, \mu_ {j}\right), \tag {2}
$$
where  $W_{1}$  refers to the first Wasserstein distance between  $\mu_{i}, \mu_{j}$ . Eq. (2) defines the Ollivier-Ricci (OR) curvature in a general setting outlined by Hoorn et al. [30]; this is in contrast to the majority of previous works in the graph setting which specify  $d_{G}$  to be the shortest-path distance and  $\mu_{i}, \mu_{j}$  to be uniform probability measures in the 1-hop neighbourhood of the node. Extending the probability measures so that they act on larger locality scales is known to be beneficial for characterising graphs [4, 26, 36]. We assume this general setting to define different notions of OR curvature on the graph, permitting us the flexibility of altering the probability measure and the metric.
Resistance Curvature. The resistance curvature for edges of a graph, as established in Devriendt and Lambiotte [19], is inspired by Ohm's Law and the concept of effective resistance, a well-studied, global metric between nodes in a weighted graph that quantifies the resistance exerted by the network
In contrast to other notions of curvature, Ollivier-Ricci curvature is defined for both edges and non-edges.
Figure 1: An overview of our pipeline for evaluating graph generative models using discrete curvature. We depict a graph's edges being coloured by  $\kappa_{\mathrm{FR}}$ , as described in Eq. (1). The ordering on edges gives rise to a curvature filtration, followed by a corresponding persistence diagram and landscape. For graph generative models, we select a curvature, apply this framework element-wise, and evaluate the similarity of the generated and reference distributions by comparing their average landscapes.
when current flows between nodes. For a graph  $G = (V, E)$ , let  $R_{ij}$  be the resistance distance between nodes  $i, j \in V$ , defined in Eq. (9). The node resistance curvature of a node  $i \in V$  is then defined as  $p_i := 1 - \frac{1}{2} \sum_{j \sim i} R_{ji}$ . The curvature of an edge  $(i, j) \in E$ , what we refer to as resistance curvature, is then defined as
$$
\kappa_ {\mathrm {R}} (i, j) := \frac {2 \left(p _ {i} + p _ {j}\right)}{R _ {i j}} \tag {3}
$$
The resistance curvature of an edge is related to the average distance between the neighbourhoods of the nodes connected by the edge.
3 Our Method
Each notion of discrete curvature yields a scalar-valued function on the edges of a graph. We use these functions to define a family of curvature filtrations based on sublevel sets. In combination with persistent homology, this enables us to assess the structural properties of a given graph at multiple scales, measured via curvature. Using metrics on aggregated topological signatures—here, in the form of persistence landscapes—we may then compare two distributions of graphs. Specifically, we propose the following scheme for graph generative model evaluation:
1. Given a specific curvature filtration, we generate a set of persistence diagrams that encode the persistent homology in dimensions 0 and 1 for each graph in the distribution. Each diagram tracks the lifespan of connected components and cycles as they appear in the filtration of a given graph, resulting in a multi-scale summary of the graph's structure.
2. To permit an analysis on the distributional level, we convert each diagram into a more suitable representation, namely a persistence landscape. As functional summaries of topological information, persistence landscapes allow for easy calculation of averages and distances [10].
3. Finally, we conduct statistical comparisons between graph distributions, e.g. permutation tests using  $p$ -norms, in this latent space, providing an expressive metric for evaluating generative models.
Figure 1 illustrates our proposed pipeline using Forman-Ricci curvature. For choosing a notion of curvature in practice, implementation details, and computational performance we refer the reader to Section 3.3, Appendix A, and Appendix G respectively. We also make our framework publicly available.<sup>6</sup>
3.1 Stability
We first discuss the general stability of topological calculations, proving that changes in topological dissimilarity are bounded by changes in the filtration functions. Moreover, we show that filtrations based on discrete curvature satisfy stability properties if the underlying graph is modified.
Topological Features. Using the persistent homology stability theorem [14], we know that if two curvature filtrations are similar on a graph, their persistence diagrams will also be similar. However, the theorem only holds for two different functions on the same graph, whereas the distributional case has not yet been addressed by the literature. Our theorem provides a (coarse) upper bound.
Source code is available at https://github.com/aidos-lab/CFGGME.
Theorem 1. Given graphs  $F = (V_{F}, E_{F})$  and  $G = (V_{G}, E_{G})$  with filtration functions  $f, g$ , and corresponding persistence diagrams  $D_{f}, D_{g}$ , we have  $d_{B}(D_{f}, D_{g}) \leq \max \{ \mathrm{dis}(f, g), \mathrm{dis}(g, f) \}$ , where  $\mathrm{dis}(f, g) := |\max_{x \in E_{F}} f(x) - \min_{y \in E_{G}} g(y)|$  and vice versa for  $\mathrm{dis}(g, f)$ .
This upper bound implies that changes on the level of persistence diagrams are bounded by changes between the input functions. The stability of our method thus hinges on the stability of the filtration functions, so we need to understand the behaviour of curvature under perturbations. Following previous work [54], we aim to understand and quantify the stability of our method in response to adding and deleting edges in the graph. Our stability theorems establish bounds on various discrete curvature measures for finite, unweighted, connected graphs in response to these perturbations. We restrict the outcome of a perturbation to graphs of the form  $G' = (V, E')$  that satisfy  $|E| \neq |E'|$  and do not change the number of connected components of a graph. Our theoretical results bound the new curvature  $\kappa'$  according to the structural properties of  $G$ . For an exhaustive list of theorems and proofs, as well as experiments analysing the change in curvature for perturbed Erdős–Rényi (ER) graphs, see Appendix B and Appendix D.
Forman-Ricci Curvature. We first analyse Forman-Ricci curvature  $\kappa_{\mathrm{FR}}$ , and prove that it is stable with respect to adding and deleting edges.
Theorem 2. If  $G'$  is the graph generated by edge addition, then the updated Forman curvature  $\kappa_{FR}'$  for pre-existing edges  $(i,j) \in E$  can be bounded by  $\kappa_{FR}(i,j) - 1 \leq \kappa_{FR}'(i,j) \leq \kappa_{FR}(i,j) + 2$ . If  $G'$  is the graph generated by edge deletion, then the updated Forman curvature  $\kappa_{FR}'$  for pre-existing edges  $(i,j) \in E$  can be bounded by  $\kappa_{FR}(i,j) - 2 \leq \kappa_{FR}'(i,j) \leq \kappa_{FR}(i,j) + 1$ .
Ollivier-Ricci Curvature. Let  $\mathcal{G} = (G, d_G, \mu)$  be a triple for specifying Ollivier-Ricci curvature calculations, with  $G$  denoting an unweighted, connected graph,  $d_G$  its associated graph metric, and  $\mu := \{\mu_v \mid v \in V\}$  a collection of probability measures at each node. Furthermore, let  $\delta_i$  denote the Dirac measure at node  $i$  and  $J(i) := W_1(\delta_i, \mu_i)$  the corresponding jump probability in the graph  $G$  as defined by Ollivier [55]. Following an edge addition or deletion, we consider an updated triple  $\mathcal{G}' = (G', d_{G'}, \mu')$ , and remark that this yields an updated Wasserstein distance  $W_1'$ , calculated in terms of the new graph metric  $d_{G'}$ .
Theorem 3. Given a perturbation (either edge addition or edge deletion) producing  $\mathcal{G}'$ , the Ollivier-Ricci curvature  $\kappa_{OR}'(i,j)$  of a pair  $(i,j)$  can be bounded via
$$
1 - \frac {1}{d _ {G ^ {\prime}} (i , j)} \left[ 2 W _ {\max } ^ {\prime} + W _ {1} ^ {\prime} \left(\mu_ {i}, \mu_ {j}\right) \right] \leq \kappa_ {O R} ^ {\prime} (i, j) \leq \frac {J ^ {\prime} (i) + J ^ {\prime} (j)}{d _ {G ^ {\prime}} (i , j)}, \tag {4}
$$
where  $J'(v) \coloneqq W_1'(\delta_v, \mu_v')$  refers to the new jump probabilities and  $W_{\mathrm{max}}' \coloneqq \max_{x \in V} W_1'(\mu_x, \mu_x')$  denotes the maximal reaction to the perturbation (measured using the updated Wasserstein distance).
Resistance Curvature. Let  $G$  be an unweighted, connected graph with a resistance distance  $R_{ij}$  and resistance curvature  $\kappa_{\mathbb{R}}(i,j)$  for each  $(i,j) \in E$ . Furthermore, let  $d_x$  denote the degree for node  $x \in V$ . We find that  $\kappa_{\mathbb{R}}$  is well-behaved under these perturbations, in the sense that edge additions can only increase the curvature, and edge deletions can only decrease it. For edge additions, we obtain the following bound (see Appendix B.1.3 for the corresponding bound for edge deletions).
Theorem 4. If  $G'$  is the graph generated by edge addition, then  $\kappa_R' \geq \kappa_R$ , with the following bound:
$$
\left| \kappa_ {R} ^ {\prime} (i, j) - \kappa_ {R} (i, j) \right| \leq \frac {\Delta_ {\mathrm {a d d}} \left(d _ {i} + d _ {j}\right)}{R _ {i j} - \Delta_ {\mathrm {a d d}}}, \tag {5}
$$
where  $\Delta_{\mathrm{add}}\coloneqq \max_{i,j\in V}\left(R_{ij} - \frac{1}{2}\bigl (\frac{1}{d_i + 1} +\frac{1}{d_j + 1}\bigr)\right)$
The implications of this section are that (i) the stability of our topological calculations largely hinges on the stability of the functions being used to define said filtrations, and (ii) all discrete curvature measures satisfy stability properties with respect to changes in graph connectivity, making curvature-based filtrations highly robust.
3.2 Expressivity
A metric between distributions should be non-zero when the distributions differ. For this to occur, our metric needs to be able to distinguish non-isomorphic graphs and be sufficiently expressive. Horn
et al. [31] showed that persistent homology with an appropriate choice of filtration is strictly more expressive than 1-WL, the 1-dimensional Weisfeiler-Le(h)man test for graph isomorphism. A similar expressivity result can be obtained for using resistance curvature as a node feature [65], underlining the general utility of curvature. We have the following general results concerning the expressivity or discriminative power of our topological representations.
Theorem 5. Given two graphs  $F = (V_F, E_F)$  and  $G = (V_G, E_G)$  with scalar-valued filtration functions  $f, g$ , and their respective persistence diagrams  $D_f, D_g$ , we have  $d_B(D_f, D_g) \geq \inf_{\eta: E_F \to E_G} \sup_{x \in E_F} |f(x) - g(\eta(x))|$ , where  $\eta$  ranges over all maps from  $E_F$  to  $E_G$ .
Theorem 5 implies that topological distances are generally more discriminative than the distances between the filtration functions. Thus, calculating topological representations of graphs based on a class of functions improves discriminative power. To further understand the expressive power of curvature filtrations, we analyse strongly-regular graphs, which are often used for studying GNN expressivity as they cannot be distinguished by  $k$ -WL, the  $k$ -dimensional Weisfeiler-Le(h)man test, if  $k \leq 3$  [2, 7, 52]. Additionally, we explore how curvature filtrations can count substructures, an important tool for evaluating and comparing expressivity [56]. To the best of our knowledge, ours is the first work to explore discrete curvature and curvature-based filtrations in this context.
Distinguishing Strongly-Regular Graphs. Strongly-regular graphs are often used to assess the expressive power of graph learning algorithms, constituting a class of graphs that are particularly hard to distinguish. We briefly recall some definitions. A connected graph  $G$  with diameter  $D$  is called distance regular if there are integers  $b_{i}, c_{i}$ ,  $(0 \leq i \leq D)$  such that for any two vertices  $x, y \in V$  with  $d(x, y) = i$ , there are  $c_{i}$  neighbours of  $y$  in  $k_{i-1}(x)$  and  $b_{i}$  neighbours of  $y$  in  $k_{i+1}(x)$ . For a distance-regular graph, the intersection array is given by  $\{b_{0}, b_{1}, \ldots, b_{D-1}; c_{1}, c_{2}, \ldots, c_{D}\}$ . A graph is called strongly regular if it is distance regular and has a diameter of 2 [18]. We first state two theoretical results about curvature.
Theorem 6 (Expressivity of curvature notions). Both Forman-Ricci curvature and Resistance curvature cannot distinguish distance-regular graphs with the same intersection array, whereas Ollivier-Ricci curvature can distinguish the Rook and Shrikhande graphs, which are strongly-regular graphs with the same intersection array.