SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Johanna Immonen*
johanna.x.immonen@helsinki.fi
Amauri H. Souza*
Aalto University Federal Institute of Ceará amauri.souza@aalto.fi
Vikas Garg
Aalto University  
YaiYai Ltd  
vgarg@csail.mit.edu
Representational limits of message-passing graph neural networks (MP-GNNs), e.g., in terms of the Weisfeiler-Leman (WL) test for isomorphism, are well understood. Augmenting these graph models with topological features via persistent homology (PH) has gained prominence, but identifying the class of attributed graphs that PH can recognize remains open. We introduce a novel concept of color-separating sets to provide a complete resolution to this important problem. Specifically, we establish the necessary and sufficient conditions for distinguishing graphs based on the persistence of their connected components, obtained from filter functions on vertex and edge colors. Our constructions expose the limits of vertex- and edge-level PH, proving that neither category subsumes the other. Leveraging these theoretical insights, we propose RePHINE for learning topological features on graphs. RePHINE efficiently combines vertex- and edge-level PH, achieving a scheme that is provably more powerful than both. Integrating RePHINE into MP-GNNs boosts their expressive power, resulting in gains over standard PH on several benchmarks for graph classification.
Topological data analysis (TDA) is a rapidly growing field that provides tools from algebraic topology for uncovering the shape (or structure) of data, allowing for efficient feature extraction. Its flagship tool is persistent homology (PH) [8], which seeks to characterize topological invariants (e.g., connected components, loops) of an underlying manifold based on data samples. Notably, PH has been successfully applied in many scientific domains, including computer vision [17, 27], drug design [23], fluid dynamics [24], and material science [25].
For graphs, PH has been used to provide global topological signatures for graph-level prediction tasks [2, 12, 14, 33, 39] and act as local message modulators in graph neural networks (GNNs) for node-level tasks [4, 40]. By leveraging learnable filtration/vectorization maps, PH has also been integrated into neural networks as a building block in the end-to-end learning process [2, 3, 13, 15, 26]. These strategies allow us to exploit topological features to boost the predictive capabilities of graph models. However, in stark contrast with the developments on the representational power of GNNs [1, 11, 28-30, 34, 35, 37], the theoretical properties of PH on graphs remain much less explored. For instance, open questions include: Which graph properties can PH capture? What is the characterization of pairs of graphs that PH cannot separate? Can we improve the expressivity of PH on graphs?
In a recent work, Rieck [32] discusses the expressivity of PH on graphs in terms of the Weisfeiler-Leman (WL) hierarchy [36]. The paper shows that, given different k-WL colorings, there exists a filtration such that the corresponding persistence diagrams differ. This result provides a lower bound for the expressivity in terms of WL hierarchy, but it does not describe the class of graphs which can be distinguished via PH. In this paper, we aim to fully characterize this class of graphs.
Figure 1: Overview of our theoretical results.
We study the expressive power of PH on attributed (or colored) graphs, viewed as 1-dim simplicial complexes. We focus on the class of graph filtrations induced by functions on these colors. Importantly, such a class is rather general and reflects choices of popular methods (e.g., topological GNNs [15]). We first analyze the persistence of connected components obtained from vertex colors. Then, we extend our analysis to graphs with edge colors. To obtain upper bounds on the expressive power of color-based PH, we leverage the notion of separating/disconnecting sets. This allows us to establish the necessary and sufficient conditions for the distinguishability of two graphs from 0-dim persistence diagrams (topological descriptors). We also provide constructions that highlight the limits of vertex- and edge-color PH, proving that neither category subsumes the other.
Based on our insights, we present RePHINE (short for “Refining PH by Incorporating Node-color into Edge-based filtration”), a simple method that exploits a subtle interplay between vertex- and edge-level persistence information to improve the expressivity of color-based PH. Importantly, RePHINE can be easily integrated into GNN layers and incurs no computational burden to the standard approach. Experiments support our theoretical analysis and show the effectiveness of RePHINE on three synthetic and six real datasets. We also show that RePHINE can be flexibly adopted in different architectures and outperforms PersLay [2] — a popular topological neural net.
In sum, our contributions are three-fold:
(Theory) We establish a series of theoretical results that characterize PH on graphs, including bounds on the expressivity of vertex- and edge-level approaches, the relationship between these approaches, and impossibility results for color-based PH — as summarized in Figure 1.
(Methodology) We introduce a new topological descriptor (RePHINE) that is provably more expressive than standard 0-dim and 1-dim persistence diagrams.
(Experiments) We show that the improved expressivity of our approach also translates into gains in real-world graph classification problems.
We consider arbitrary graphs  $G = (V, E, c, X)$  with vertices  $V = \{1, 2, \ldots, n\}$ , edges  $E \subseteq V \times V$  and a vertex-coloring function  $c: V \to X$ , where  $X$  denotes a set of  $m$  colors or features  $\{x_1, x_2, \ldots, x_m\}$  such that each color  $x_i \in \mathbb{R}^d$ . We say two graphs  $G = (V, E, c, X)$  and  $G' =$
$(V', E', c', X')$  are isomorphic (denoted by  $G \cong G'$ ) if there is a bijection  $g: V \to V'$  such that  $(v, w) \in E$  iff  $(g(v), g(w)) \in E'$  and  $c = c' \circ g$ . Here, we also analyze settings where graphs have an edge-coloring function  $l$ .
A filtration of a graph  $G$  is a finite nested sequence of subgraphs of  $G$ , that is,  $G_{1} \subseteq G_{2} \subseteq \ldots \subseteq G$ . Although the design of filtrations can be flexible [12], a typical choice consists of leveraging a vertex filter (or filtration) function  $f: V \to \mathbb{R}$  for which we can obtain a permutation  $\pi$  of  $n$  vertices such that  $f(\pi(1)) \leq f(\pi(2)) \cdots \leq f(\pi(n))$ . Then, a filtration induced by  $f$  is an indexed set  $\{G_{f(\pi(i))}\}_{i=1}^{n}$  such that  $G_{f(\pi(i))} \subseteq G$  is the subgraph induced by the set of vertices  $V_{f(\pi(i))} = \{v \in V \mid f(v) \leq f(\pi(i))\}$ . Note that filtration functions which give the same permutation of vertices induce the same filtration. Persistent homology keeps track of the topological features of each subgraph in a filtration. For graphs  $G$ , these features are either the number of connected components or independent cycles (i.e., 0- and 1-dim topological features, denoted respectively by the Betti numbers  $\beta_G^0$  and  $\beta_G^1$ ) and can be efficiently computed using computational homology. In particular, if a topological feature first appears in  $G_{f(\pi(i))}$  and disappears in  $G_{f(\pi(j))}$ , then we encode its persistence as a pair  $(f(\pi(i)), f(\pi(j)))$ ; if a feature does not disappear in  $G_{f(\pi(n))} = G$ , then its persistence is  $(f(\pi(i)), \infty)$ . The collection of all pairs forms a multiset that we call persistence diagram [5]. We use  $\mathcal{D}^0$  and  $\mathcal{D}^1$  to refer to the persistence diagrams for 0- and 1-dim topological features respectively. Appendix A provides a more detailed treatment for persistent homology.
Recent works have highlighted the importance of adopting injective vertex filter functions. Hofer et al. [13] show that injectivity of parameterized functions  $f_{\theta}:V\to \mathbb{R}$  is a condition for obtaining well-defined gradients with respect to the parameters  $\theta$ , enabling end-to-end filtration learning. Also, Horn et al. [15] show that for any non-injective function, we can find an arbitrarily close injective one that is at least as powerful at distinguishing non-isomorphic graphs as the original (non-injective) function. However, Lemma 1 shows that filtrations induced by injective functions on vertices may result in inconsistent persistence diagrams; namely, different diagrams for isomorphic graphs.
Lemma 1 (Injective vertex-based filtrations can generate inconsistent persistence diagrams). Consider persistence diagrams obtained from injective vertex filter functions. There are isomorphic graphs  $G \cong G'$  such that their persistence diagrams are different, i.e.,  $\mathcal{D}_G \neq \mathcal{D}_{G'}$ .
To avoid inconsistent diagrams, we need to employ permutation equivariant filter functions — see [32, Lemma 2]. Common choices include vertex degree [12], eigenvalues of the graph Laplacian [2], and GNN layers [13], which are permutation equivariant by construction. Another option is to define graph filtrations based on vertex/edge colors [15], which are also equivariant by design, i.e., if  $G \cong G'$  with associated bijection  $g$ , then  $c(v) = c'(g(v)) \forall v \in V$ . Notably, color-based filtrations generalize the GNN-layers case since we could redefine vertex/edge-coloring functions to take the graph structure as an additional input. Thus, we now turn our attention to color-based filtrations.
Color-based filtrations. Let  $f: X \to \mathbb{R}$  be an injective function. Therefore,  $f$  must assign a strict total order for colors, i.e., there is a permutation  $\pi: \{1, \ldots, m\} \to \{1, \ldots, m\}$  such that  $f(x_{\pi(1)}) < \cdots < f(x_{\pi(m)})$ . We define the vertex-color filtration induced by  $f$  as the indexed set  $\{G_i\}_{i=1}^m$  where  $G_i = (V_i, E_i, c_i, X_i)$ , with  $X_i = \{x_{\pi(1)}, x_{\pi(2)}, \ldots, x_{\pi(i)}\}$ ,  $V_i = \{v \in V \mid c(v) \in X_i\}$ ,  $E_i = \{(v, w) \in E \mid c(v) \in X_i, c(w) \in X_i\}$ , and  $c_i = \{(v, c(v)) \mid v \in V_i\}$ . Similarly, we can define the edge-color filtration induced by  $f$  as  $\{G_i\}_{i=1}^m$  where  $G_i = (V, E_i, l_i, X_i)$  with  $X_i = \{x_{\pi(1)}, \ldots, x_{\pi(i)}\}$ ,  $E_i = \{(v, w) \in E \mid l(v, w) \in X_i\}$ , and  $l_i = \{((v, w), l(v, w)) \mid (v, w) \in E_i\}$ .
We denote the elements of a persistence diagram  $\mathcal{D}$  as pairs  $(f(x^{(b)}), f(x^{(d)}))$ , where  $x^{(b)}, x^{(d)} \in X$  are the colors associated with the birth and death of a hole (topological feature) in a filtration induced by  $f(\cdot)$ . In the following, we use the notation  $\{\{\cdot\}\}$  to denote multisets.
3 The power of 0-dim persistent homology under color-based filtrations
In this section, we analyze the representational power of persistent homology when adopting color-based filtrations. We focus on the persistence of connected components (0-dimensional holes). We separately discuss vertex-color (Section 3.1) and edge-color (Section 3.2) filtrations, and then compare these approaches in Section 3.3. Proofs for all Lemmas and Theorems are in Appendix B.
3.1 Vertex-color filtrations
To help characterize the expressivity of persistent homology, we propose classifying persistence pairs  $(f(x^{(b)}), f(x^{(d)}))$  as either real holes, almost holes, or trivial holes. In particular, if  $f(x^{(d)}) \neq \infty$
and  $f(x^{(b)}) \neq f(x^{(d)})$ , we say the pair  $(f(x^{(b)}), f(x^{(d)}))$  is an almost hole. If  $f(x^{(b)}) = f(x^{(d)})$ , the pair is called a trivial hole. Finally, we call  $(f(x^{(b)}), f(x^{(d)}))$  a real hole if  $f(x^{(d)}) = \infty$ .
3.1.1 Real holes as connected components
Real holes denote topological features that persist until a filtration reaches the entire graph. Thus, real holes from 0-dim persistence diagrams are associated with connected components, regardless of the filtration. Regarding the relationship between filtrations and real holes, Lemma 2 establishes the necessary and sufficient condition for the existence of a filtration that produces persistence diagrams with distinct real holes. Such a condition is associated with the notion of component-wise colors. Formally, let  $G$  and  $G'$  be two graphs with connected components  $C_1, \ldots, C_k$  and  $C_1', \ldots, C_{k'}$ , respectively. Also, let  $X_i = \{c(v) \mid v \in V_{C_i}\}$  be the set of colors in  $C_i$  and, similarly,  $X_i'$  be the colors in  $C_i'$ . We say that  $G$  and  $G'$  have distinct component-wise colors if  $\{X_i\}_{i=1}^k \neq \{X_i'\}_{i=1}^{k'}$ .
Lemma 2 (Equivalence between component-wise colors and real holes). Let  $G$  and  $G'$  be two graphs. There exists some vertex-color filtration such that their persistence diagrams  $\mathcal{D}_G^0$  and  $\mathcal{D}_{G'}^0$  have different multisets of real holes iff  $G$  and  $G'$  have distinct component-wise colors.
3.1.2 Almost holes as separating sets
Now we turn our attention to the characterization of almost holes. Our next result (Lemma 3) reveals the connection between almost holes and separating sets. Here, a separating set  $S$  of a graph  $G$  is a subset of its vertices whose removal disconnects some connected component of  $G$ .
Lemma 3 (Almost holes and separating sets). Regarding the relationship between almost holes and separating sets, the following holds:
1. Let  $(f(x^{(b)}), f(x^{(d)}))$  be an almost hole from a vertex-color filtration. Then the set  $S = \{w \in V \mid f(c(w)) \geq f(x^{(d)})\}$  is a separating set of  $G$ .
2. Let  $S$  be a separating set of  $G$  that splits a connected component  $C \subseteq G$  into  $k$  components  $C_1, C_2, \ldots, C_k$ . Then, there exists a filtration that produces  $k - 1$  almost holes if the set of colors of vertices in  $\cup_{i=1}^k V_{C_i}$  is disjoint from those of the remaining vertices, i.e.,  $\{c(v) \mid v \in V \setminus \cup_{i=1}^k V_{C_i}\} \cap \{c(v) \mid v \in \cup_{i=1}^k V_{C_i}\} = \emptyset$ .
The relationship between almost holes and separating sets elucidated in Lemma 3 raises the question if we can use separating sets (obtained from colors) to compare almost holes across different graphs. The answer is no: even if the diagrams of two graphs only differ in their multisets of almost holes, the graphs might not have separating sets that split them into different numbers of components. For instance, consider the graphs in Figure 2, where numbers denote filter values. The persistence diagrams  $\mathcal{D}_G^0 = \{(1,\infty),(2,3),(2,3),(3,3),(3,4),(4,4)\}$  and  $\mathcal{D}_{G'}^0 = \{(1,\infty),(2,3),(2,4),(3,3),(3,3),(4,4)\}$  only differ in almost holes. Still, we cannot pick colors whose removal of associated vertices would result in different numbers of components.
Figure 2: We cannot use color-based separating sets to compare almost holes across graphs. Although these filtrations produce different almost holes, there is no way to remove colors s.t. the resulting graphs have different numbers of components.
Next, we introduce the notion of color-separating sets (Definition 1). Importantly, Lemma 4 leverages this definition to characterize the graphs that can be distinguished based on almost holes. Specifically, it establishes that whenever the
diagrams of two graphs differ in their multiset of almost holes, we can build a color-separating set.
Definition 1 (Color-separating sets). A color-separating set for a pair of graphs  $(G, G')$  is a set of colors  $Q$  such that the subgraphs induced by  $V \setminus \{w \in V \mid c(w) \in Q\}$  and  $V' \setminus \{w \in V' \mid c'(w) \in Q\}$  have distinct component-wise colors.
We note that when  $G$  and  $G'$  have identical component-wise colors, the sets  $\{w \in V \mid c(w) \in Q\}$  and  $\{w \in V' \mid c'(w) \in Q\}$  induced by the color-separating set  $Q$  are separating sets for  $G$  and  $G'$ .
Lemma 4 (Distinct almost holes imply distinct color-separating sets). Let  $\mathcal{D}_G^0$ ,  $\mathcal{D}_{G'}^0$ , be persistence diagrams for  $G$  and  $G'$ . If the diagrams  $\mathcal{D}_G^0$ ,  $\mathcal{D}_{G'}^0$  differ in their multiset of almost holes, then there is a color-separating set for  $G$  and  $G'$ .
3.1.3 Bounds on the expressivity of vertex-color persistent homology
Regardless of the filtration, vertex-color PH always allows counting the numbers of connected components and vertices of a graph. If all vertices have the same color, then we cannot have any expressive power beyond  $\beta^0$  and  $|V|$  — when all vertices are added simultaneously, there cannot be almost holes as the finite living times of the holes are 0. Also, all real holes are identical, and we have  $\mathcal{D}^0 = \{(1,\infty),\ldots ,(1,\infty),(1,1),\ldots ,(1,1)\}$ , with  $|\mathcal{D}^0 | = |V|$ .
For graphs with  $m \geq 1$  colors, Lemma 5 shows that sets of birth times correspond to vertex colors. As a consequence, if the multisets of vertex colors differ for graphs  $G$  and  $G'$ , then the corresponding persistence diagrams are also different in all filtrations.
Lemma 5 (Equivalence between birth times and vertex colors). There is a bijection between the multiset of birth times and the multiset of vertex colors in any vertex-color filtration.
From Lemma 5, we can recover the multiset of colors from the persistence diagram and, consequently, distinguish graphs with different multisets. However, persistent homology uses vertex colors as input, and we do not need persistence diagrams to construct or compare such multisets. This highlights the importance of death times to achieve expressivity beyond identifying vertex colors. In fact, for non-trivial cases, the expressivity highly depends on the choice of filtration.
We have discussed the importance of color-separating sets (Lemma 4) and component-wise vertex colors (Lemma 2). With these notions, Theorem 1 formalizes the limits of expressivity that may be achieved with suitable filtration and characterize which pairs of graphs can, at best, be distinguished by comparing their persistence diagrams. Here, we only consider pairs of graphs that cannot be distinguished by their multiset of colors, as this corresponds to a trivial case.
Theorem 1 (The expressive power of vertex-color filtrations). For any two graphs  $G$  and  $G'$  with identical multiset of colors  $\{c(v) : v \in V\} = \{c'(v) : v \in V'\}$ , there exists a filtration such that  $\mathcal{D}_G^0 \neq \mathcal{D}_{G'}^0$  if and only if there is a color-separating set for  $G$  and  $G'$ .
3.2 Edge-color filtrations
We now consider the expressivity of 0-dim persistent homology obtained from edge-color filtrations. The persistence diagrams are constructed exactly the same way. However, in this case, all holes are born at the same time (all vertices appear in  $G_0$ ). This implies that all real holes are identical. Also, the diagrams do not contain trivial holes since  $G_0$  does not have edges. All holes are either real holes or almost holes (of the form  $(0, d)$ ). We also note that persistence diagrams will always have almost holes unless the graph is edgeless.
(b)
Figure 3: (a)  $G$  and  $G'$  differ in their multisets of colors, but no edge-color filtration can distinguish them. For instance, assume that  $f(\text{'blue'}) = 1 < 2 = f(\text{'orange'})$ . Then,  $\mathcal{D}_G^0 = \mathcal{D}_{G'}^0 = \{(0, \infty), (0, 1), (0, 1), (0, 1)\}$ . The same holds for  $f(\text{'blue'}) > f(\text{'orange'})$ . (b) Graphs that have different disconnecting sets, but for which we can find filtrations that lead to identical diagrams.
Analogously to separating sets in vertex-color filtrations, Lemma 6 characterizes edge-based almost holes as disconnecting sets — a set of edges whose removal would increase the number of components.
Lemma 6 (Edge-based almost holes as disconnecting sets). Let  $(0, f(x^{(d)}))$  be an almost hole from an edge-color filtration. Then  $S = \{e \in E \mid f(l(e)) \geq f(x^{(d)})\}$  is a disconnecting set of  $G$ .
Lemma 6 tells us how to construct a disconnecting set from an almost hole. Now, suppose we are given a disconnecting set  $S$ . Can we build an edge-color filtration for which  $S$  can be obtained from
an almost hole? In other words, can we obtain a diagram with an almost hole  $(0, f(x^{(d)}))$  such that  $\{e \in E \mid f(l(e)) \geq f(x^{(d)})\}$  is equal to  $S$ ? Lemma 7 shows that if the colors of edges in  $S$  are distinct from those in  $E \setminus S$ , then there is a filtration that induces a persistence diagram with an almost hole from which we can reconstruct  $S$ .
Lemma 7 (Reconstructing a disconnecting set). Let  $G = (V, E, l, X)$  be a graph and  $S \subseteq E$  be a disconnecting set for  $G$ . If the set of colors of  $S$  is disjoint from that of  $E \setminus S$ , then there exists a filtration such that  $S = \{e \in E \mid f(l(e)) \geq f(x^{(d)})\}$  for an almost hole  $(0, f(x^{(d)})) \in \mathcal{D}^0$ .
3.2.1 Bounds on the expressivity of edge-color persistent homology
Similar to the vertex-color case, in any edge-color filtration, we have that  $|\mathcal{D}^0| = |V|$  and the number of real holes is  $\beta^0$ . Also, the lowest expressivity is achieved when all edges have the same color. In this case, two graphs with different numbers of vertices or connected components have different persistence diagrams (and can be distinguished); otherwise, they cannot be distinguished.
We have seen in Lemma 5 that vertex-color filtrations encode colors as birth times. In contrast, birth times from edge-color filtrations are always trivially equal to zero. Thus, we cannot generalize Lemma 5 to edge-color filtrations. Instead, we can show there are graphs with different multiset of edge colors such that the graphs have the same persistence diagrams for any filtration (see Figure 3(a)).
Let us now consider lower limits for graphs with  $m > 1$  edge colors. We can show that even if two graphs have different disconnecting sets (obtained from colors), there are filtrations that induce the same persistence diagrams. To see this behavior, consider the two graphs in Figure 3(b), where the deletion of blue edges disconnects one of the graphs but not the other. Although we can build an edge-color filtration that separates these graphs (i.e.,  $\mathcal{D}_G^0 \neq \mathcal{D}_{G'}^0$ ), if we choose  $f(\text{'green}) = 3$ ,  $f(\text{'orange}) = 2$ , and  $f(\text{'blue'}) = 1$ , we obtain  $\mathcal{D}_G^0 = D_{G'}^0 = \{(0, \infty), (0, 1), (0, 2), (0, 2), (0, 2), (0, 2)\}$ . Interestingly, even if two graphs have different sets of edge colors, we might still find filtrations that induce identical diagrams. The reason is that unlike vertex-color filtrations where trivial holes make sure that all vertices are represented in the diagrams, in edge-color filtrations there are no trivial holes. As a result, persistence diagrams from edge-color filtrations do not account for edges that do not lead to the disappearance of connected components.
Lemma 6 and Lemma 7 showed that edge-based almost holes can be characterized as disconnecting sets, somewhat analogously to vertex-based almost holes as separating sets. We complete the analogy by introducing the notion of color-disconnecting sets in Definition 2. We then use this notion to fully characterize the expressive power of edge-color persistent homology in Theorem 2. More specifically, the existence of a color-disconnecting set between a given pair of graphs is a necessary and sufficient condition for distinguishing them based on 0-dimensional persistence diagrams.
Definition 2 (Color-disconnecting sets). A color-disconnecting set for a pair of graphs  $(G, G')$  is a set of colors  $Q$  such that if we remove the edges of colors in  $Q$  from  $G$  and  $G'$ , we obtain subgraphs with different numbers of connected components.
Theorem 2 (The expressive power of edge-color filtrations). Consider two graphs  $G$  and  $G'$ . There exists an edge-color filtration such that  $\mathcal{D}_G^0 \neq \mathcal{D}_{G'}^0$  if and only if there is a color-disconnecting set for  $G$  and  $G'$ .
3.3 Vertex-color versus edge-color filtrations