SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To compare vertex- and edge-color persistence diagrams, we consider graphs with vertex-coloring functions  $c(\cdot)$  from which we derive edge-coloring ones  $l(\cdot)$ . In particular, for a graph  $G = (V,E,c,X)$ , its edge-coloring function  $l:E\to X^2$  is defined as  $l(v,w) = \{c(v),c(w)\}$ .
Figure 4: Illustration of graphs that cannot be distinguished based on (a) edge-color filtrations, (b) vertex-color filtrations, and (c) both vertex- and edge-color filtrations.
Recall that only vertex-color filtrations can 1) encode multiset of colors and 2) have real holes with different birth times. Naturally, we can find pairs of graphs  $(G,G^{\prime})$  for which we can obtain  $\mathcal{D}_G^0\neq \mathcal{D}_{G'}^0$  from vertex-color filtrations, but not from edge-color ones. Consider the graphs in Figure 4(a). The vertex-color filtration  $f(\text{'blue'}) = 1,f(\text{'orange'}) = 2$  produces  $\mathcal{D} = \{\{(1,\infty),(1,\infty),(1,2),(2,2),(2,2)\} \}$  and  $\mathcal{D}' = \{\{(1,\infty),(1,1),(2,\infty),(2,2),(2,2)\} \}$ . However, there is no edge-color filtration that would tell them apart — there are only two possible edge-color filtrations, leading to either  $\mathcal{D} = \{\{(0,\infty),(0,\infty),(0,1),(0,2),(0,2)\} \} = \mathcal{D}'$ , or  $\mathcal{D} = \{\{(0,\infty),(0,\infty),(0,1),(0,1),(0,2)\} \} = \mathcal{D}'$ .
We can also show that there are graphs that can be distinguished by edge-color filtrations but not by vertex-color ones. Intuitively, one can think of this as a result of edge colors being more fine-tuned. For instance, consider the graphs in Figure 4(b). We can separate these graphs using the function  $f(\text{'orange'}) = 1$ ,  $f(\text{'blue-orange'}) = 2$ , and  $f(\text{'blue'}) = 3$ , which yields  $\mathcal{D}_G^0 = \{(0,\infty),(0,1),(0,1),(0,2),(0,2),(0,3)\} \neq \{(0,\infty),(0,1),(0,1),(0,2),(0,2),(0,2)\} = \mathcal{D}_{G'}^0$ . However, since there is no color-separating set for  $G$  and  $G'$ , by Theorem 1, we have that  $\mathcal{D}_G = \mathcal{D}_{G'}$  for all vertex-color filtrations. Theorem 3 formalizes the idea that none of the classes of color-based filtrations subsumes the other. In addition, Figure 4(c) illustrates that there are very simple non-isomorphic graphs that PH under both vertex- and edge-color filtrations cannot distinguish.
Theorem 3 (Edge-color vs. vertex-color filtrations). There exist non-isomorphic graphs that vertex-color filtrations can distinguish but edge-color filtrations cannot, and vice-versa.
4 Going beyond persistent homology
We now leverage the theoretical results in Section 3 to further boost the representational capability of persistent homology. In particular, we propose modifying edge-color persistence diagrams to account for structural information that is not captured via the original diagrams. We call the resulting approach RePHINE (Refining PH by incorporating node-color into edge-based filtration). Notably, RePHINE diagrams are not only provably more expressive than standard color-based ones but also make 1-dimensional topological features redundant. Additionally, we show how to integrate RePHINE into arbitrary GNN layers for graph-level prediction tasks.
Edge-color diagrams with missing holes. A major drawback of edge-color filtrations is that information about the multisets of (edge) colors is lost, i.e., it cannot be recovered from persistence diagrams. To reconstruct disconnecting sets, we need the edge-color permutation given by the filtration function and the number of edges — both of which cannot be deduced from diagrams alone.
To fill this gap, we introduce the notion of missing holes. Conceptually, missing holes correspond to edges that are not associated with the disappearance of any connected component in a given filtration. By design, we set the birth time of missing holes to 1 — this distinguishes them from real and almost holes, which have birth times equal to 0. The death time of a missing hole corresponds to the first filtration step  $f(x)$  that its corresponding edge color  $x$  appears. We note that missing holes correspond to cycles obtained from 1-dim persistence diagrams.
As an example, consider the edge-color filtration in Figure 5, which produces  $\mathcal{D}^0 = \{(0,\infty),(0,1),(0,2),(0,2),(0,4)\}$ . We note that the orange edge and one of the orange-green ones do not 'kill' any 0-dim hole. This results in the missing holes (1,3) and (1,4). Clearly, missing holes bring in additional expressivity as, e.g., it would be possible to distinguish graphs that only differ in the orange edge in Figure 5. Still, edge-color diagrams with missing holes are not more expressive than vertex-color ones. For instance, they cannot separate the two graphs in Figure 3(a).
Augmenting edge-color diagrams with vertex-color information. To improve the expressivity of persistent homology, a simple approach is to merge tuples obtained from independent vertex- and edge-color filtrations. However, this would double the computational cost while only allowing distinguishing pairs of graphs that could already be separated by one of the filtrations. Ideally, we would like to go beyond the union of vertex- and edge-color persistence diagrams.
As in Section 3.3, we consider graphs with edge colors obtained from vertex-coloring functions. Also, we assume that  $f_{v}$  and  $f_{e}$  are independent vertex- and edge-color filtration functions, respectively. We propose adding two new elements to the tuples of edge-color diagrams with missing holes. Our augmented tuple is  $(b,d,\alpha ,\gamma)$  where  $\alpha$  and  $\gamma$  are the additional terms. Recall that, in any edge-color filtration,  $G_{0}$  has  $|V|$  connected components. Then, we can associate real or almost holes of edge-color diagrams with vertices in  $G$ . With this in mind, we define RePHINE diagrams as follows.
Figure 5: RePHINE diagrams. At  $G_{1}$ , one component dies and creates the almost hole (0,1,2,1). We also save that two nodes were discovered at 1 (fourth component), with colors equal to 2 (third component). At step 2, two other holes are killed, resulting in two tuples (0,2,1,2). At  $G_{3}$ , we obtain the missing hole (1,3,0,0). Finally,  $G_{4}$  creates one almost hole and one missing hole.
Definition 3 (RePHINE diagram). The RePHINE diagram of a filtration on a graph  $G$  is a multiset of cardinality  $|V| + \beta_G^1$ , with elements of form  $(b, d, \alpha, \gamma)$ . There are two cases:
- Case  $b = 0$  (real or almost holes). Now,  $b$  and  $d$  correspond to birth and death times of a component as in edge-color filtration. We set  $\alpha(w) = f_v(c(w))$  and  $\gamma(w) = \min_{v \in \mathcal{N}(w)} f_e(\{c(w), c(v)\})$ , where  $w$  is the vertex that is associated with the almost or real hole. Vertices are matched with the diagram elements as follows: An almost hole  $(b,d)$  corresponds to an edge merging two connected components,  $T_1, T_2$ . Each of these connected components has exactly one vertex,  $w_{T_1}$  or  $w_{T_2}$ , which has not yet been associated with any element of the RePHINE diagram. Let  $w = \arg \max_{w' \in \{w_{T_1}, w_{T_2}\}} f_v(c(w'))$ , or if  $f_v(c(w_{T_1})) = f_v(c(w_{T_2}))$ , then  $w = \arg \min_{w' \in \{w_{T_1}, w_{T_2}\}} \gamma(w')$ . The vertices that are associated with real holes are vertices that have not died after the last filtration step.
- Case  $b = 1$  (missing holes). Here, the entry  $d$  is the filtration value of an edge  $e$  that did not kill a hole but produces a cycle that appears at the filtration step associated with adding the edge  $e$ . The entries  $\alpha$  and  $\gamma$  take uninformative values (e.g., 0).
Figure 5 provides an example of RePHINE diagrams. Further details of the procedure can be found in Appendix C. Notably, our scheme can be computed efficiently at the same cost as standard persistence diagrams and is consistent — we obtain identical diagrams for any two isomorphic colored graphs.
Theorem 4 (RePHINE is isomorphism invariant). Let  $G$ ,  $G'$  be isomorphic graphs. Then, any edge-color and vertex-color filtrations produce identical RePHINE diagrams for  $G$  and  $G'$ .
In addition, Theorem 5 shows that RePHINE diagrams are strictly more expressive than those from both vertex- and edge-color filtrations, including 0- and 1-dim topological features. Figure 4(c) provides an example of graphs that cannot be recognized by any color-based filtration, but for which we can obtain distinct RePHINE diagrams.
Theorem 5 (RePHINE is strictly more expressive than color-based PH). Let  $\mathcal{D},\mathcal{D}'$  be the persistence diagrams associated with any edge or vertex-color filtration of two graphs. If  $\mathcal{D}\neq \mathcal{D}'$ , then there is a filtration that produces different RePHINE diagrams. The converse does not hold.
Despite its power, there are simple non-isomorphic graphs RePHINE cannot distinguish. In particular, if two graphs have one color, RePHINE cannot separate graphs of equal size with the same number of components and cycles. For example, star and path graphs with 4 vertices of color  $c_{1}$  produce identical RePHINE diagrams of the form  $\{(0,d,a,d),(0,d,a,d),(0,d,a,d),(0,\infty ,a,d)\}$ , where  $d = f_{e}(\{c_{1},c_{1}\})$  and  $a = f_{v}(c_{1})$  for arbitrary edge- and vertex-color filtration functions.
Combining RePHINE and GNNs. RePHINE diagrams can be easily incorporated into general GNN layers. For instance, one can follow the scheme in [15] to combine non-missing hole information with node features and leverage missing holes as graph-level attributes. However, here we adopt a simple scheme that processes RePHINE tuples using DeepSets [38]. These topological embeddings are then grouped using a pooling layer and concatenated with the graph-level GNN embedding. The resulting representation is fed to a feedforward network to obtain class predictions. Formally, let  $\mathcal{N}_G(u)$  denote the set of neighbors of vertex  $u$  in  $G$ , and  $h_u^{(0)} = c(u)$  for all  $u \in V$ . We compute
Figure 6: Average learning curves for RePHINE, PH, and GCN on connected cubic graphs. RePHINE can learn representations in cases where PH and GNNs struggle to capture structural information. RePHINE shows better expressivity and fitting capability on Cub10-2 and Cub12-3.
GNN and RePHINE embeddings (denoted by  $r^{(\ell)}$ ) at layer  $\ell$  recursively as:
$$
\tilde {h} _ {u} ^ {(\ell)} = \operatorname {A G G} ^ {(\ell)} \left(\left\{\left\{h _ {w} ^ {(\ell - 1)} \mid w \in \mathcal {N} _ {G} (u) \right\} \right\}\right) \quad \forall u \in V
$$
$$
\mathcal {R} ^ {(\ell)} = \operatorname {R E P H I N E} \left(f _ {v} ^ {(\ell)}, f _ {e} ^ {(\ell)}, \{\{h _ {u} ^ {(\ell)} \} \} _ {u \in V}\right)
$$
$$
h _ {u} ^ {(\ell)} = \operatorname {U P D A T E} ^ {(\ell)} \left(h _ {u} ^ {(\ell - 1)}, \tilde {h} _ {u} ^ {(\ell)}\right) \quad \forall u \in V
$$
$$
r ^ {(\ell)} = \phi^ {(\ell)} \left(\sum_ {d \in \mathcal {R} ^ {(\ell)}} \psi^ {(\ell)} (d)\right)
$$
where  $f_{v}^{(\ell)}, f_{e}^{(\ell)}, \psi^{(\ell)}, \phi^{(\ell)}, \mathrm{AGG}^{(\ell)}$ , and  $\mathrm{UPDATE}^{(\ell)}$  are arbitrary non-linear mappings, usually implemented as feedforward neural nets. After  $L$  layers, we obtain the combined RePHINE-GNN graph-level representation as  $[\mathrm{POOL}_1(\{r^{(\ell)}\}_\ell) \parallel \mathrm{POOL}_2(\{h_u^{(L)}\}_u)]$ , where  $\mathrm{POOL}_1$  is either mean or concatenation, and  $\mathrm{POOL}_2$  is an order invariant operation.
In this section, we compare RePHINE to standard persistence diagrams from an empirical perspective. Our main goal is to evaluate whether our method enables powerful graph-level representation, confirming our theoretical analysis. Therefore, we conduct two main experiments. The first one leverages an artificially created dataset, expected to impose challenges to persistent homology and MP-GNNs. The second experiment aims to assess the predictive performance of RePHINE in combination with GNNs on popular benchmarks for graph classification. All methods were implemented in PyTorch [31], and our code is available at https://github.com/Aalto-QuML/RePHINE.
Synthetic data. We consider three datasets of cubic graphs (or 3-regular graphs): cub08, cub10, and cub12 [6]. These graphs cannot be distinguished by 1-WL and color-based PH as all vertices share the same color. Thus, we modify the datasets by changing the colors of 1, 2, or 3 vertices in each graph sample, resulting in the modified datasets cub08-1, cub10-2, and cub12-3. Also, we randomly partition each dataset and create a balanced binary classification task. We expect this to keep the hardness of the task while allowing some distinguishability.
We compare standard 0-dim persistence diagrams from vertex-color filtrations (referred to as PH) to 0-dim RePHINE (i.e., no missing holes). Both approaches are processed using DeepSets with exactly the same structure and optimization procedure. Also, they operate on the original colors, not on GNN embeddings. For completeness, we report results for a 2-layer graph convolutional network (GCN) [22] followed by an MLP. We are interested in assessing if the persistence modules can overfit the observed graphs. We also monitor if the methods obtain different representations for each graph, measured in terms of the proportion of unique graph embeddings over training (which we call expressivity). We provide further details and additional results with 1-dim persistence diagrams in the supplementary material.
Table 1: Predictive performance on graph classification. We denote in bold the best results. For ZINC, lower is better. For most datasets, RePHINE is the best-performing method.
GNN	Diagram	NCI109↑	PROTEINS ↑	IMDB-B ↑	NCI1 ↑	MOLHIV ↑	ZINC ↓
GCN	-	76.46 ± 1.03	70.18 ± 1.35	64.20 ± 1.30	74.45 ± 1.05	74.99 ± 1.09	0.875 ± 0.009
PH	77.92 ± 1.89	69.46 ± 1.83	64.80 ± 1.30	79.08 ± 1.06	73.64 ± 1.29	0.513 ± 0.014
RePHINE	79.18 ± 1.97	71.25 ± 1.60	69.40 ± 3.78	80.44 ± 0.94	75.98 ± 1.80	0.468 ± 0.011
GIN	-	76.90 ± 0.80	72.50 ± 2.31	74.20 ± 1.30	76.89 ± 1.75	70.76 ± 2.46	0.621 ± 0.015
PH	78.35 ± 0.68	69.46 ± 2.48	69.80 ± 0.84	79.12 ± 1.23	73.37 ± 4.36	0.440 ± 0.019
RePHINE	79.23 ± 1.67	72.32 ± 1.89	72.80 ± 2.95	80.92 ± 1.92	73.71 ± 0.91	0.411 ± 0.015
Figure 6 shows the learning curves for 2000 epochs, averaged over five runs. Notably, for all datasets, the expressivity of RePHINE is significantly higher than those from PH and similar to GNN's. On cub10-2, while PH and GNN obtain accuracies of around 0.5, RePHINE allows a better fit to the observed data, illustrated by higher accuracy and lower loss values.
Real-world data. To assess the performance of RePHINE on real data, we use six popular datasets for graph classification (details in the Supplementary): PROTEINS, IMDB-BINARY, NCI1, NCI109, MOLHIV and ZINC [7, 16, 20]. We compare RePHINE against standard vertex-color persistence diagrams (simply called PH here). Again, we do not aim to benchmark the performance of topological GNNs, but isolate the effect of the persistence modules. Thus, we adopt default (shallow) GNN architectures and process the persistence diagrams exactly the same way using DeepSets. We report the mean and standard deviation of predictive metrics (AUC for MOLHIV, MAE for ZINC, and Accuracy for the remaining) over five runs. We provide further implementation details in Appendix C.
Table 1 shows the results of PH and RePHINE combined with GCN [22] and GIN [37] models. Notably, RePHINE consistently outperforms PH, being the best-performing method in 10 out of 12 experiments. Overall, we note that GIN-based approaches achieve slightly better results. Our results suggest that RePHINE should be the default choice for persistence descriptors on graphs.
Table 2: PersLay vs. RePHINE: Accuracy results on graph classification.
Method	NCI109	PROTEINS	IMDB-B	NCI1
PersLay	70.12 ± 0.83	67.68 ± 1.94	68.60 ± 5.13	68.86 ± 0.86
RePHINE+Linear	73.27 ± 1.69	71.96 ± 1.85	70.40 ± 2.97	74.94 ± 1.35
Comparison to PersLay [2]. We also compare our method against another topological neural network, namely, PersLay. Since PersLay does not leverage GNNs, we adapted our initial design for a fair comparison. Specifically, we compute RePHINE diagrams with learned filtration functions and apply a linear classifier to provide class predictions. Also, we concatenate the vectorial representations of the RePHINE diagrams with the same graph-level features obtained using PersLay. We refer to our variant as RePHINE+Linear. Table 2 reports accuracy results over 5 runs on 4 datasets. For all datasets, RePHINE+Linear achieves higher accuracy, with a significant margin overall.
6 Conclusion, Broader Impact, and Limitations
We resolve the expressivity of persistent homology methods for graph representation learning, establishing a complete characterization of attributed graphs that can be distinguished with general node- and edge-color filtrations. Central to our analyses is a novel notion of color-separating sets.
Much like how WL test has fostered more expressive graph neural networks (GNNs), our framework of color-separating sets enables the design of provably more powerful topological descriptors such as RePHINE (introduced here). RePHINE is computationally efficient and can be readily integrated into GNNs, yielding empirical gains on several real benchmarks.
We have not analyzed here other types of filtrations, e.g., those based on the spectral decomposition of graph Laplacians. Future work should also analyze the stability, generalization capabilities, and local versions of RePHINE. Overall, we expect this work to spur principled methods that can leverage both topological and geometric information, e.g., to obtain richer representations for molecules in applications such as drug discovery and material design.
This work was supported by Academy of Finland (Flagship programme: Finnish Center for Artificial Intelligence FCAI) and a tenure-track starting grant by Aalto University. We also acknowledge the computational resources provided by the Aalto Science-IT Project from Computer Science IT. We are grateful to the anonymous area chair and reviewers for their constructive feedback. Johanna Immonen thanks Tuan Anh Pham, Jannis Halbey, Negar Soltan Mohammadi, Yunseon (Sunnie) Lee, Bruce Nguyen and Nahal Mirzaie for their support and for all the fun during her research internship at Aalto University in the summer of 2022.
[1] P. Barceló, E. V. Kostylev, M. Monet, J. Pérez, J. L. Reutter, and J.-P. Silva. The logical expressiveness of graph neural networks. In International Conference on Learning Representations (ICLR), 2020.
[2] M. Carrière, F. Chazal, Y. Ike, T. Lacombe, M. Royer, and Y. Umeda. PersLay: A neural network layer for persistence diagrams and new graph topological signatures. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
[3] M. Carriere, F. Chazal, M. Glisse, Y. Ike, H. Kannan, and Y. Umeda. Optimizing persistent homology based functions. In International Conference on Machine Learning (ICML), 2021.
[4] Y. Chen, B. Coskunuzer, and Y. Gel. Topological relational learning on graphs. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[5] D. Cohen-Steiner, H. Edelsbrunner, and J. Harer. Stability of persistence diagrams. In Proceedings of the Twenty-First Annual Symposium on Computational Geometry, 2005.
[6] K. Coolsaet, S. D'hondt, and J. Goedgebeur. House of graphs 2.0: A database of interesting graphs and more. Discrete Applied Mathematics, 325:97-107, 2023.
[7] V. P. Dwivedi, C. K. Joshi, A. T. Luu, T. Laurent, Y. Bengio, and X. Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 24(43):1-48, 2023.
[8] Edelsbrunner, Letscher, and Zomorodian. Topological persistence and simplification. Discrete & Computational Geometry, 28(4), 2002.
[9] H. Edelsbrunner and J. Harer. Computational Topology - an Introduction. American Mathematical Society, 2010.
[10] M. Fey and J. E. Lenssen. Fast graph representation learning with PyTorch Geometric. In Workshop track of the International Conference on Representation Learning (ICLR), 2019.
[11] V. Garg, S. Jegelka, and T. Jaakkola. Generalization and representational limits of graph neural networks. In International Conference on Machine Learning (ICML), 2020.
[12] C. Hofer, R. Kwitt, M. Niethammer, and A. Uhl. Deep learning with topological signatures. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[13] C. Hofer, F. Graf, B. Rieck, M. Niethammer, and R. Kwitt. Graph filtration learning. In International Conference on Machine Learning (ICML), 2020.
[14] C. D. Hofer, R. Kwitt, and M. Niethammer. Learning representations of persistence barcodes. Journal of Machine Learning Research, 20(126):1-45, 2019.
[15] M. Horn, E. De Brouwer, M. Moor, Y. Moreau, B. Rieck, and K. Borgwardt. Topological graph neural networks. In International Conference on Learning Representations (ICLR), 2022.
[16] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687, 2020.
[17] Xiaoling Hu, Fuxin Li, Dimitris Samaras, and Chao Chen. Topology-preserving deep image segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[18] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
Supplementary material: Going beyond persistent homology using persistent homology
A Persistent homology
Persistent homology (PH) is one of the workhorses for topological data analysis (TDA). A central idea underlying PH is to investigate the multiresolution structure in data through the lens of low-dimensional topological features such as connected components (0-dimensional), loops (1-dimensional), and voids (2-dimensional). Here, we provide a brief description of PH, and how it extends to graphs. In particular, we do not present proofs and do not show that the constructions are well-defined. For a detailed treatment, we refer the reader to [13], [9].
We will first define homological groups. They allow to characterise p-dimensional holes in a topological space such as a simplicial complex. We present the theory for simplicial complexes, as our focus is on 1-dimensional simplicial complexes (i.e. graphs).
Let  $K$  be a simplicial complex. The  $p$ -chains are formal sums  $c = \sum a_i \sigma_i$ , where  $a_i \in \mathbb{Z}/2\mathbb{Z}$  and  $\sigma_i$  are  $p$ -simplices in  $K$ . One can think of  $p$ -chain as a set of  $p$ -simplices such that  $a_i = 1$ . Together with componentwise addition,  $p$ -chains form the group  $C_p(K)$ .
Now, consider a simplex  $\sigma = (v_0,\dots,v_p)\in K$ . We can define a boundary for  $\sigma$  by
$$
\partial_ {p} \sigma = \sum_ {j = 0} ^ {p} (v _ {0}, \dots , v _ {j - 1}, v _ {j + 1}, \dots , v _ {p}),
$$
i.e.,  $\partial_p\sigma$  is a sum of the  $(p - 1)$ -dimensional faces of  $\sigma$ . We can extend this to define a boundary homomorphism  $\partial_p:C_p(K)\to C_{p - 1}(K)$  where  $\partial_p\sum a_i\sigma_i = \sum a_i\partial_p\sigma_i$ . Thus, we can define a sequence of groups
$$
\dots C _ {p + 1} (K) \xrightarrow {\partial_ {p + 1}} C _ {p} (K) \xrightarrow {\partial_ {p}} C _ {p - 1} (K) \dots ,
$$
each connected with a boundary homomorphism. This sequence is chain complex, and it is the last definition we need in order to consider homology groups.
The  $p$ th homology group is a group of  $p$ -chains with empty boundary (i.e.  $\partial_p\sigma = 0$ ) such that each of these particular  $p$ -chains (cycles) are a boundary of a different simplex in  $C_{p + 1}(K)$ . So, we can define the homology group as the quotient space
$$
H _ {p} = \ker \partial_ {p} / \operatorname {I m} \partial_ {(p + 1)}.
$$
The rank of  $H_{p}$  is equal to the  $p$ th Betti number  $(\beta_{p})$ . Then, let us see how the homology groups can be refined to gain persistent homology groups.