SlowGuess/NeurIPS_2023_OCR huggingface dataset:
we note that in a filtration where the colors of  $S$  are added the latest, there must either be more almost holes  $(0,f(x^{(d)}))$  in  $\mathcal{D}_G^0$  than in  $\mathcal{D}_{G'}^0$  with  $x^{(d)}\in S$ , or alternatively  $\beta_G^0\neq \beta_{G'}^0$ . In both cases,  $\mathcal{D}_G^0\neq \mathcal{D}_{G'}^0$  for some filter function  $f$ .
$[\Rightarrow]$  To prove the forward direction of the Theorem, we consider the cases where the edge-color diagrams differ in 1) their size, 2) the number of real holes, and 3) their almost holes.
Proof. Case 1:  $|\mathcal{D}_G| \neq |\mathcal{D}_{G'}|$ . Again, this corresponds to a trivial case, since if  $|\mathcal{D}_G| \neq |\mathcal{D}_{G'}|$ , then  $|V| \neq |V'|$ . Now,  $Q = X \cup X'$  is a color-disconnecting set.
Case 2:  $\mathcal{D}_G$  and  $\mathcal{D}_{G'}$  differ in their real holes. If there is a different count of real holes, then  $\beta_G^0 \neq \beta_{G'}^0$ , and  $Q = \emptyset$  is a color-disconnecting set.
Case 3:  $\mathcal{D}_G$  and  $\mathcal{D}_{G'}$  only differ in their almost holes. We now assume that  $|\mathcal{D}_G| = |\mathcal{D}_{G'}|$  and  $\beta_G^0 = \beta_{G'}^0$ , but  $\mathcal{D}_G \neq \mathcal{D}_{G'}$ . This means that there is some  $(0, d) \in \mathcal{D}$  such that there are more almost holes with this death time in  $\mathcal{D}_G$  than in  $\mathcal{D}_{G'}$ , without loss of generality. There may be several such almost holes (for which the diagrams differ) with distinct death times. Let's denote the set of the death times for these almost holes by  $D$ . Then, let  $d_{min}$  be the minimum of the death times in  $D$ , i.e.  $d_{min} = \min_{d \in D} d$ . Let us show that the set  $Q = \{x \in X \cup X' \mid f(x) > d_{min}\}$  disconnects  $G'$  into more connected components than  $G$ . For any lower filtration step, the induced subgraphs must have as many connected components because the almost holes corresponding to those steps match, and  $|\mathcal{D}_G| = |\mathcal{D}_{G'}|$ , i.e.  $|V| = |V'|$ , which means that at filtration step 0, we begin with equally many connected components. At filtration step  $d_{min}$ , we connect more components in  $G$  than in  $G'$  because there are more almost holes corresponding to this step in  $\mathcal{D}_G$  than in  $\mathcal{D}_{G'}$ . Now,  $Q$  must be a color-disconnecting set.
B.10 Proof of Theorem 3: Edge-color vs. vertex-color filtrations
Proof. This Theorem is proved in Section 3.3. In particular, Figure 3(a) provides an example of pairs of graphs that can be distinguished by vertex-color filtrations but not from edge-color ones. On the other hand, the graphs in Figure 3(b) can be distinguished by edge-color filtrations but not from vertex-color ones. This concludes the proof.
B.11 Proof of Theorem 4: RePHINE is isomorphism invariant
Proof. RePHINE diagram's isomorphism invariance stems from the fact that it is a function of a filtration on graph, and this filtration is gained from isomorphism invariant colorings. If this assumption is violated and the colorings are not gained in an invariant way, RePHINE diagrams can also be inconsistent.
It is easy to check that the tuples (b,d) are isomorphism invariant - when  $b = 0$ , these tuples correspond to diagrams gained from edge-color filtration. In this case, we can check the conditions given by Theorem 2 and note none of the conditions may be met with isomorphic graphs. With regard to  $b = 1$ , the set of missing holes is multiset of edge colors that did not appear in edge-color filtration diagram. This set can thus be gained by considering the multiset of edge colours and the edge-color diagram, which are both isomorphism invariant.
Further, it is also easy to see that the tuples  $(\alpha, \gamma)$  are invariant. When  $b = 0$ , the set of  $\alpha$ 's corresponds to the multiset of vertex colours, and for each vertex,  $\gamma = \min_{v \in \mathcal{N}(w)} f_e(\{c(w), c(v)\})$ .
However, the crucial part is how these two tuples are concatenated, i.e., how each of the vertices are associated with real and almost-holes. In particular, we need to check when two connected components are merged at a filtration step  $i$  and RePHINE compares the representatives (i.e. vertices of a connected component which have not yet 'died') of the two components, we will end up with same diagram elements of form  $(b,i,\alpha ,\gamma)$  regardless of the order we add the edges of color with filtration value  $i$ . In other words, while the RePHINE algorithm considers one edge at a time and does only pairwise comparisons between merged connected components, the order or these comparisons must not affect the decision on which vertices are associated with death time  $i$ . Let's consider what happens when adding all the edges of a color results in merging more than two components. Assume there is a new connected components constituting of old connected components  $T_{1},T_{2},\ldots,T_{n}$ . Now, there are two different cases. Assume first that there is a strict minimum among the vertex filtration values of the old representatives. Then, any pairwise comparison will lead to choosing this minimum
as the representative of the new connected component and all the other vertices will die at this filtration step. Then, assume there is no strict minimum but a tie between two or more representatives. Then, there will be comparisons based on  $\gamma$ , but choosing maximum of these is also permutation invariant function. In case there are two (or more) representatives such that there is a tie based on the vertex filtration values and  $\gamma$  values, choosing at random any of these leads to the same diagram. Lastly, note that for each real hole,  $(b,d) = (0,\infty)$ , and so it does not thus matter how each of the vertices are matched to the real holes, when rest of vertices are associated with almost-holes in an invariant way.
B.12 Proof of Theorem 5: RePHINE is strictly more expressive than color-based PH
Let  $\mathcal{R}_G$  denote the RePHINE diagram for a graph  $G$ . Similarly, let  $\mathcal{D}_{v,G}$  and  $\mathcal{D}_{e,G}$  denote persistence diagrams associated with vertex- and edge-color filtrations of  $G$ . We assume that  $\mathcal{D}_{v,G} = (\mathcal{D}_{v,G}^0,\mathcal{D}_{v,G}^1)$  and  $\mathcal{D}_{e,G} = (\mathcal{D}_{e,G}^0,\mathcal{D}_{e,G}^1)$  include 0- and 1-dim persistence diagrams. We want to show that for two graphs  $G$  and  $G'$
(i) if there is a vertex-color filtration such that  $\mathcal{D}_{v,G} \neq \mathcal{D}_{v,G'}$  then there is a filtration that leads to  $\mathcal{R}_G \neq \mathcal{R}_{G'}$ .
(ii) if there is a edge-color filtration such that  $\mathcal{D}_{e,G} \neq \mathcal{D}_{e,G'}$  then there is a filtration that leads to  $\mathcal{R}_G \neq \mathcal{R}_{G'}$ .
These results would show that RePHINE is at least as expressive as color-based persistence diagrams. We further show that
(iii) there is a pair of non-isomorphic graphs for which we can obtain  $\mathcal{R}_G \neq \mathcal{R}_{G'}$  but  $\mathcal{D}_{v,G} = \mathcal{D}_{v,G'}$  and  $\mathcal{D}_{e,G} = \mathcal{D}_{e,G'}$  for all vertex- and edge-color filtrations.
Proof. Part (i):  $\mathcal{D}_{v,G} \neq \mathcal{D}_{v,G'} \to \mathcal{R}_G \neq \mathcal{R}_{G'}$ . Let  $f$  be the vertex-color function associated with the standard diagrams  $\mathcal{D}$ . We can choose the RePHINE's vertex-level function  $f_v$  such that  $f_v = f$ . We note that the original diagrams can be obtained from an auxiliary edge-level filter function  $f_a$  where  $f_a(u,w) = \max(f(c(u)), f(c(w)))$ . The procedure is described in Algorithm 1.
Let  $f_{e}$  be the edge-color filter function of RePHINE. If we choose  $f_{e} = f_{a}$ , then RePHINE contains in the second and third elements of its tuples exactly the same persistence information of the vertex-color diagrams. Note that in this case, we do not even need to require injectivity of the edge-color filter  $f_{e}$  since the max function is not injective. Regarding the 1-dim features, for any tuple  $(d,\infty)$  in the 1-dim persistence diagram, we have a missing hole  $(1,d,\cdot ,\cdot)$  that comprises the same information. Thus, we have constructed vertex- and edge-color functions such that  $\mathcal{D}_{v,G}\neq \mathcal{D}_{v,G^{\prime}}\to \mathcal{R}_{G}\neq \mathcal{R}_{G^{\prime}}$ .
Part (ii):  $\mathcal{D}_{e,G} \neq \mathcal{D}_{e,G'} \to \mathcal{R}_G \neq \mathcal{R}_{G'}$ . This is a trivial case as RePHINE consists of an augmented version of standard edge-color diagrams. Let  $f$  be the edge-color (injective) filter function associated with the standard diagrams. In this case, we can simply set  $f_e = f$ , where  $f_e$  is RePHINE's edge-color filter. Then, the first and second elements of RePHINE's tuples correspond to  $\mathcal{D}_e$ . Regarding the 1-dim features, the only difference is the way the information is encoded. While we adopted the convention  $(1,d)$  for missing holes, the standard diagrams often use  $(d,\infty)$ . The relevant information is the same. Therefore, RePHINE is at least as expressive as edge-color persistence diagrams.
Part (iii). To show that RePHINE is strictly more expressive than color-based PH, it suffices to provide an example of two graphs for which there is a filtration such that  $\mathcal{R}_G \neq \mathcal{R}_{G'}$  but these graphs cannot be separated from any vertex- or edge-color filtration. We use the pair of graphs in Figure 3(c). We note that these graphs have no cycles, making 1-dim persistence information trivial.
We first note that their multisets of colors are identical and there is no color-separating sets for these two graphs — i.e., there is no subset of colors whose removal would separate the graphs into distinct component-wise colors. Thus, by Theorem 1, there is no vertex-color filtration s.t.  $\mathcal{D}_{v,G} \neq \mathcal{D}_{v,G'}$ .
Also, we have that  $|V| = |V'|$  and  $X = X'$  and  $\beta_G^0 = \beta_{G'}^0$ , and there is no color-disconnecting set for  $G$  and  $G'$  (i.e., there is no edge colors whose removal would generate subgraphs with different number of components). By Theorem 2, these graphs cannot be separated by any edge-color filtration.
However, if we choose the filter functions  $f_{v}(\text{blue}) = 1$ ,  $f_{v}(\text{orange}) = 2$ ,  $f_{e}(\text{blue-orange}) = 3$ , we obtain distinct RePHINE di
Table S1 reports summary statistics of the real-world datasets used in this paper. For the IMDB-B dataset, we use uninformative features (vector of ones) for all nodes. NCI1, NCI109, Proteins, and IMDB-B are part of the TU Datasets<sup>2</sup>, a vast collection of datasets commonly used for evaluating graph kernel methods and GNNs. MOLHIV is the largest dataset (over 41K graphs) and is part of the Open Graph Benchmark<sup>3</sup>. We also consider a regression task using the ZINC dataset — a subset of the popular ZINC-250K chemical compounds [19], which is particularly suitable for molecular property prediction [7].
Table S1: Statistics of the datasets.
Dataset	#graphs	#classes	#node labels	Avg #nodes	Avg #edges
NCI1	4110	2	37	29.87	32.30
IMDB-B	1000	2	-	19.77	96.53
PROTEINS (full)	1113	2	3	39.06	72.82
NCI109	4127	2	38	29.68	32.13
MOLHIV	41127	2	9	25.5	27.5
ZINC	12000	-	28	23.16	49.83
The cubic datasets (Cubic08, Cubic10, and Cubic12) comprise non-isomorphic 3-regular graphs with 8, 10, and 12 vertices, respectively. These datasets contain 5 (Cubic08), 19 (Cubic10), and 85 (Cubic12) graphs and can be downloaded at https://houseofgraphs.org/metamodely/cubic. For each dataset, we create a balanced graph classification problem by randomly assigning each graph a binary class. Also, since the graphs do not have node features, we add a scalar feature to each vertex, i.e.,  $c(v) = 1$  for all  $v$ . However, this would make 1WL-GNNs and PH fail to distinguish any pair of graphs. Thus, we change the features of some arbitrary vertices of each graph, making  $c(v) = -1$  for 1 vertex in graphs from Cubic08, 2 vertices in Cubic10, and 3 vertices in Cubic12 — we denote the resulting datasets as Cubic08-1, Cubic10-2, and Cubic12-3. Given the modified datasets, we aim to assess if the existing methods can overfit (correctly classify all) the samples.
C.2 Models
We implement all models using the PyTorch Geometric Library [10].
Synthetic data. The GNN architecture consists of a GCN with 2 convolutional layers followed by a sum readout layer and an MLP (one hidden layer) with ReLU activation. The resulting architecture is:  $\mathrm{Conv}(1,36)\rightarrow \mathrm{Conv}(36,16)\rightarrow \mathrm{sum - readout}\rightarrow \mathrm{BN}(16)\rightarrow \mathrm{MLP}(16,24,1)$ , where BN denotes a batch norm layer [18]. For the PH model, we consider standard vertex-color filtration functions. In particular, we apply the same procedure as Hofer et al. [13], Horn et al. [15] to compute the persistence tuples. We only consider 0-dim persistence diagrams. The filtration function consists of an MLP with 8 hidden units and ReLU activation followed by a component-wise sigmoid function: Sigmoid(MLP(1,8,4)) — i.e., we use 4 filtration functions with shared parameters. Since we can associate persistence tuples with vertices, we concatenate the resulting diagrams to obtain a  $|V|\times (4*2)$  matrix  $[\mathcal{D}_1^0,\mathcal{D}_2^0,\mathcal{D}_3^0,\mathcal{D}_4^0]$ , where  $\mathcal{D}_i^0$  denotes the 0-dim diagram obtained using the  $i$ -th filtration function. This procedure was also employed by Horn et al. [15]. The obtained diagrams are processed using a DeepSet layer with mean aggregator and internal MLP function  $(\Psi)$  with 16 hidden and output units, MLP  $(4*2,16,16)$ . We then apply a linear layer on top of the aggregated features. The overall DeepSet architecture is: MLP  $(4*2,16,16)\rightarrow \mathrm{Mean}$  Aggregator  $\rightarrow$  Linear(16, 16). Finally, we obtain class predictions using BatchNorm followed by a single-hidden-layer MLP with 16 hidden units: BN  $(16)\rightarrow \mathrm{MLP}(16,16,1)$ .
RePHINE uses the same overall architecture as the PH model. The only differences are that i) RePHINE tuples are 3-dimensional (as opposed to 2-dimensional in PH), and ii) RePHINE addi-
grams given by  $\mathcal{R}_G = \{(0,4,1,4),(0,\infty ,1,3),(0,3,2,3),(0,3,2,3)\}$  and  $\mathcal{R}_{G^{\prime}} = \{(0,\infty ,1,3),(0,4,1,3),(0,3,2,3),(0,3,2,3)\}$ .
$^{2}$ https://chrsmrrs.github.io/datasets/
<sup>3</sup>https://ogb.stanford.edu
tionally leverages an edge-level filtration function. Such a function follows the architecture of the vertex-level one, i.e., Sigmoid(MLP(1, 8, 4)). We note that RePHINE tuples are 3-dimensional instead of 4-dimensional because we removed their uninformative first component (equal to 0) since we only use 0-dim diagrams. In other words, we do not leverage missing holes.
Regarding the training, all models follow the same setting: we apply the Adam optimizer [21] for 2000 epochs with an initial learning rate of  $10^{-4}$  that is decreased by half every 400 epochs. We use batches of sizes 5, 8, 32 for the cubic08, cubic10, and cubic12 datasets, respectively. All results are averaged over 5 independent runs (different seeds). For all models, we obtain the expressivity metric by computing the uniqueness of graph-level representations extracted before the final MLP, with a precision of 5 decimals. Importantly, these choices of hyperparameters ensure that all models have a similar number of learned parameters: 1177 (RePHINE), 1061 (PH), and 1129 (GCN).
Real-world data. For computing the standard vertex-color persistence diagrams, we use the code available by Horn et al. [15], which consists of a parallel implementation in PyTorch of the pseudocode in Algorithm 1. Moreover, we apply a multiple filtration scheme and concatenate the 0-dim persistence diagrams to form matrix representations — again similarly to the design in [15]. Then, we apply a DeepSet architecture of the form: MLP(TupleSize * nFiltrations, OutDim, OutDim)  $\rightarrow$  Mean Aggregator  $\rightarrow$  Linear(OutDim, OutDim). We use MLPs to define vertex- and edge-level filtration functions. For the 1-dimensional persistence tuples (or missing holes), we first process the tuples from each filtration function using a shared DeepSet layer and then apply mean pooling to obtain graph-level representations — this avoids possibly breaking isomorphism invariance by concatenating 1-dimensional diagrams. We sum the 0- and 1-dim embeddings and send the resulting vector to an MLP head. The resulting topological embeddings are concatenated with last-layer GNN embeddings and fed into a final MLP classifier.
We carry out grid-search for model selection. More specifically, we consider a grid comprised of a combination of  $\{2,3\}$  GNN layers and  $\{2,4,8\}$  filtration functions. We set the number of hidden units in the DeepSet and GNN layers to 64, and of the filtration functions to 16 — i.e., the vertex/edge-color filtration functions consist of a 2-layer MLP with 16 hidden units. For the largest datasets (ZINC and MOLHIV), we only use two GNN layers. The GNN node embeddings are combined using a global mean pooling layer. Importantly, for all datasets, we use the same architecture for RePHINE and color-based persistence diagrams.
For the TUDatasets, we obtain a random  $80\% / 10\% / 10\%$  (train/val/test) split, which is kept identical across five runs. The ZINC and MOLHIV datasets have public splits. All models are initialized with a learning rate of  $10^{-3}$  that is halved if the validation loss does not improve over 10 epochs. We apply early stopping with patience equal to 40.
Comparison to PersLay. We followed the guidelines in the official code repository regarding the choice of hyper-parameters. In particular, PersLay applies fixed filtration functions obtained from heat kernel signatures of the graphs with different parameters, resulting in extended and ordinary diagrams for 0 and 1-dimensional topological features. For RePHINE+Linear, we carry out a simple model selection procedure using grid-search for the number of filtration functions (\{4,8\}) and the number of hidden units (\{16,64\}) in the DeepSet models.
Hardware. For all experiments, we use Tesla V100 GPU cards and consider a memory budget of 32GB of RAM.
C.3 Computing RePHINE diagrams
Algorithm 2 describes the computation of RePHINE diagrams. The pseudocode has been written for clarity, not efficiency. The replacement for  $\infty$  in real holes depends on the choice of edge and vertex filter functions. In all experiments, we employed the logistic function to the output of the feedforward networks (i.e., filtered values lie in [0, 1]) and used 1 to denote the death time of real holes.
Here, we complement the experiments on synthetic data, providing illustrations of the learned persistence diagrams and reporting results obtained when we combine 0- and 1-dimensional diagrams.
In Figure S3, we show the concatenation of the learned persistence diagrams at the end of the training procedure for RePHINE and PH (i.e., standard vertex-color filtrations). In these examples, the RePHINE diagrams are different while the PH ones are identical. We can observe this behavior by
Algorithm 2 RePHINE   
Require:  $V,E$  ,eValues,vValues Vertices,edges,and edge/vertex-color filter values   
uf  $\leftarrow$  UNIONFIND(|V|)   
pers0  $\leftarrow$  zeros(|V|,4) Initialize the persistence tuples   
pers1  $\leftarrow$  zeros(|E|,4)   
pers0[:,3]  $\leftarrow$  vValues Pre-set the birth' times of each node SINDICES,SVALUES  $\leftarrow$  SORT(eValues)   
for e,weight  $\in$  Pair(SINDICES,SVALUES) do  $\triangleright$  Pair is equivalent to the zip function in Python (v,w)  $\leftarrow$  e if pers0[v,4] = 0 then Save the first filtration step that a node is discovered end if if pers0[w,4] = 0 then   
pers0[w,4]  $\leftarrow$  weight   
end if   
younger  $\leftarrow$  uf.find(v) younger denotes the component that will die older  $\leftarrow$  uf.find(w)   
if younger = older then A cycle was detected pers1[e,1]  $\leftarrow$  1 pers1[e,2]  $\leftarrow$  weight   
pers1[e,[3,4]]  $\leftarrow$ $\infty$  continue   
else if vValues[younger] = vValues[older] then if pers0[younger, 4] < pers0[older, 4] then Additional disambiguation step younger,older,v,w  $\leftarrow$  older,younger,w,v Flip younger,older,and node ids end if else if vValues[younger] < vValues[older] then younger,older,v,w  $\leftarrow$  older,younger,w,v   
end if   
end if   
pers0[younger,2]  $\leftarrow$  weight   
uf.merge(v,w) Merge two connected components   
end for   
for  $r\in$  uf.rootso do   
pers0[r,2]  $\leftarrow \infty$    
end for   
R  $\leftarrow$  JOIN(pers0,pers1)   
return R
carefully inspecting the multisets of vectors at each row of the concatenated tuples (each row of the plots in Figure S3). For instance, consider the diagrams in Figure S3(b): in the RePHINE diagram for the top graph, there is a row with 3 yellow entries which do not appear at the diagram for the bottom graph. However, the representations obtained from Standard PH are identical for these graphs.
In Figure 6 we reported results using only 0-dimensional topological features. For completeness, Figure S4 shows learning curves when exploiting both 0 and 1-dimensional diagrams. Overall, we can again observe that RePHINE produces higher expressivity and better fitting capability in comparison to vertex-color persistence diagrams.
(a) Cubic08-1
(b) Cubic10-2
(c) Cubic12-3
(d) Cubic10-2
(e) Cubic12-3
(f) Cubic12-3
Data  $(+1)$
Data (-1)
Figure S3: 0-dimensional diagrams obtained from RePHINE and PH (standard vertex-color filtrations). These represent pairs of graphs for which the learning procedure in RePHINE could yield different representations, whereas PH produced identical graph-level embeddings.
cub08-1
cub08-1
cub10-2
Figure S4: Average learning curves for RePHINE and PH on cubic graphs, using both 0-dim and 1-dim persistence diagrams. Again, RePHINE can better fit the graph samples.
cub12-3