SlowGuess/NeurIPS_2023_OCR huggingface dataset:
B Related Work: Simulating Message Passing
We investigate previous approaches to simulating GNNs with MPNNs on transformed graphs by analyzing papers and source code of different architectures. We begin with algorithms from Corollary 3.6. GSN-e / GSN-v [Bouritsas et al., 2022] and DS-WL / DS-GN [Bevilacqua et al., 2021] were not implemented via graph transformations. In practice however, there is little difference between their implementation of DS-WL / DS-GN and our implementation with graph transformations: Bevilacqua et al. [2021] implement subgraphs as separate graphs while we implement them as separate subgraphs of the same graphs. For  $\mathrm{VV}_C$ -GNNs [Sato et al., 2019], we could find no indication that they were implemented with graph transformations.  $M_{k}$ -GNNs [Papp and Wattenhofer, 2022] is a purely theoretic work and has not been implemented by the corresponding authors. For GMP [Wijesinghe and Wang, 2022], we are not sure if their code can count as being implemented as a graph transformation as they adapt message passing layers to use their structural information. However, for this they change the definition of the layers whereas we only change the graphs. Morris et al. [2019] introduced  $k$ -GNNs and implemented them with MPNNs on a transformed graph. Morris et al. [2019] even call this simulation and give a short proof that WL can simulate  $k$ -WL on transformed graphs. This corresponds to our concept of strong simulation, and this property of  $k$ -WL was already noticed by Otto [1997] [Grohe et al., 2021].
For  $\delta -k$ -(L)GNN, Morris et al. [2020] write that they implemented  $\delta -k$ -(L)GNNs
[...] using PYTORCH GEOMETRIC, using a Python-wrapped C++11 preprocessing routine to compute the computational graphs for the higher-order GNN. We used the GIN-ε layer to express  $f_{\mathrm{mrg}}^{W_1}$  and  $f_{\mathrm{aggr}}^{W_2}$  of [...].
An inspection of their code reveals that they create a similar graph structure as we propose in the proof of Corollary 3.6. For  $\delta$ - $k$ -GNN they create a graph where the vertices represent the  $k$ -tuples. They add an edge between two nodes if the tuples are  $j$ -neighbors, with  $j \in [[k]]$ . Just like in our graph transformation the edge type encodes whether the tuples are local or global  $j$ -neighbors and the neighborhood  $j$ . For  $(k,s)$ -SpecqNet (or  $(k,s)$ -LWL), Morris et al. [2022] write that  $(k,s)$ -SpecqNets were implemented
[...] using PyTorch Geometric, using a Python-wrapped C++11 preprocessing routine to compute the computational graphs for the higher-order GNNs. We used the GIN-ε layer to express  $f_{\mathrm{mrg}}^{W_1}$  and  $f_{\mathrm{aggr}}^{W_2}$  of [...]
Similar to our graph transformation from Corollary 3.6, they create a new graph where each node corresponds to an ordered subgraph  $\mathbf{v} \in V_G^k$  of size  $k$  with at most  $s$  connected components in this subgraph. Just like in our graph transformation, edges are added between nodes if their tuples only differ at position  $j \in [[k]]$  and the nodes at position  $j$  are neighbors. Additionally, the edge features in the transformed graph encode the edge features from the original graph in a way that allows them to encode  $j$ . Thus, their graph transformation follows ours. Both Morris et al. [2020] and Morris et al. [2022] reference Morris et al. [2019] to argue that there exist instantiations powerful enough for  $f_{\mathrm{mrg}}^{W_1}$  and  $f_{\mathrm{aggr}}^{W_2}$ . For  $k$ -OSWL / OSAN, Qian et al. [2022] state in their paper
[...] the computation of  $k$ -OSWL's coloring relies on the simple and easy-to-implement 1-WL.
It follows that the authors of these four papers were aware of simulation. However, to the best of our knowledge they have never formalized or fully explained their idea. In contrast, we fully formalize the idea of simulation and provide proofs that show that these graph transformations provably simulate the corresponding GNNs. Additionally, we provide recipes for obtaining graph transformations for augmented message passing algorithms.
To the best of our knowledge, so far Simplicial Message Passing Networks [Bodnar et al., 2021b], CWN [Bodnar et al., 2021a], DSS-GNN / WL [Bevilacqua et al., 2021],  $K$ -hop message passing and KP-GNNs [Feng et al., 2022] have not been implemented using weak simulation. However, another topological deep learning network the combinatorial complex neural network [Hajij et al., 2023] has been implemented based on GNNs. For this, the authors use a transformation to an augmented Hasse graph by transforming cells to vertices and adding edges between vertices corresponding to adjacent cells. This is similar to  $\overline{\mathrm{CWN}}$ , our proposed transformation from regular cell complexes to graph. Specifically, Hajij et al. [2023] state the following about their implementation TopoEmbedX:
TopoEmbedX [...] supports higher-order representation learning on cell complexes, simplicial complexes, and CCs. [...] TopoEmbedX converts a given higher-order domain into a subgraph of the corresponding augmented Hasse graph, and then utilizes existing graph representation learning algorithms to compute the embedding of elements of this subgraph.
Finally, Velicković [2022] proposes that all functions over graphs that are of interest can be computed with message passing over a transformed graph. They call this augmented message passing which inspired our use of this term. They discuss how to perform some particular variants of non-standard message passing through standard message passing. This is helpful and demonstrates the wide applicability of this approach. However, their work is intended as a positional paper and thus mostly stays high level and does not provide sufficient formal proofs to back up their claims. Providing the necessary proofs is non-trivial: once one understands simulation it is straightforward to apply it to a large number of architectures and it is simple to prove simulatability for any one given architecture. However, difficulty arises when one tries to prove this for a large number of architectures. Thus, the difficulty is not proving simulatability but defining a framework that allows us to prove this for many different architectures. We provide such a framework based on color update functions. Furthermore, Velicković [2022] does not provide a formal definition of simulation (or in their case augmented message passing). Indeed, it is trivially possible to build maximally expressive MPNNs by precomputing the isomorphism class of the input graph. Thus, without a restriction on the transformation, the general statement that a GNN can be implemented as an MPNN plus a transformation is only meaningful if the transformation is in some sense close to what the GNN does. We solve this issue through our careful definition of structure-to-graph encoding which restrict how the transformation can access features of the structure and through the definition of weak / strong simulation that pose a runtime restriction on the transformation.
C Proofs for Strong Simulation (Section 3)
C.1 Rewriting AMPs (Lemma C.1)
We use the following Lemma that allows us to rewrite AMPs without losing the strong simulation property. For a function  $\phi$  we use  $\mathrm{IM}[\phi]$  to denote the image of  $\phi$ .
Lemma C.1. Let  $\phi$  be a color update function. We are given a set of objects with neighborhood functions from the domain of  $\phi$  and colors. Let  $\Gamma(\phi)$  be a color update functions such that there exists an injective (bijective) mapping  $A$  between  $IM[\phi]$  and  $IM[\Gamma[\phi]]$ . Then, a coloring obtained from  $\Gamma(\phi)$  refines a coloring obtained from  $\phi$ . Furthermore, if the mapping  $A$  is bijective the coloring from  $\phi$  refines the coloring from  $\Gamma(\phi)$ .
This lemma allows us to simplify AMPs by replacing functions (S4) with the identity function i.e.  $f(\phi)$  is replaced by  $\phi$ . This transformation is bijective if  $f$  is injective. Furthermore, we can use this to flatten tuples (A4) (S3) i.e.  $(\ldots, (\phi_1, \ldots, \phi_n), \ldots)$  is replaced by  $(\ldots, \phi_1, \ldots, \phi_n, \ldots)$ . Flattening tuples with the same structure is bijective. Thus, we can apply these transformations without losing the simulatable property. We prove the correctness of the above lemma.
Proof. Let  $A$  be the injective mapping from the color assigned to objects from  $\phi$  to  $\Gamma(\phi)$ . For an arbitrary iteration, let  $\pi_w, \pi_p$  be the colors assigned to  $w \in W, p \in P$  by  $\Gamma(\phi)$  and  $c_w, c_p$  the colors assigned to them by  $\phi$ . We start by showing that if there exists the required injective mapping, then the coloring  $\pi$  from  $\Gamma(\phi)$  refines the coloring  $c$  from  $\phi$ . For this, we assume that  $\phi$  can distinguish  $w, p$  and show that this implies that  $\Gamma(\phi)$  can also distinguish them. Formally, we show  $c_w \neq c_p \Rightarrow \pi_w \neq \pi_p$ , this is the contraposition of the definition of refinement. Suppose  $c_w \neq c_p$ , then from  $\pi_w = A(c_w)$  and  $\pi_p = A(c_p)$  and the fact that  $A$  is injective follows that  $\pi_w \neq \pi_p$ .
Next, we show that if  $A$  is bijective then the colorings from  $\phi$  and  $\Gamma(\phi)$  refine each other. We assume that  $A$  is bijective. From the above argument, it then follows that the coloring from  $\Gamma(\phi)$  refines the coloring from  $\phi$ . It remains to show that the coloring from  $\phi$  refines that from  $\Gamma(\phi)$ . Similar to above, we show that if  $\Gamma(\phi)$  can distinguish  $w$  and  $p$ , then so can  $\phi$ . Thus, we assume  $\pi_w \neq \pi_p$  and show  $c_w \neq c_p$ . Using the bijectiveness of  $A$  it follows from our assumption that  $A^{-1}(\pi_w) \neq A^{-1}(\pi_p)$  and thus that  $c_w \neq c_p$ . This concludes the proof.
Algorithm 1 Helper algorithm for atoms
Output: updated graph  $G = (V,E,F_{\mathrm{graph}})$
Input: atom  $\rho$  of type (A1), (A2), (A3), graph  $(V, E, F_{\mathrm{graph}})$ , object  $w$ , object  $q$ , indices  $i, j, l$ .
if  $\rho$  is of type (A2) or (A3) then return  $(V,E,F_{\mathrm{graph}})$    
else  $\rho (w,c_w,q,c_q)$  is a constant  $k(w,q)$  if  $w$  is  $q$  then  $F_{\mathrm{graph}}(w)\coloneqq \mathrm{ENC}(F_{\mathrm{graph}}(w),i,j,l,k(w,w))$  else  $F_{\mathrm{graph}}((q,w))\coloneqq \mathrm{ENC}(F_{\mathrm{graph}}((q,w)),i,j,l,k(w,q))$  end if   
end if   
return  $(V,E,F_{\mathrm{graph}})$
Algorithm 2 Helper algorithm for (S2)
Output: updated graph  $G = (V,E,F_{\mathrm{graph}})$
Input: AMP (S2) of shape  $\{\{\rho(w, c_w^t, q, c_q^t) \mid q \in \mathcal{N}_s(w)\}\}$  where  $\rho$  is an atom and  $\ell(\mathcal{N}_s) = 1$ ; graph  $(V, E, F_{\mathrm{graph}})$ , relational structure  $(U, \mathcal{N}_s)$  without features, indices  $i, j$ .
for  $u\in U$  do   
for  $q\in \mathcal{N}_s(u)$  do   
 $e\coloneqq (q,u)$    
if  $e\notin E$  then   
 $E\coloneqq E\cup \{e\}$ $F_{\mathrm{graph}}(e) =$  empty vector   
end if   
 $F_{\mathrm{graph}}(e)\coloneqq \mathrm{ENC}(F_{\mathrm{graph}}(e),i,j,0,s)$    
end for   
end for   
/\*  $\rho$  has the shape  $(\rho_{1}(w,c_{w},\Omega_{1},c_{q}),\ldots ,\rho_{m}(w,c_{w},\Omega_{m},c_{q}))^{*} /$    
for  $l\in [m]$  do   
/\*  $\rho_l$  is either an (A1), (A2) or an (A3) \*/   
for  $u\in U$  do   
for  $q\in \mathcal{N}_s(u)$  do   
 $(V,E,F_{\mathrm{graph}}):= \mathrm{Algorithm~1}(\rho_l,(V,E,F_{\mathrm{graph}}),u,q,i,j,l)$    
end for   
end for   
end for   
return  $(V,E,F_{\mathrm{graph}})$
C.2 Augmented Message Passing can be Simulated (Theorem 3.5)
We are interested in showing the following theorem.
To prove the above theorem we define augmented message encoding (AME) in Algorithm 4. For this we assume a function  $\mathrm{ENC}$ :  $(x,i,j,l,c)\mapsto x^{\prime}$ , that updates a feature  $x$  to  $x^{\prime}$  by attaching the color  $c$  at the indices  $i,j,l$ . We assume that applying ENC multiple times never overwrites information if different indices are used. For example, we can think of the feature  $x$  as a 3d-tensor such that  $\mathrm{ENC}(x,i,j,l,c)$  updates this vector by writing the entry  $c$  at position  $i,j,l$ . Observe, that in Algorithm 4 it never happens that the same indices are used, thus whenever AME encodes information in the graph, this information does not get overwritten by latter operations. We prove the theorem by showing that  $\mathrm{AME + WL}$  can simulate any given AMP.
Proof. (Theorem 3.5) Let  $\phi$  be an arbitrary augmented message passing algorithm. We consider two arbitrary relational structures from the domain of  $\phi$ , say  $X = (U, \mathcal{N}_1, \ldots, \mathcal{N}_k, F)$  and  $X' = (U', \mathcal{N}_1', \ldots, \mathcal{N}_k', F')$ . Let  $(U, E_1, F_1) = R(X)$  and  $(U', E_2, F_2) = R(X')$ . We prove (1) that
Algorithm 3 Helper algorithm for AMP
Input: AMP color update function  $\phi = (\phi_1,\dots ,\phi_n)$  where each individual AMP is (A1), (A2), (A3) or (S2), graph  $(V,E,F_{\mathrm{graph}})$ , relational structure  $(U,\mathcal{N}_1,\ldots ,\mathcal{N}_k)$  without features where  $\ell (\mathcal{N}_s) = 1$  for all  $s\in [[k]]$ , index  $i$ .  
Output: updated graph  $G = (V,E,F_{\mathrm{graph}})$   
for  $j\in [[n]]$  do  
if  $\phi_j$  is (A1), (A2) or (A3) then  
for  $u\in U$  do  
 $(V,E,F_{\mathrm{graph}}):= \operatorname {Algorithm}1(\phi_j,(V,E,F_{\mathrm{graph}}),u,u,i,j,0)$   
end for  
else  
/\*  $\phi_i$  has the shape  $\{\{\rho (w,c_w,q,c_q)\mid q\in \mathcal{N}_s(w)\}\}$  where  $\rho$  is an atom and  $s\in [[k]]^{*}/$ $(V,E,F_{\mathrm{graph}}):= \operatorname {Algorithm}2(\phi_i,(V,E,F_{\mathrm{graph}}),(U,\mathcal{N}_s),i,j)$   
end if  
end for  
return  $(V,E,F_{\mathrm{graph}})$
Algorithm 4 Augmented message encoding (AME)
Input: relational structure  $X = (U,\mathcal{N}_1,\dots ,\mathcal{N}_k,F)$  where  $\ell (N_i) = 1$  for every  $i\in [[k]]$  , AMP color update function  $\phi$    
Output: graph  $G = (U,E,F^{\prime})$ ${}^{*}R_{\mathrm{graph}}{}^{*}/$ $E\coloneqq \emptyset$ $F_{\mathrm{graph}}\coloneqq$  empty features   
Replace function applications (S4) in  $\phi$  by the identity function, i.e.,  $f(\phi)$  becomes  $\phi$    
Flatten all (A4), (S3) constructions in  $\phi$  except the outermost, i.e.,  $(\ldots ,(\phi_1,\ldots ,\phi_n),\ldots)$  becomes  $(\ldots ,\phi_1,\ldots ,\phi_n,\ldots)$ ${}^{*}\phi$  has the shape  $(\phi_1,\dots ,\phi_n)$  where each  $\phi_{i}$  with  $i\in [[n]]$  is one of (A1), (A2), (A3) or (S2)  ${}^{*}/$ $(U,E,F_{\mathrm{graph}})\coloneqq \mathrm{Algorithm~3}(\phi ,(U,E,F_{\mathrm{graph}}),(U,\mathcal{N}_1,\dots ,\mathcal{N}_k),0)$ ${}^{*}R_{\mathrm{feat}}{}^{*}/$ $F^{\prime}\coloneqq$  empty features   
for  $u\in U$  do   
 $F^{\prime}(u)\coloneqq \mathrm{concat}\left(F(u),F_{\mathrm{graph}}(u)\right)$    
end for   
for  $e\in E$  do   
 $F^{\prime}(e)\coloneqq \mathrm{concat}\left(F(e),F_{\mathrm{graph}}(e)\right)$    
end for   
return  $(U,E,F^{\prime})$
$\mathrm{AME}_{\phi}$  is a structure-to-graph encoding such that  $R(X)$  runs in  $\mathcal{O}(\tau_{\phi}(|U|))$ ; (2) for every  $t \geq 0$  the coloring  $\mathrm{WL}^t(R(X), F_1|_U) \cup \mathrm{WL}^t(R(X', F_2|_{U'})$  on  $\mathrm{AME}_{\phi}(X)$  refines the coloring  $\phi^t(X, F|_U) \cup \phi^t(X', F'|_{U'})$ .
We begin by proving (1). Observe that Algorithm 4 can be split into  $R_{\mathrm{graph}}$  and  $R_{\mathrm{feat}}$  as required by the definition of structure-to-graph encoding. Furthermore, Algorithm 4 first iterates through all AMPs in  $\phi$  and all possible objects that are relevant to these AMPs. Hence, Algorithm 4 runs in the same runtime as one iteration of AMP which is in  $\mathcal{O}(\tau_{\phi}(|U|))$ .
Next, we prove (2). We use  $\pi_u^t,\pi_r^t$  and  $c_{u}^{t},c_{r}^{t}$  to denote the colors assigned to vertex / object  $u,r\in (U\cup U^{\prime})$  in iteration  $t$  by WL on  $R(X),R(X^{\prime})$  or  $\phi$  on  $X,X^{\prime}$ , respectively. We show by induction that if WL cannot distinguish  $u$  and  $r$  in a given iteration  $t$  then neither can  $\phi$ . Formally, we show  $\pi_u^t = \pi_r^t\Rightarrow c_u^t = c_r^t$  which implies that  $\mathrm{WL}^t (R(X),F_1|_U)\cup \mathrm{WL}^t (R(X'),F_2|_{U'})$  refines  $\phi^t (X,F|_U)\cup \phi^t (X',F'|_{U'})$  as the domain of the coloring computed by  $\phi$  is the same as the domain of the coloring computed by WL  $(U\cup U^{\prime})$ .
Algorithm 4 starts by replacing all function applications (S4) by the identity function and flattens all tuples (A4) (S3). Recall that this transformation is either injective or bijective. By applying Lemma C.1 they do not influence the strong simulation property. From now on we assume that  $\phi = (\phi_1, \ldots, \phi_n)$  such that all  $\phi_j$  with  $j \in [[n]]$  are either (A1), (A2) (A3), or (S2) and if they are an atom, then the first and third function argument are identical. We use  $w$  to denote the object the AMP is defined to update. Algorithm 3 iterates through all  $\phi_1, \ldots, \phi_n$  and encodes them in the graph.
Base case: We show that  $\pi_u^0 = \pi_r^0\Rightarrow c_u^0 = c_r^0$  . For this, we assume that  $\pi_u^0 = \pi_r^0$  . Since the initial colors of WL correspond to the vertex features which encode the object features of  $X$  and  $X^{\prime}$  that are used as the initial coloring of AMP, this implies that  $c_{u}^{0} = c_{r}^{0}$
Induction hypothesis: We assume that  $\pi_u^t = \pi_r^t\Rightarrow c_u^t = c_r^t$  holds for all  $t\leq T$
Induction step: We show that  $\pi_u^{T + 1} = \pi_r^{T + 1}\Rightarrow c_u^{T + 1} = c_r^{T + 1}$ . We assume  $\pi_u^{T + 1} = \pi_r^{T + 1}$ . We iterate through all AMPs from  $\phi_1,\ldots ,\phi_n$  and show that they must all return the same result for  $u$  and  $r$  which implies  $c_{u}^{T + 1} = c_{r}^{T + 1}$ . We do a case distinction on the type of this AMP. Note that in AMP the object / edge features are referred to us  $F(\cdot)$  whereas in the relational structures  $X$  and  $X^{\prime}$  they are denoted as  $F(\cdot)$  and  $F^{\prime}(\cdot)$ . Below, when we argue that two AMP constructions are equal and use the  $F(\cdot)$  notation and neglect  $F^{\prime}(\cdot)$  for the sake of simplicity.
(A1)  $\left(k(u,u)\stackrel {!}{=}k(r,r)\right)$ . This atom was encoded into the graph by Algorithm 1. These constants are part of the vertex features of the graphs  $R(X),R(X^{\prime})$  and thus are part of the initial coloring  $\pi_u^0,\pi_r^0$ . Since WL refines colorings, the assumption that  $\pi_u^{T + 1} = \pi_r^{T + 1}$  implies  $\pi_u^0 = \pi_r^0$ . Thus,  $k(u,u) = k(r,r)$ .
(A2)  $\left(F(u)\stackrel {!}{=}F(r)\right)$ . As the object features are encoded into the vertex features of  $R(X),R(X^{\prime})$  (see  $R_{\mathrm{feat}}$  in Algorithm 4), they are part of the initial coloring  $\pi_u^0,\pi_r^0$ . Since WL refines colorings, the assumption that  $\pi_u^{T + 1} = \pi_r^{T + 1}$  implies  $\pi_u^0 = \pi_r^0$ . Thus,  $F(u) = F(r)$ .
(A3)  $\left(c_u^T\stackrel {!}{=}c_r^T\right)$ . This atom was not encoded into the graph. From our assumption we know that  $\pi_u^{T + 1} = \pi_r^{T + 1}$ . Since WL refines colorings this implies that  $\pi_u^T = \pi_r^T$ . By applying the induction hypothesis we obtain that  $c_{u}^{T} = c_{r}^{T}$ .
(S2)  $\left(\{\{\rho(u, c_u^T, x, c_x^T) \mid x \in \mathcal{N}_i(u)\}\} \stackrel{!}{=} \{\{\rho(r, c_r^T, y, c_y^T) \mid y \in \mathcal{N}_i(r)\}\}\right)$ . This AMP was encoded into the graph by Algorithm 2. Note that the neighborhood relation  $\mathcal{N}_i$  was encoded by edges with edge features labeled  $i$ , this means that if two nodes have the same color  $\pi_u^{T+1} = \pi_r^{T+1}$  then for every neighborhood relation  $\mathcal{N}_i$  there exists a bijective  $\alpha: \mathcal{N}_i(u) \to \mathcal{N}_i(r)$  such that for all  $x \in \mathcal{N}_i(u)$  it holds that  $\pi_x^T = \pi_{\alpha(x)}^T$  and that the edges  $(x,u)$ ,  $(\alpha(x), r)$  have the same edge features. From the induction hypothesis it then follows that  $c_x^T = c_{\alpha(x)}^T$ . Let  $\rho(\cdot, c_\cdot^T, \Omega, c_\Omega^T) = (\rho_1(\cdot, c_\cdot^T, \Omega_1, c_\Omega^T), \ldots, \rho_m(\cdot, c_\cdot^T, \Omega_m, c_{\Omega_m}^T))$  for some  $m \geq 1$ . We show for an arbitrary  $j \in [m]$  that  $\rho_j(u, c_u^T, \Omega_j, c_{\Omega_j}^T) = \rho_j(r, c_r^T, \Omega_j', c_{\Omega_j'}^T)$  where either  $\Omega_j = u, \Omega_j' = r$ , or  $\Omega_j = x, \Omega_j' = \alpha(x)$ . We do a case distinction on the type of  $\rho_j$ .
(A1)  $\rho_{j} = k(a,b)$ . We do a case distinction whether  $a = b$ :
* Case  $a = b$ . By assumption we need to show  $k(u, u) \stackrel{!}{=} k(r, r)$ . Recall that the vertex features  $F_{\mu}(u), F_{\nu}(r)$  encode  $k(u, u)$  and  $k(r, r)$  where  $\mu, \nu \in \{1, 2\}$  (see Algorithm 1), respectively. Hence, the initial colors  $\pi_u^0$  and  $\pi_r^0$  also encode  $k(u, u)$  and  $k(r, r)$ . As WL refines colorings and  $\pi_u^T = \pi_r^T$  it holds that  $\pi_u^0 = \pi_r^0$  which implies that  $k(u, u) = k(r, r)$ .
* Case  $a \neq b$ . Then  $(b, a)$  forms an edge meaning we need to show that  $k(u, x) \stackrel{!}{=} k(r, \alpha(x))$ . Recall that the feature  $F_{\mu}((x, u))$  encodes  $k(u, x)$  and that feature  $F_{\nu}((\alpha(x), r))$  encodes  $k(r, \alpha(x))$  where,  $\mu, \nu \in \{1, 2\}$  (Algorithm 1). From the definition of  $\alpha$  it follows that the edges  $(x, u)$  and  $(\alpha(x), r)$  have the same features and thus that  $k(u, x) = k(r, \alpha(x))$ .
(A2) We do a case distinction:
* Case  $F(u) \stackrel{!}{=} F(r)$ . See proof of (A2) above.
* Case  $F(x) \stackrel {!}{=} F(\alpha(x))$ . We know that  $\pi_x^T = \pi_{\alpha(x)}^T$ . Recall that the object features are encoded in the vertex features and thus in the initial coloring  $\pi_x^0$ . Since WL refines colorings, it follows from  $\pi_x^T = \pi_{\alpha(x)}^T$  that  $\pi_x^0 = \pi_{\alpha(x)}^0$  which implies  $F(x) = F(\alpha(x))$ .
* Case  $F((x, u)) \stackrel {!}{=} F((\alpha(x), r))$ . As argued above we know that the edges  $(x, u)$  and  $(\alpha(x), r)$  have the same features which implies  $F((x, u)) = F((\alpha(x), r))$ .
(A3)  $\rho_{j}\stackrel {!}{=}c_{\sigma}^{T}$  . We do a case distinction on  $\sigma$  ..
* Case  $\sigma$  is the vertex to update i.e.,  $u$  and  $r$ . Then, we need to show that  $c_u^T \stackrel{!}{=} c_r^T$ . We know that  $\pi_u^T = \pi_r^T$  and by the induction hypothesis it follows that  $c_u^T = c_r^T$ .
* Case  $\sigma$  is the neighboring vertex i.e.,  $x$  and  $\alpha(x)$ . Then, we need to show that  $c_x^T \stackrel {!}{=} c_{\alpha(x)}^T$ . We combine  $\pi_x^T = \pi_{\alpha(x)}^T$  with the induction hypothesis to obtain  $c_x^T = c_{\alpha(x)}^T$ .
Thus we conclude that  $\rho_{j}(u,c_{u}^{T},\Omega_{j},c_{x}^{T}) = \rho_{j}\left(r,c_{r}^{T},\Omega_{j}^{\prime},c_{\alpha (x)}^{T}\right)$  which means that  $\rho (u,c_u^T,x,c_x^T) = \rho \left(r,c_r^T,\alpha (x),c_{\alpha (x)}^T\right)$ . This concludes the proof of the induction step.
C.3 Strongly Simulating GNNs
We prove a theorem that allows us to simulate a combination of multiple AMPs that differ only by function applications (S4). This allows us to simulate GNNs whose layers correspond to AMPs.
Theorem C.2. Let  $\phi_1, \ldots, \phi_l$  be  $l \geq 1$  AMPs that only differ in function application (S4). Let  $R = AME_{\psi}$  be a structure-to-graph encoding where  $\psi$  is the AMP obtained by removing function applications from  $\phi_1$ . Then, it holds for every pair of relational structures  $X, X'$  from the domain of  $\phi_1, \ldots, \phi_l$  and every  $l \geq t \geq 1$  that  $t$  iterations of WL on  $R(X)$  and  $R(X')$  refines the coloring produced by  $(\phi_1 \circ \ldots \circ \phi_t)$  on  $X$  and  $X'$ .
Note that it does not matter whether we construct  $\psi$  by removing function applications from  $\phi_{1}$  or any other AMP from  $\phi_1,\ldots ,\phi_l$  as these AMP only differ by function application.