SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $F_{1}(a) = F_{1}(b) = \square$ ,
-  $F_{2}(a) = F_{2}(d) = \square$ ,
We define the two relational structures as  $X_{1} = (U,\mathcal{N}_{1},\mathcal{N}_{2},F_{1})$  and  $X_{2} = (U,\mathcal{N}_{1},\mathcal{N}_{2},F_{2})$ . We define a color update function with non-pairwise aggregations over these relational structures:
$$
c _ {v} ^ {t + 1} = \mathrm {H A S H} \left(c _ {v} ^ {t}, \left\{\left\{c _ {w} ^ {t} \mid w \in \mathcal {N} _ {1} (v) \right\} \right\}, \left\{\left\{(c _ {x} ^ {t}, c _ {y} ^ {t}) \mid (x, y) \in \mathcal {N} _ {2} (v) \right\} \right\}\right),
$$
$$
\pi_ {1} = \operatorname {H A S H} (\triangle , \emptyset , \{\{\left(\square , \square\right) (\bigcirc , \bigcirc) \} \}),
$$
$$
\pi_ {2} = \operatorname {H A S H} (\triangle , \emptyset , \{\{(\square , \bigcirc), (\bigcirc , \square) \} \}).
$$
Obviously  $\pi_1 \neq \pi_2$ . Suppose there exists a structure-to-graph encoding  $R$  such that WL can distinguish the color of the vertex  $x$  corresponding to object  $x$  for the two graphs  $R(X_1), R(X_2)$ . We use  $\tau_1, \tau_2$  to denote the color of vertex  $x$  after one iteration of WL on  $R(X_1), R(X_2)$ , respectively. Since the color of  $x$  depends on  $a, b, c, d$ , we can assume that there are edges from these vertices to  $x$  in both graphs. Since  $R$  is a structure-to-graph encoding it makes use of the mappings  $R_{\mathrm{graph}}$  and  $R_{\mathrm{feat}}$ . By definition of  $R_{\mathrm{graph}}$  it has no access to the features thus the graph created for  $X_1, X_2$  by  $R_{\mathrm{graph}}$  are identical. Hence, the graphs  $R(X_1)$  and  $R(X_2)$  only differ by the features assigned to  $a, b, c, d$ . By definition of WL, the colors  $\tau_1, \tau_2$  only depend on the features of the neighbors of  $x$  and are independent from any other vertices (or adjacencies between other vertices). Let  $Y = y_1, \ldots, y_m$  be all neighbors of  $x$  that are not  $a, b, c, d$ . Note that the features of the vertices of  $Y$  is the same for both graphs as all vertices from  $Y$  are either not from  $U$  or they are assigned the same color by  $F_1$  and  $F_2$ . We add the relational structure  $X_i$  (for  $i \in \{1, 2\}$ ) as a subscript to the color notation to indicate where the color is originally coming from, e.g.,  $c_{x,X_1}^0$  is the initial color of vertex  $x$  in graph  $R(X_1)$ . By definition of WL it holds that:
$$
\tau_ {1} = \operatorname {H A S H} \left(c _ {x, X _ {1}} ^ {0}, \left\{\left\{c _ {a, X _ {1}} ^ {0}, c _ {b, X _ {1}} ^ {0}, c _ {c, X _ {1}} ^ {0}, c _ {d, X _ {1}} ^ {0} \right\} \right\} \cup \left\{\left\{c _ {y, X _ {1}} ^ {0} \mid y \in Y \right\} \right\}\right),
$$
As  $R_{\mathrm{graph}}$  assigns the same features to  $x$  and all vertices of  $Y$  for the two graphs and  $R_{\mathrm{feat}}$  concatenate the vertex features with the features from the structures, it follows that  $c_{x,X_1}^0 = c_{x,X_2}^0$  and  $\{\{c_{y,X_1}^0 \mid y \in Y\}\} = \{\{c_{y,X_2}^0 \mid y \in Y\}\}$ . As we intend to argue that there exists no  $R$  such that  $\tau_1 \neq \tau_2$  we intend to argue that
By definition the color of a vertex  $v$  from  $a, b, c, d$  in graph  $X_{i}$  from  $X_{1}, X_{2}$  where  $F_{\mathrm{graph}, i}(v)$  is the color assigned to  $v$  by  $R_{\mathrm{graph}}(X_{i})$ :
We prove the two multisets of color are equal by showing  $c_{a,X_1}^0 = c_{a,X_2}^0$ ,  $c_{b,X_1}^0 = c_{d,X_2}^0$ ,  $c_{c,X_1}^0 = c_{c,X_2}^0$  and  $c_{d,X_1}^0 = c_{b,X_2}^0$ . This follows from the observation that two vertices  $x \in V_{R(X_1)}$  and  $y \in R_{R(X_2)}$  are indistinguishable by  $R_{\mathrm{graph}}$  if they are in the same position of a tuple in  $\mathcal{N}_2$ . For example, both  $b$  and  $d$  are in position 2 of their respective tuple. This means that  $R_{\mathrm{graph}}$  has to assign the same features to both  $b$  and  $d$  implying  $F_{\mathrm{graph},1}(b) = F_{\mathrm{graph},2}(d)$ . As  $F_{1}(b) = F_{2}(d)$  it follows that  $c_{b,X_1}^0 = c_{d,X_2}^0$ . An analogous argument allows us to show the rest of the equalities which implies  $\tau_1 = \tau_2$ . This contradicts the initial assumption and shows that no structure-to-graph encoding  $R$  exists that allows for strong simulation of non-pairwise message passing on  $X_1$  and  $X_2$ . This proves the theorem.
E Proofs for Weak Simulation (Section 5)
E.1 Weak simulation of nested aggregations and non-pairwise message passing (Theorem 5.2)
We prove the correctness of Theorem 5.2.
For this we define the class of generalized augmented message passing (gAMP) color update functions. This class contains augmented message passing (AMP) but also non-pairwise message passing and nested aggregations.
Definition E.1. (Generalized Augmented Message Passing. Let  $X = (U, \mathcal{N}_1, \ldots, \mathcal{N}_k, F)$  be a relational structure. We call a color update function  $\phi$  generalized augmented message passing (gAMP) if for all  $u \in U$  and coloring  $c$  of  $X$ , it can recursively be defined as exactly one of the following five forms:
(W1)  $\phi(u, X, c) = \psi(u, (U, \mathcal{M}_1, \ldots, \mathcal{M}_r, F), c)$  where  $\psi$  is AMP and  $\{\mathcal{M}_1, \ldots, \mathcal{M}_r\}$  is a subset of  $\{\mathcal{N}_1, \ldots, \mathcal{N}_k\}$  only containing neighborhood functions  $\mathcal{N}$  where  $\ell(\mathcal{N}) = 1$ .
(W2)  $\phi(u, X, c) = \{(c_{u_1}, \ldots, c_{u_m}) \mid (u_1, \ldots, u_m) \in \mathcal{N}_i(u)\}$  where  $i \in [k]$ . This is called non-pairwise message passing.
(W3)  $\phi(u, X, c) = \{\{\psi(x, (U, \mathcal{M}_1, \ldots, \mathcal{M}_r, F), c) \mid x \in \mathcal{N}_i(u)\}\}$  where  $\psi$  is AMP that contains a neighborhood aggregation (S2),  $i \in [k]$  and  $\{\mathcal{M}_1, \ldots, \mathcal{M}_r\}$  is a subset of  $\{\mathcal{N}_1, \ldots, \mathcal{N}_k\}$  only containing neighborhood functions  $\mathcal{N}$  where  $\ell(\mathcal{N}) = 1$ . This construction is called a nested aggregation.
(W4)  $\phi(u, X, c) = (\phi_1(u, X, c), \ldots, \phi_n(u, X, c)$  where  $\phi_1, \ldots, \phi_n$  are gAMP.
(W5)  $\phi(u, X, c) = f(\psi(u, X, c))$  where  $f$  maps colors to colors and  $\psi$  is gAMP.
We want to prove that we can weakly simulate all coloring algorithms in this class. For this we define a structure-to-graph encoding in Algorithm 6 for every color update function in this class. We use the same encoding function ENC as defined in Appendix C. We assume that if a feature function is called with an vertex, object or edge that is assigned no features, it returns a fixed default color. We prove that combining Algorithm 6 with WL is at least as expressive as the corresponding non-standard message passing algorithm.
Proof. (Theorem 5.2) This proof works similarly to the proof of Theorem 3.5. We prove weak simulatability for  $\zeta = 2$ . Let  $\phi$  be an arbitrary generalized augmented message passing algorithm. We consider two arbitrary relational structures from the domain of  $\phi$ , say  $X = (U, \mathcal{N}_1, \dots, \mathcal{N}_k, F)$  and  $X' = (U', \mathcal{N}_1', \dots, \mathcal{N}_k', F')$ . Let  $(V_1, E_1, F_1) = R(X)$  and  $(V_2, E_2, F_2) = R(X')$ . We prove (1) that  $\mathrm{gAME}_{\phi}$  is a structure-to-graph encoding that runs in  $\mathcal{O}(\tau_{\phi}(|U|))$ ; (2) for every  $t \geq 0$  the coloring  $\mathrm{WL}^{2t}(R(X), F_1|_{V_1}) \cup \mathrm{WL}^{2t}(R(X'), F_2|_{V_2})$  on  $\mathrm{gAME}_{\phi}(X)$  refines the coloring  $\phi^t(X, F|_U) \cup \phi^t(X', F'|_{U'})$ .
We begin by proving (1). Observe that Algorithm 6 can be split into  $R_{\mathrm{graph}}$  and  $R_{\mathrm{feat}}$  as required by the definition of structure-to-graph encoding. Furthermore, Algorithm 6 first iterates through all gAMPs in  $\phi$  and all possible objects that are relevant to these gAMPS. Hence, Algorithm 6 runs in the same runtime as one iteration of gAMP which is in  $\mathcal{O}(\tau_{\phi}(|U|))$ .
Next, we prove (2). We use  $\pi_u^t,\pi_r^t$  and  $c_{u}^{t},c_{r}^{t}$  to denote the colors assigned to vertex / object  $u,r\in (U\cup U^{\prime})$  in iteration  $t$  by WL on  $R(X),R(X^{\prime})$  or  $\phi$  on  $X,X^{\prime}$ , respectively. Let  $u,r\in (U\cup U^{\prime})$  be arbitrary objects / vertices. We show by induction that if WL cannot distinguish  $u$  and  $r$  in iteration  $2t$  then  $\phi$  cannot distinguish them in iteration  $t$ . Formally, we show that  $\pi_u^{2t} = \pi_r^{2t}\Rightarrow c_u^t = c_r^t$  which implies that  $\mathrm{WL}^{2t}(R(X),F_1|_{V_1})\cup \mathrm{WL}^{2t}(R(X'),F_2|_{V_2})$  refines  $\phi^t (X,F|_U)\cup \phi^t (X',F'|_{U'})$  as  $(U\cup U^{\prime})\subseteq (V_{1}\cup V_{2})$  (by Algorithm 5).
Algorithm 6 starts by replacing all function applications (S4), (W5) by the identity function and flattens all tuples (A4), (S3), (W4). Recall that this transformation is either injective or bijective. By applying Lemma C.1 they do not influence the strong simulation property and thus also do not influence the weak simulation property. From now on we assume that  $\phi = (\phi_1, \ldots, \phi_n)$  such that all  $\phi_j$  with  $j \in [[n]]$  are either (W1) (which means they are one of (A1), (A2) (A3), (S2)), (W2), or (W3) and if they are an atom, then the first and third function argument are identical. We use  $w$  to denote the object the gAMP is defined to update. Algorithm 3 iterates through all  $\phi_1, \ldots, \phi_n$  and encodes them in the graph.
Base case: We show that  $\pi_u^0 = \pi_r^0 \Rightarrow c_u^0 = c_r^0$ . For this, we assume that  $\pi_u^0 = \pi_r^0$ . As the initial colors correspond to vertex or object features and Algorithm 6 encodes the object features in the vertex features it follows that  $c_u^0 = c_r^0$ .
Algorithm 5  $R_{\mathrm{graph}}$  for generalized augmented message encoding
Input: relational structure  $X = (U, \mathcal{N}_1, \dots, \mathcal{N}_k)$  without features, gAMP color update function  $\phi$  that has the shape  $(\phi_1, \dots, \phi_n)$  where each  $\phi_i$  with  $i \in [[n]]$  is one of (A1), (A2), (A3), (S2), (W2) or (W3).
Output: graph  $G = (V, E, F_{\mathrm{graph}})$
$$
V := W
$$
$$
E := \emptyset
$$
$F_{\mathrm{graph}} \coloneqq$  empty features
for  $i\in \llbracket n\rrbracket$  do
if  $\phi_{i}$  is (A1), (A2) or (A3) then
for  $u\in U$  do
$$
(V, E, F _ {\text {g r a p h}}) := \text {A l g o r i t h m} 1 (\phi_ {i}, (V, E, F _ {\text {g r a p h}}), u, u, i, 0, 0)
$$
else if  $\phi_i$  is (S2) which aggregates on neighborhood  $\mathcal{N}$  then
$(V,E,F_{\mathrm{graph}})\coloneqq \mathrm{Algorithm}2(\phi_i,(V,E,F_{\mathrm{graph}},(U,\mathcal{N}),i,0)$
else if  $\phi_{i}$  is (W2) then
$$
/ ^ {*} \phi_ {i} \text {h a s} \left\{\left(\left(c _ {u _ {1}} ^ {t}, \dots , c _ {u _ {m}} ^ {t}\right) \mid (u _ {1}, \dots , u _ {m}) \in \mathcal {N} _ {j} (u) \right\} \right\} \text {w h e r e} j \in [ k ] ^ {*} /
$$
for  $(u_{1},\ldots ,u_{m})\in \mathcal{N}_{j}(u)$  do
$$
V := V \cup \left\{u _ {\left(u _ {1}, \dots , u _ {m}\right)} \right\}
$$
$$
E := E \cup \left\{\left(u _ {u _ {1}, \dots , u _ {m}}, u\right) \right\}
$$
$$
F \left(u _ {\left(u _ {1}, \dots , u _ {m}\right)}\right) := \operatorname {E N C} (\text {e m p t y f e a t u r e s}, i, 0, 0, 1)
$$
$$
F \left(u _ {u _ {1}, \dots , u _ {m}}, u\right) := \operatorname {E N C} (\text {e m p t y f e a t u r e s}, i, 0, 0, 1)
$$
for  $r\in \llbracket m\rrbracket$  do
$$
E := E \cup \left\{\left(u _ {r}, u _ {u _ {1}, \dots , u _ {m}}\right) \right\}
$$
$$
F \left(u _ {r}, u _ {u _ {1}, \dots , u _ {m}}\right) := \operatorname {E N C} (\text {e m p t y f e a t u r e s}, i, r, 0, 1)
$$
else if  $\phi_{i}$  is (W3) then
$I^{*}\phi_{i}$  has shape  $\{\{\psi (x,(U,\mathcal{M}_1,\ldots \mathcal{M}_r,F),c)\mid x\in \mathcal{N}_j(u)\} \}$  where  $\psi$  is AMP  $^ { * } / ^ { * }$
Replace function applications (S4) in  $\psi$  by the identity function, i.e.,  $f(\varphi)$  becomes  $\varphi$ .
Flatten tuples (A4), (S3) in  $\psi$  except outermost, i.e.,  $(\ldots, (\varphi_1, \ldots, \varphi_r), \ldots)$  becomes  $(\ldots, \varphi_1, \ldots, \varphi_r, \ldots)$ .
$\mathit{/^{*}}$ $\psi$  has shape  $(\psi_{1},\dots ,\psi_{n})$  where each  $\psi_{i}$  with  $i\in \llbracket n\rrbracket$  is one of (A1), (A2), (A3) or (S2)  $^{\ast} / \cdot$
$(V,E,F_{\mathrm{graph}})\coloneqq \mathrm{Algorithm}3(\psi ,(V,E,F_{\mathrm{graph}}),(U,\mathcal{M}_1,\ldots \mathcal{M}_r),i)$
for  $x\in \mathcal{N}_j(u)$  do
$$
E := E \cup \{(x, u) \}
$$
$$
F (x, u) := \operatorname {E N C} (\text {e m p t y f e a t u r e s}, i, 0, 0, 1)
$$
return  $(V,E,F_{\mathrm{graph}})$
Induction hypothesis: We assume that  $\pi_u^{2t} = \pi_r^{2t}\Rightarrow c_u^t = c_r^t$  holds for all  $t\leq T$
Induction step: We show that  $\pi_u^{2(T + 1)} = \pi_r^{2(T + 1)}\Rightarrow c_u^{T + 1} = c_r^{T + 1}$ . We assume  $\pi_u^{2(T + 1)} = \pi_r^{2(T + 1)}$  which is equivalent to  $\pi_u^{2T + 2} = \pi_r^{2T + 2}$ . We prove for an arbitrary gAMP from  $\phi_1,\ldots ,\phi_n$  that it returns the same result for  $u$  and  $r$  which implies  $c_u^{T + 1} = c_r^{T + 1}$ . We do a case distinction on the type of this gAMP. Note that in gAMP and AMP the object / edge features are referred to us  $F(\cdot)$  whereas in the relational structures  $X$  and  $X^{\prime}$  they are denoted as  $F(\cdot)$  and  $F^{\prime}(\cdot)$ . Below, when we argue that two AMP constructions are equal and use the  $F(\cdot)$  notation and neglect  $F^{\prime}(\cdot)$  for the sake of simplicity.
Algorithm 6 Generalized augmented message encoding (gAME)
Input: relational structure  $X = (U,\mathcal{N}_1,\dots ,\mathcal{N}_k,F)$  gAMP color update function  $\phi$    
Output: graph  $G = (V,E,F^{\prime})$    
Replace function applications (W5), (S4) in  $\phi$  by the identity function, i.e.,  $f(\phi)$  becomes  $\phi$    
Flatten all (W4), (A4), (S3) constructions in  $\phi$  except the outermost, i.e.,  $(\ldots ,(\phi_1,\ldots ,\phi_n),\ldots)$  becomes  $(\ldots ,\phi_{1},\ldots ,\phi_{n},\ldots)$    
/\*  $\phi$  has the shape  $(\phi_1,\dots ,\phi_n)$  where each  $\phi_{i}$  with  $i\in [[n]]$  is one of (A1), (A2), (A3), (S2), (W2) or (W3) \*/   
 ${}^{*}R_{\mathrm{graph}}{}^{*}/$ $(V,E,F_{\mathrm{graph}}):= \mathrm{Algorithm~5}((U,\mathcal{N}_1,\dots ,\mathcal{N}_k),\phi)$ ${}^{*}R_{\mathrm{feat}}{}^{*}/$ $F^{\prime}\coloneqq$  empty features   
for  $v\in V$  do  
 $F^{\prime}(v)\coloneqq \mathrm{concat}\left(F(v),F_{\mathrm{graph}}(v)\right)$    
end for   
for  $e\in E$  do  
 $F^{\prime}(e)\coloneqq \mathrm{concat}\left(F(e),F_{\mathrm{graph}}(e)\right)$    
end for   
return  $(V,E,F^{\prime})$
(W1)  $\left(\phi_s(u,X,c)\stackrel {!}{=} \phi_s(r,X,c)\right)$ . Then,  $\phi_{s}$  is one of (A1), (A2), (A3), (S2). Observe that, then Algorithm 6 performs the same operations as Algorithm 4: for atoms (A1), (A2), (A3) it instantiates Algorithm 1 and for (S2) it instantiates Algorithm 2. We apply the induction step of the proof of Theorem 3.5 in a slightly altered form: instead of proving  $\pi_u^{T + 1} = \pi_r^{T + 1}\Rightarrow$ $c_{u}^{T + 1} = c_{r}^{T + 1}$  we instead prove  $\pi_u^{2T + 1} = \pi_r^{2T + 1}\Rightarrow c_u^T = c_r^T$ . Note, that by the induction hypothesis we already know the stronger fact  $\pi_u^{2T + 2} = \pi_r^{2T + 2}$ . However, we only use the fact  $\pi_u^{2T + 1} = \pi_r^{2T + 1}$  so that we can later apply this argument in a context where only the weaker  $\pi_u^{2T + 1} = \pi_r^{2T + 1}$  holds. We obtain a proof for  $\pi_u^{2T + 1} = \pi_r^{2T + 1}\Rightarrow c_u^T = c_r^T$  from the induction step in the proof of Theorem 3.5 by replacing  $\pi_t^T$  and  $\pi_r^T$  by  $\pi_r^{2T}$  and  $\pi_r^{2T + 1}$ , respectively. Thus, we know that for AMP  $\phi_{s}$  it holds that  $\phi_s(u,X,c) = \phi_s(r,X,c)$ .
(W2)  $\left(\left\{\left(c_{u_1}^T,\dots ,c_{u_m}^T\right)\mid (u_1,\dots ,u_m)\in \mathcal{N}_j(u)\right\} \right\} \stackrel {!}{=}\left\{\left(c_{r_1}^T,\dots ,c_{r_m}^T\right)\mid (r_1,\dots ,r_m)\in \mathcal{N}_j(r)\right\}$ . From  $\pi_u^{2T + 2} = \pi_r^{2T + 2}$  it follows that there exists a bijective function  $\alpha :\mathcal{N}_G(u)\to \mathcal{N}_G(r)$  that maps vertices assigned the same color by  $\pi^{2T + 1}$  to another. Note, that  $\mathcal{N}_G(u)$  is the set of all neighboring vertices of vertex  $u$  in the graph generated from  $X$  or  $X^{\prime}$ . By the construction of the edge features, it follows that  $\alpha$  maps vertices from  $\{u_{u_1,\ldots ,u_m}\mid (u_1,\ldots ,u_m)\in \mathcal{N}_j(u)\}$  to  $\{r_{r_1,\ldots ,r_m}\mid (r_1,\ldots ,r_m)\in \mathcal{N}_j(r)\}$ . Let  $\beta$  be the restriction of the function  $\alpha$  to these two sets as its domain and image, respectively. This implies that
$$
\left\{\left\{\pi_ {u _ {u _ {1}}, \dots , u _ {m}} ^ {2 T + 1} \mid (u _ {1}, \dots , u _ {m}) \in \mathcal {N} _ {j} (u) \right\} \right\} = \left\{\left\{\pi_ {r _ {r _ {1}}, \dots , r _ {m}} ^ {2 T + 1} \mid (r _ {1}, \dots , r _ {m}) \in \mathcal {N} _ {j} (r) \right\} \right\}.
$$
Let  $x \in \{u_{u_1, \ldots, u_m} \mid (u_1, \ldots, u_m) \in \mathcal{N}_j(u)\}$  be arbitrary with  $x = u_{u_1, \ldots, u_m}$ . Then, we call  $\beta(x) = r_{r_1, \ldots, r_m}$ . Note that all neighbors that send messages to  $x$  are  $u_1, \ldots, u_m$ . Similarly, all neighbors of  $\beta(x)$  that send messages to  $\beta(x)$  are  $r_1, \ldots, r_m$ . Since  $\pi_x^{2T+1} = \pi_{\beta(x)}^{2T+1}$  it holds that there exists a bijective function  $\gamma(x): \{u_1, \ldots, u_m\} \to \{r_1, \ldots, r_m\}$  mapping vertices assigned the same color by  $\pi^{2T}$  to each other. Note for every  $i \in [[m]]$  the edge connecting  $u_i$  to  $x$  is labeled with  $i$  and the same holds for  $r_i$  and  $\beta(x)$ . Thus,  $\gamma(u_i) = r_i$  which implies  $(u_1^{2T}, \ldots, u_m^{2T}) = (r_1^{2T}, \ldots, r_m^{2T})$ . From the induction hypothesis it holds that  $(c_{u_1}^T, \ldots, c_{u_m}^T) = (c_{r_1}^T, \ldots, c_{r_m}^T)$ . From the fact that  $\beta$  is a bijection and by iterating over all possible vertices for  $x$  we obtain
$$
\left\{\left\{\left(c _ {u _ {1}} ^ {T}, \dots , c _ {u _ {m}} ^ {T}\right) \mid (u _ {1}, \dots , u _ {m}) \in \mathcal {N} _ {j} (u) \right\} \right\} = \left\{\left\{\left(c _ {r _ {1}} ^ {T}, \dots , c _ {r _ {m}} ^ {T}\right) \mid (r _ {1}, \dots , r _ {m}) \in \mathcal {N} _ {j} (r) \right\} \right\}.
$$
(W3)  $\left(\left\{\theta(x, X', c^T) \mid x \in \mathcal{N}_j(u)\right\}\right) \stackrel {!}{=} \left\{\left\{\theta(x, X' c^T) \mid x \in \mathcal{N}_j(r)\right\}\right)$  where  $\theta$  is AMP,  $X' = (U, \mathcal{M}_1, \ldots, \mathcal{M}_r, F)$  and all neighborhoods of  $X'$  have  $\ell(\cdot) = 1$ . We use the function  $\alpha$  defined above. By definition all vertices from  $\mathcal{N}_j(u)$  and  $\mathcal{N}_j(r)$  are connected to  $u$  and  $r$  by an edge with a special edge label encoding  $j$ . Thus, there exists a restriction from a bijective  $\alpha$  to a bijective  $\beta: \mathcal{N}_j(u) \to \mathcal{N}_j(r)$  such that for every  $x \in \mathcal{N}_j(u)$  it holds that  $\pi_x^{2T+1} = \pi_{\beta(x)}^{2T+1}$ . Next, we can apply the above induction step proof for a gAMP that corresponds to AMP (W1). From this proof it follows that for every  $x \in \mathcal{N}_j(u)$  it holds that  $\theta(x, X', c^T) = \theta(\beta(x), X', c^T)$ . Together with the fact that  $\beta$  is bijective we obtain that
$$
\left(\left\{\left\{\theta \left(x, X ^ {\prime}, c ^ {T}\right) \mid x \in \mathcal {N} _ {j} (u) \right\} \right\} = \left\{\left\{\theta \left(x, X ^ {\prime}, c ^ {T}\right) \mid x \in \mathcal {N} _ {j} (r) \right\} \right\}\right)
$$
which contradicts the initial assumption.
E.2 List of Weakly Simulatable Algorithms (Corollary 5.3)
Next, we leverage Theorem 5.2 to prove Corollary 5.3.
Corollary E.2. The following GNNs and variants of WL can be weakly simulated:
Proof. Note that this has already been proven for Message Passing Simplicial Networks and CW Networks in Theorem F.1. We start by proving the corollary for DSS [Bevilacqua et al., 2021] by showing that its color update rule performs gAMP. We reuse the notation from the proof of Corollary 3.6. The update rule for DSS-WL can be written as
$$
c _ {v, S} ^ {t + 1} = \operatorname {H A S H} \left(c _ {v, S} ^ {t}, \left\{\left\{c _ {x, S} ^ {t} \mid x \in \mathcal {N} _ {S} (v) \right\} \right\}, C _ {v} ^ {t}, \left\{\left\{C _ {x} ^ {t} \mid x \in \mathcal {N} _ {G} (v) \right\} \right\}\right)
$$
where  $\mathcal{N}_S(v)$  is the set of all neighbors of vertex  $v$  in subgraph  $S$ ,  $\mathcal{N}_G(v)$  is the set of all neighbors of  $v$  in the original graph, and  $C_v^t$  contains the colors of  $v$  across different subgraphs
$$
C _ {v} ^ {t} = \left\{\left\{c _ {v, S ^ {\prime}} ^ {t} \mid S ^ {\prime} \in \pi (G) \text {a n d} v \in V \left(S ^ {\prime}\right) \right\} \right\}.
$$
We define the set of objects  $W$  as the disjoint union of all vertices in subgraphs  $W = \bigcup_{S\in \pi (G)}V(S)$  and identify each object from  $W$  by its original vertex and the subgraph it created. The update rule applies a HASH function (W5) to a tuple (W4) of other gAMPs. The first element of the tuple is  $c_{v,S}^{t}$  which is a color atom (A3). The second element is an aggregation over neighbors (S2). The third element can also be thought as an aggregation over neighbors (S2)  $C_v^t = \{\{c_{v,S'}^t\mid S'\in \pi (G)$  and  $v\in V(S')\} \} = \{\{c_{v,S'}^t\mid S'\in \mathcal{N}_S^r (v,S)\} \}$  where  $\mathcal{N}_S^{\prime}(v,S) = \{S^{\prime}\mid S^{\prime}\in \pi (G)$  and  $v\in V(S^{\prime})\}$ . Finally, the fourth element of the tuple is a nested aggregation (W3) that aggregates over the AMP defined in the previous sentence (with  $x$  instead of  $v$ ) for all  $x\in \mathcal{N}_G(v)$ . Thus, the color update function performs gAMP and DSS can be weakly simulated.
There is some more subtlety to this. The notation  $\mathcal{N}_S$  implies a separate neighborhood function  $\mathcal{N}_S$  for each subgraph  $S$  which would give an unbounded number of different neighborhood functions which does not directly work with our definitions. Instead, we need to create a new neighborhood function  $\mathcal{N}_1((v,s)) = \{x\mid x\in \mathcal{N}_S(v)\}$ . The same holds for the following definition of  $\mathcal{N}_S'$ .
Next, we prove the corollary for KP-GNNs [Feng et al., 2022],  $K$ -hop message passing [Feng et al., 2022] will follow as a special case of this. The color update function of KP-GNNs is defined as
$$
m _ {v} ^ {l + 1, k} = \mathbf {M E S} _ {k} ^ {l, \text {n o r m a l}} \left(\left\{\left\{(h _ {u} ^ {l}, e _ {u v}) \mid u \in Q _ {v, G} ^ {k, t} \right\} \right\}\right) + f (G _ {v, G} ^ {k, t}),
$$
$$
f \left(G _ {v, G} ^ {k, t}\right) = \operatorname {E M B} \left(\left(E \left(Q _ {v, G} ^ {k, t}\right), C _ {k} ^ {k ^ {\prime}}\right)\right),
$$
$$
h _ {v} ^ {l + 1, k} = \mathrm {U P D} _ {k} ^ {l} \left(m _ {v} ^ {l + 1, k}, h _ {v} ^ {l}\right),
$$
$$
h _ {v} ^ {l + 1} = \operatorname {C O M B I N E} ^ {l} \left(\left\{\left\{h _ {v} ^ {l + 1, k} \mid k = 1, 2, \dots , K \right\} \right\}\right).
$$
Here,  $t \in \{\gcd, \operatorname{spd}\}$  with  $\gcd =$  graph diffusion and  $\operatorname{spd} =$  shortest path distance. The set  $Q_{v,G}^{k,\gcd}$  contains all nodes from  $V(G)$  that can be reached within  $k$ -steps of random walk diffusion steps from  $v$ . The set  $Q_{v,G}^{k,\operatorname{spd}}$  contains all nodes from  $V(G)$  whose shortest-path to node  $v$  is exactly  $k$ .
Let  $W = V(G) \cup \{v_{k} \mid v \in V(G), k \in [[K]]\}$  be a set of objects with associated features  $F$ . We define three neighborhood functions  $\mathcal{N}_1, \mathcal{N}_2, \mathcal{N}_3$ :
1.  $\mathcal{N}_1(v) = \{v_k\mid k\in [[K]]\}$
2.  $\mathcal{N}_2(v_k) = \{v\}$ ,
3.  $\mathcal{N}_3(v_k) = Q_{v,G}^{k,t}$ .