SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $G = (V, E, R, c, \eta)$  be a knowledge graph where  $\eta$  is a pairwise coloring. We denote by  $G^2$  the knowledge graph  $G^2 = (V \times V, E', R, c_{\eta})$  where  $E' = \{r((u, w), (u, v)) \mid r(w, v) \in E, r \in R\}$  and  $c_{\eta}$  is the node coloring  $c_{\eta}((u, v)) = \eta(u, v)$ . Note that the coloring  $c$  is irrelevant in the construction. Intuitively,  $G^2$  encodes the adjacency relation between pairs of nodes of  $G$  used in C-MPNNs. We stress that  $G$  and  $G^2$  have the same relation type set  $R$ . If  $\mathcal{A}$  is a C-MPNN and  $\mathcal{B}$  is an R-MPNN, we write  $h_{\mathcal{A}, G}^{(t)}(u, v) := h_q^{(t)}(u, v)$  and  $h_{\mathcal{B}, G^2}^{(t)}((u, v)) := h^{(t)}((u, v))$  for the features computed by  $\mathcal{A}$  and  $\mathcal{B}$  over  $G$  and  $G^2$ , respectively. We sometimes write  $\mathcal{N}_r^H(v)$  to emphasize that the neighborhood is taken over the knowledge graph  $H$ . Finally, we say that an initial feature map  $y$  for  $G^2$  satisfies target node distinguishability if  $y((u, u)) \neq y((u, v))$  for all  $u \neq v$ .
We have the following equivalence between C-MPNNs and R-MPNNs without global readout:
Proposition A.7. Let  $G = (V, E, R, \boldsymbol{x}, \eta)^3$  be a knowledge graph where  $\boldsymbol{x}$  is a feature map, and  $\eta$  is a pairwise coloring. Let  $q \in R$  be any query relation. Then:
1. For every  $C$ -MPNN without global readout  $\mathcal{A}$  with  $T$  layers, there is an initial feature map  $\mathbf{y}$  for  $G^2$  an  $R$ -MPNN without global readout  $\mathcal{B}$  with  $T$  layers such that for all  $0 \leq t \leq T$  and  $u, v \in V$ , we have  $\pmb{h}_{\mathcal{A},G}^{(t)}(u,v) = \pmb{h}_{\mathcal{B},G^2}^{(t)}((u,v))$ .
2. For every initial feature map  $\pmb{y}$  for  $G^2$  satisfying target node distinguishability and every  $R$ -MPNN without global readout  $\mathcal{B}$  with  $T$  layers, there is a  $C$ -MPNN without global readout  $\mathcal{A}$  with  $T$  layers such that for all  $0 \leq t \leq T$  and  $u, v \in V$ , we have  $\pmb{h}_{\mathcal{A}, G}^{(t)}(u, v) = \pmb{h}_{\mathcal{B}, G^2}^{(t)}((u, v))$ .
Proof. We start with item (1). The sought R-MPNN  $\mathcal{B}$  has the same history function and the same message, aggregation, and update functions as  $\mathcal{A}$  for all the  $T$  layers. The initial feature map  $\pmb{y}$  is defined as  $\pmb{y}((u,v)) = \mathrm{INIT}(u,v,q)$ , where  $\mathrm{INIT}$  is the initialization function of  $\mathcal{A}$ .
We show the equivalence by induction on  $t$ . For  $t = 0$ , we have  $h_{\mathcal{A}}^{(0)}(u,v) = \mathrm{INIT}(u,v,q) = \pmb{y}((u,v)) = \pmb{h}_{\mathcal{B}}^{(0)}((u,v))$ . For the inductive case, take  $u,v \in V$ . We have
$$
\begin{array}{l} \boldsymbol {h} _ {\mathcal {A}} ^ {(t + 1)} (u, v) = \operatorname {U P D} \left(\boldsymbol {h} _ {\mathcal {A}} ^ {(f (t))} (u, v), \operatorname {A G G} \left(\left\{\left\{\operatorname {M S G} _ {r} \left(\boldsymbol {h} _ {\mathcal {A}} ^ {(t)} (u, w), \boldsymbol {z} _ {q}\right) \right| w \in \mathcal {N} _ {r} ^ {G} (v), r \in R \right\}\right)\right) \\ = \operatorname {U P D} \left(\boldsymbol {h} _ {\mathcal {B}} ^ {(f (t))} ((u, v)), \operatorname {A G G} \left(\left\{\left. \left\{\operatorname {M S G} _ {r} \left(\boldsymbol {h} _ {\mathcal {B}} ^ {(t)} ((u, w)), \boldsymbol {z} _ {q}\right) \right| (u, w) \in \mathcal {N} _ {r} ^ {G ^ {2}} ((u, v)), r \in R \right\}\right)\right) \right. \\ = \boldsymbol {h} _ {\boldsymbol {B}} ^ {(t + 1)} ((u, v)). \\ \end{array}
$$
For item (2), we take  $\mathcal{A}$  to have the same history function and the same message, aggregation, and update functions than  $\mathcal{B}$ , for all the  $T$  layers, and initialization function  $\mathrm{INIT}$  such that  $\mathrm{INIT}(u,v,q) = y((u,v))$ . The argument for the equivalence is the same as item (1).
3The pairwise coloring  $\eta$  does not play any role in the proposition.
Regarding WL algorithms, we have a similar equivalence:
Proposition A.8. Let  $G = (V, E, R, c, \eta)$  be a knowledge graph where  $\eta$  is a pairwise coloring. For all  $t \geq 0$  and  $u, v \in V$ , we have that  $\mathsf{rawl}_2^{(t)}(u, v)$  computed over  $G$  coincides with  $\mathsf{rw} \mathsf{l}_1^{(t)}((u, v))$  computed over  $G^2 = (V \times V, E', R, c_{\eta})$ .
Proof. For  $t = 0$ , we have  $\mathsf{rawl}_2^{(0)}(u,v) = \eta (u,v) = c_\eta ((u,v)) = \mathsf{rw}l_1^{(0)}((u,v))$ . For the inductive case, we have
$$
\begin{array}{l} \left. \mathbf {r a w l} _ {2} ^ {(t + 1)} (u, v) = \tau \left(\mathbf {r a w l} _ {2} ^ {(t)} (u, v), \{\{\left(\mathbf {r a w l} _ {2} ^ {(t)} (u, w), r\right) \mid w \in \mathcal {N} _ {r} ^ {G} (v), r \in R) \} \}\right) \right. \\ = \tau \left(\mathsf {r w l} _ {1} ^ {(t)} ((u, v)), \{\{\left(\mathsf {r w l} _ {1} ^ {(t)} ((u, w)), r\right) \mid (u, w) \in \mathcal {N} _ {r} ^ {G ^ {2}} ((u, v)), r \in R) \} \}\right) \\ = r w l _ {1} ^ {(t + 1)} ((u, v)). \\ \end{array}
$$
Now we are ready to prove Theorem 5.1.
For  $G = (V, E, R, \boldsymbol{x}, \eta)$ , we consider  $G^2 = (V \times V, E', R, c_{\eta})$ . We start with item (1). Let  $\mathcal{A}$  be a C-MPNN without global readout with  $T$  layers and initialization INIT satisfying INIT ≡ η and let  $0 \leq t \leq T$ . Let  $\boldsymbol{y}$  be an initial feature map for  $G^2$  and  $\mathcal{B}$  be an R-MPNN without global readout with  $T$  layers as in Proposition A.7, item (1). Note that  $\boldsymbol{y} \equiv c_{\eta}$  as  $\boldsymbol{y}((u, v)) = \mathrm{INIT}(u, v, q)$ . We can apply Theorem A.1, item (1) to  $G^2$ ,  $\boldsymbol{y}$  and  $\mathcal{B}$  and conclude that  $\mathsf{rw}|_1^{(t)} \preceq h_{\mathcal{B},G^2}^{(t)}$ . This implies that  $\mathsf{raw}|_1^{(t)} \preceq h_{\mathcal{A},G}^{(t)}$ .
For item (2), let  $T \geq 0$  and  $f$  be a history function. We can apply Theorem A.1, item (2), to  $G^2$  to obtain an initial feature map  $\pmb{y}$  with  $\pmb{y} \equiv c_{\eta}$  and an R-MPNN without global readout  $\mathcal{B}$  with  $T$  layers and history function  $f$  such that for all  $0 \leq t \leq T$ , we have  $\mathrm{rw}|_1^{(t)} \equiv h_{\mathcal{B},G^2}^{(t)}$ . Note that  $\pmb{y}$  satisfies target node distinguishability since  $\eta$  does. Let  $\mathcal{A}$  be the C-MPNN without global readout obtained from Proposition A.7, item (2). We have that  $\mathrm{rw}|_2^{(t)} \equiv h_{\mathcal{A},G}^{(t)}$  as required.
Remark A.9. Note that item (2) holds for the basic model of C-MPNNs (without global readout), presented in Section 4.1, with the three proposed message functions  $\mathrm{M}\mathrm{s}\mathrm{G}_{r}^{1},\mathrm{M}\mathrm{s}\mathrm{G}_{r}^{2},\mathrm{M}\mathrm{s}\mathrm{G}_{r}^{3}$ , as they can express vector scaling.
Finally, note that the extension of item (1) in Theorem 5.1 to the case with global readouts is straightforward. Indeed, we can apply exactly the same argument we used for the case without global readout. As for two pairs of the form  $(u,v)$  and  $(u,v^{\prime})$  the global readout vectors are identical (to READ( $\{h_{w|u,q}^{(t)} \mid w \in V\}$ )), this argument still works.
The proof is a reduction to the one-dimensional case, that is, R-MPNNs without global readout and a logic denoted by  $\mathsf{rFO}_{\mathrm{cnt}}^2$ . So in order to show Theorem 5.2 we need some definitions and auxiliary results.
Fix a set of relation types  $R$  and a set of node colors  $\mathcal{C}$ . We consider knowledge graphs of the form  $G = (V, E, R, c)$  where  $c$  is a node coloring assigning colors from  $\mathcal{C}$ . In this context, logical formulas can refer to relation types from  $R$  and node colors from  $\mathcal{C}$ . Following Barceló et al. [3], a logical (node) classifier is a unary formula expressible in first-order logic (FO), classifying each node  $u$  on a knowledge graph  $G$  according to whether the formula holds or not for  $u$  over  $G$ .
We define a fragment  $\mathsf{rFO}_{\mathrm{cnt}}^2$  of FO as follows. A  $\mathsf{rFO}_{\mathrm{cnt}}^2$  formula is either  $a(x)$  for  $a \in \mathcal{C}$ , or one of the following, where  $\varphi$  and  $\psi$  are  $\mathsf{rFO}_{\mathrm{cnt}}^2$  formulas,  $N \geq 1$  is a positive integer and  $r \in R$ :
$$
\neg \varphi (x), \quad \varphi (x) \wedge \psi (x), \quad \exists^ {\geq N} y (\varphi (y) \wedge r (y, x)).
$$
We remark that  $\mathsf{rFO}_{\mathrm{cnt}}^2$  is actually the fragment of FO used in Barceló et al. [3] to characterize GNNs, adapted to multiple relations. It is well-known that  $\mathsf{rFO}_{\mathrm{cnt}}^2$  is equivalent to graded modal logic [8].
The following proposition provides useful translations from  $\mathsf{rFO}_{\mathrm{cnt}}^2$  to  $\mathsf{rFO}_{\mathrm{cnt}}^3$  and vice versa. Recall from Section A.2, that given a knowledge graph  $G = (V,E,R,\eta)$  where  $\eta$  is a pairwise coloring, we define the knowledge graph  $G^{2} = (V\times V,E^{\prime},R,c_{\eta})$  where  $E^{\prime} = \{r((u,w),(u,v))\mid r(w,v)\in E,r\in R\}$  and  $c_{\eta}$  is the node coloring  $c_{\eta}((u,v)) = \eta (u,v)$ .
Proposition A.10. We have the following:
1. For all  $\mathsf{rFO}_{\mathrm{cnt}}^3$  formula  $\varphi (x,y)$ , there is a formula  $\tilde{\varphi} (x)$  in  $\mathsf{rFO}_{\mathrm{cnt}}^2$  such that for all knowledge graph  $G = (V,E,R,\eta)$ , we have  $G,u,v\models \varphi$  if and only if  $G^{2},(u,v)\models \tilde{\varphi}$ .
2. For all formula  $\varphi(x)$  in  $\mathsf{rFO}_{\mathrm{cnt}}^2$ , there is a  $\mathsf{rFO}_{\mathrm{cnt}}^3$  formula  $\tilde{\varphi}(x,y)$  such that for all knowledge graph  $G = (V,E,R,\eta)$ , we have  $G,u,v \models \tilde{\varphi}$  if and only if  $G^2,(u,v) \models \varphi$ .
Proof. We start with item (1). We define  $\tilde{\varphi}(x)$  by induction on the formula  $\varphi(x, y)$ :
1. If  $\varphi (x,y) = a(x,y)$  for color  $a$ , then  $\tilde{\varphi} (x) = a(x)$ .
2. If  $\varphi (x,y) = \neg \psi (x,y)$ , then  $\tilde{\varphi} (x) = \neg \tilde{\psi} (x)$ .
3. If  $\varphi (x,y) = \varphi_{1}(x,y)\wedge \varphi_{2}(x,y)$ , then  $\tilde{\varphi} (x) = \tilde{\varphi}_{1}(x)\wedge \tilde{\varphi}_{2}(x)$ .
4. If  $\varphi(x, y) = \exists^{\geq N} z (\psi(x, z) \wedge r(z, y))$  then  $\tilde{\varphi}(x) = \exists^{\geq N} y (\tilde{\psi}(y) \wedge r(y, x))$ .
Fix  $G = (V, E, R, \eta)$  and  $G^2 = (V \times V, E', R, c_{\eta})$ . We show by induction on the formula  $\varphi$  that  $G, u, v \models \varphi$  if and only if  $G^2, (u, v) \models \tilde{\varphi}$ .
For the base case, that is, case (1) above, we have that  $\varphi(x,y) = a(x,y)$  and hence  $G, u, v \models \varphi$  iff  $\eta(u,v) = a$  iff  $c_{\eta}((u,v)) = a$  iff  $G^2$ ,  $(u,v) \models \tilde{\varphi}$ .
Now we consider the inductive case. For case (2) above, we have  $\varphi(x,y) = \neg \psi(x,y)$ . Then  $G, u, v \models \varphi$  iff  $G, u, v \not\models \psi$  iff  $G^2$ ,  $(u,v) \not\models \tilde{\psi}$  iff  $G^2$ ,  $(u,v) \models \tilde{\varphi}$ .
For case (3), we have  $\varphi(x,y) = \varphi_1(x,y) \wedge \varphi_2(x,y)$ . Then  $G, u, v \models \varphi$  iff  $G, u, v \models \varphi_1$  and  $G, u, v \models \varphi_2$  iff  $G^2$ ,  $(u,v) \models \tilde{\varphi}_1$  and  $G^2$ ,  $(u,v) \models \tilde{\varphi}_2$  iff  $G^2$ ,  $(u,v) \models \tilde{\varphi}$ .
Finally, for case (4), we have  $\varphi(x,y) = \exists^{\geq N}z(\psi(x,z) \wedge r(z,y))$ . Assume  $G, u, v \models \varphi$ , then there exist at least  $N$  nodes  $w \in V$  such that  $G, u, w \models \psi$  and  $r(w,v) \in E$ . By the definition of  $G^2$ , there exist at least  $N$  nodes in  $G^2$  of the form  $(u,w)$  such that  $G^2, (u,w) \models \tilde{\psi}$  and  $r((u,w), (u,v)) \in E'$ . It follows that  $G^2, (u,v) \models \tilde{\varphi}$ . On the other hand, suppose  $G^2, (u,v) \models \tilde{\varphi}$ . Then there exist at least  $N$  nodes  $(o,o')$  in  $G^2$  such that  $G, (o,o') \models \tilde{\psi}$  and  $r((o,o'), (u,v)) \in E'$ . By definition of  $G^2$  each  $(o,o')$  must be of the form  $(o,o') = (u,w)$  for some  $w \in V$  such that  $r(w,v)$ . Then there are at least  $N$  nodes  $w \in V$  such that  $G, u, w \models \psi$  and  $r(w,v) \in E$ . It follows that  $G, u, v \models \varphi$ .
Item (2) is similar. We define  $\tilde{\varphi}(x, y)$  by induction on the formula  $\varphi(x)$ :
1. If  $\varphi (x) = a(x)$  for color  $a$ , then  $\tilde{\varphi} (x,y) = a(x,y)$ .
2. If  $\varphi (x) = \neg \psi (x)$ , then  $\tilde{\varphi} (x,y) = \neg \tilde{\psi} (x,y)$ .
3. If  $\varphi (x) = \varphi_{1}(x)\wedge \varphi_{2}(x)$ , then  $\tilde{\varphi} (x,y) = \tilde{\varphi}_{1}(x,y)\wedge \tilde{\varphi}_{2}(x,y)$ .
4. If  $\varphi(x) = \exists^{\geq N} y (\psi(y) \wedge r(y, x))$  then  $\tilde{\varphi}(x, y) = \exists^{\geq N} z (\tilde{\psi}(x, z) \wedge r(z, y))$ .
Following the same inductive argument from item (1), we obtain that  $G, u, v \models \tilde{\varphi}$  if and only if  $G^2, (u, v) \models \varphi$ .
The following theorem is an adaptation of Theorem 4.2 from Barceló et al. [3]. The main difference with Barceló et al. [3] is that here we need to handle multiple relation types.
Theorem A.11. A logical classifier is captured by  $R$ -MPNNs without global readout if and only if it can be expressed in  $rFO_{\mathrm{cnt}}^2$ .
Proof. We start with the backward direction. Let  $\varphi(x)$  be a formula in  $\mathsf{rFO}_{\mathrm{cnt}}^2$  for relation types  $R$  and node colors  $\mathcal{C}$ . Let  $\varphi_1, \ldots, \varphi_L$  be an enumeration of the subformulas of  $\varphi$  such that if  $\varphi_i$  is a subformula of  $\varphi_j$ , then  $i \leq j$ . In particular,  $\varphi_L = \varphi$ . We shall define an R-MPNN without global readout  $\mathcal{B}_{\varphi}$  with  $L$  layers computing  $L$ -dimensional features in each layer. The idea is that at layer  $\ell \in \{1, \ldots, L\}$ , the  $\ell$ -th component of the feature  $h_v^{(\ell)}$  is computed correctly and corresponds to 1 if  $\varphi_\ell$  is satisfied in node  $v$ , and 0 otherwise. We add an additional final layer that simply outputs the last component of the feature vector.
We use models of R-MPNNs of the following form:
$$
\boldsymbol {h} _ {v} ^ {(t + 1)} = \sigma \Big (\boldsymbol {W} \boldsymbol {h} _ {v} ^ {(t)} + \sum_ {r \in R} \sum_ {w \in \mathcal {N} _ {r} (v)} \boldsymbol {W} _ {r} \boldsymbol {h} _ {w} ^ {(t)} + \boldsymbol {b} \Big),
$$
where  $\pmb{W} \in \mathbb{R}^{L \times L}$  is a parameter matrix and  $\pmb{b} \in \mathbb{R}^{L}$  is the bias term. As message function  $\mathrm{MSG}_r$  we use  $\mathrm{MSG}_r(\pmb{h}) = \pmb{W}_r\pmb{h}$ , where  $\pmb{W}_r \in \mathbb{R}^{L \times L}$  is a parameter matrix. For the non-linearity  $\sigma$  we use the truncated ReLU function  $\sigma(x) = \min(\max(0, x), 1)$ . The  $\ell$ -th row of  $\pmb{W}$  and  $\pmb{W}_r$ , and the  $\ell$ -th entry of  $\pmb{b}$  are defined as follows (omitted entries are 0):
1. If  $\varphi_{\ell}(x) = a(x)$  for a color  $a \in \mathcal{C}$ , then  $\pmb{W}_{\ell \ell} = 1$ .
2. If  $\varphi_{\ell}(x) = -\varphi_{k}(x)$  then  $W_{\ell k} = -1$  and  $b_{\ell} = 1$
3. If  $\varphi_{\ell}(x) = \varphi_{j}(x)\wedge \varphi_{k}(x)$  then  $W_{\ell j} = 1$ $W_{\ell k} = 1$  and  $b_{\ell} = -1$
4. If  $\varphi_{\ell}(x) = \exists^{\geq N}y(\varphi_k(y)\wedge r(y,x))$  then  $(W_{r})_{\ell k} = 1$  and  $b_{\ell} = -N + 1$
Let  $G = (V, E, R, c)$  be a knowledge graph with node colors from  $\mathcal{C}$ . In order to apply  $\mathcal{B}_{\varphi}$  to  $G$ , we choose initial  $L$ -dimensional features  $h_v^{(0)}$  such that  $(h_v^{(0)})_\ell = 1$  if  $\varphi_\ell = a(x)$  and  $a$  is the color of  $v$ , and  $(h_v^{(0)})_\ell = 0$  otherwise. In other words, the  $L$ -dimensional initial feature  $h_v^{(0)}$  is a one-hot encoding of the color of  $v$ . It follows from the same arguments than Proposition 4.1 in Barceló et al. [3] that for all  $\ell \in \{1, \dots, L\}$  we have  $(h_v^{(t)})_\ell = 1$  if  $G, v \models \varphi_\ell$  and  $(h_v^{(t)})_\ell = 0$  otherwise, for all  $v \in V$  and  $t \in \{\ell, \dots, L\}$ . In particular, after  $L$  layers,  $\mathcal{B}_{\varphi}$  calculates  $h_v^{(L)}$  such that  $(h_v^{(L)})_L = 1$  if  $G, v \models \varphi$  and  $(h_v^{(L)})_L = 0$  otherwise. As layer  $L + 1$  extracts the  $L$ -th component of the feature vector, the result follows.
For the forward direction, we follow the strategy of Theorem 4.2 from Barceló et al. [3]. Given a knowledge graph  $G = (V,E,R,c)$  and a number  $L\in \mathbb{N}$ , we define the unravelling of  $v\in V$  at depth  $L$ , denoted  $\mathrm{Unr}_G^L (v)$  is the knowledge graph having:
- A node  $(v, u_1, \ldots, u_i)$  for each directed path  $u_i, \ldots, u_1, v$  in  $G$  of length  $i \leq L$ .
- For each  $r \in R$ , a fact  $r((v, u_1, \ldots, u_i), (v, u_1, \ldots, u_{i-1}))$  for all facts  $r(u_i, u_{i-1}) \in E$  (here  $u_0 \coloneqq v$ ).
Each node  $(v, u_1, \ldots, u_i)$  is colored with  $c(u_i)$ , that is, the same color as  $u_i$ .
Note that the notion of directed path is defined in the obvious way, as for directed graphs but ignoring relation types. Note also that  $\mathsf{Unr}_G^L (v)$  is a tree in the sense that the underlying undirected graph is a tree.
The following proposition is a trivial adaptation of Observation C.3 from Barceló et al. [3]. We write  $\mathsf{Unr}_G^L (v)\simeq \mathsf{Unr}_{G'}^L (v')$  if there exists an isomorphism  $f$  from  $\mathsf{Unr}_G^L (v)$  to  $\mathsf{Unr}_{G'}^L (v')$  such that  $f(v) = v'$ .
Proposition A.12. Let  $G$  and  $G'$  be two knowledge graphs and  $v$  and  $v'$  be nodes in  $G$  and  $G'$ , respectively. Then, for all  $L \in \mathbb{N}$ , we have that  $\mathsf{rw} |_1^{(L)}(v)$  on  $G$  coincides with  $\mathsf{rw} |_1^{(L)}(v')$  on  $G'$  if and only if  $\mathsf{Unr}_G^L(v) \simeq \mathsf{Unr}_{G'}^L(v')$ .
As a consequence of Theorem A.1, we obtain:
Proposition A.13. Let  $G$  and  $G'$  be two knowledge graphs and  $v$  and  $v'$  be nodes in  $G$  and  $G'$ , respectively, such that  $\mathsf{Unr}_G^L(v) \simeq \mathsf{Unr}_{G'}^L(v')$  for all  $L \in \mathbb{N}$ . Then for any R-MPNN without global readout with  $T$  layers, we have that  $\pmb{h}_v^{(T)}$  on  $G$  coincides with  $\pmb{h}_{v'}^{(T)}$  on  $G'$ .
Finally, the following theorem follows from Theorem C.5 in Barceló et al. [3], which in turn follows from Theorem 2.2 in Otto [23]. The key observation here is that the results from Otto [23] are actually presented for multi-modal logics, that is, multiple relation types.
Theorem A.14. [23] Let  $\alpha$  be a unary FO formula over knowledge graphs. If  $\alpha$  is not equivalent to a rFO $_{\text{cnt}}^2$ . formula, then there exist two knowledge graphs  $G$  and  $G'$ , and two nodes  $v$  in  $G$  and  $v'$  in  $G'$  such that  $\mathsf{Unr}_G^L(v) \simeq \mathsf{Unr}_{G'}^L(v')$  for all  $L \in \mathbb{N}$  and such that  $G, v \models \alpha$  but  $G', v' \not\models \alpha$ .
Now we are ready to obtain the forward direction of the theorem. Suppose that a logical classifier  $\alpha$  is captured by an R-MPNN without global readout  $\mathcal{B}$  with  $T$  layers, and assume by contradiction that  $\alpha$  is not equivalent to a  $\mathsf{rFO}_{\mathrm{cnt}}^2$  formula. Then we can apply Theorem A.14 and obtain two knowledge graphs  $G$  and  $G'$ , and two nodes  $v$  in  $G$  and  $v'$  in  $G'$  such that  $\mathsf{Unr}_G^L(v) \simeq \mathsf{Unr}_{G'}^L(v')$  for all  $L \in \mathbb{N}$  and such that  $G, v \models \alpha$  but  $G', v' \not\models \alpha$ . Applying Proposition A.13, we have that  $\pmb{h}_v^{(T)}$  on  $G$  coincides with  $\pmb{h}_{v'}^{(T)}$  on  $G'$ , and hence  $\mathcal{B}$  classifies either both  $v$  and  $v'$  as true over  $G$  and  $G'$ , respectively, or both as false. This is a contradiction.
Now Theorem 5.2 follows easily. Let  $\alpha$  be a logical binary classifier and suppose it is captured by a C-MPNN without global readout  $\mathcal{A}$ . By Proposition A.7, item (1), we know that  $\mathcal{A}$  can be simulated by a R-MPNN without global readout  $\mathcal{B}$  over  $G^2$ . In turn, we can apply Theorem A.11 and obtain a formula  $\varphi$  in  $\mathsf{rFO}_{\mathrm{cnt}}^2$  equivalent to  $\mathcal{B}$ . Finally, we can apply the translation in Proposition A.10, item (2), and obtain a corresponding formula  $\tilde{\varphi}$  in  $\mathsf{rFO}_{\mathrm{cnt}}^3$ . We claim that  $\tilde{\varphi}$  captures  $\mathcal{A}$ . Let  $G$  be a knowledge graph and  $u, v$  two nodes. We have that  $G, u, v \models \tilde{\varphi}$  iff  $G^2$ ,  $(u, v) \models \varphi$  iff  $\mathcal{B}$  classifies  $(u, v)$  as true over  $G^2$  iff  $\mathcal{A}$  classifies  $(u, v)$  as true over  $G$ .
The other direction is obtained analogously following the reverse translations.
A.4 Proof of Theorem 5.3
We start by formally defining the logic  $\mathsf{erFO}_{\mathrm{cnt}}^3$ , which is a simple extension of  $\mathsf{rFO}_{\mathrm{cnt}}^3$  from Section 5.2. Fix a set of relation types  $R$  and a set of pair colors  $\mathcal{C}$ . Recall we consider knowledge graphs of the form  $G = (V, E, R, \eta)$  where  $\eta$  is a mapping assigning colors from  $\mathcal{C}$  to pairs of nodes from  $V$ . The logic  $\mathsf{erFO}_{\mathrm{cnt}}^3$  contains only binary formulas and it is defined inductively as follows: First,  $a(x, y)$  for  $a \in \mathcal{C}$ , is in  $\mathsf{erFO}_{\mathrm{cnt}}^3$ . Second, if  $\varphi(x, y)$  and  $\psi(x, y)$  are in  $\mathsf{erFO}_{\mathrm{cnt}}^3$ ,  $N \geq 1$  is a positive integer, and  $r \in R$ , then the formulas
$$
\neg \varphi (x, y), \quad \varphi (x, y) \wedge \psi (x, y), \quad \exists^ {\geq N} z (\varphi (x, z) \wedge r (z, y)), \quad \exists^ {\geq N} z (\varphi (x, z) \wedge \neg r (z, y))
$$
are also in  $\mathsf{erFO}_{\mathrm{cnt}}^3$ . As expected,  $a(u,v)$  holds in  $G = (V,E,R,\eta)$  if  $\eta (u,v) = a$ , and  $\exists^{\geq N}z(\varphi (u,z)\wedge \ell (z,v))$  holds in  $G$ , for  $\ell \in R\cup \{\neg r\mid r\in R\}$ , if there are at least  $N$  nodes  $w\in V$  for which  $\varphi (u,w)$  and  $\ell (w,v)$  hold in  $G$ .
Intuitively,  $\mathsf{erFO}_{\mathrm{cnt}}^{3}$  extends  $\mathsf{rFO}_{\mathrm{cnt}}^{3}$  with negated modalities, that is, one can check for non-neighbors of a node. As it turns out,  $\mathsf{erFO}_{\mathrm{cnt}}^{3}$  is strictly more expressive than  $\mathsf{rFO}_{\mathrm{cnt}}^{3}$ .
Proposition A.15. There is a formula in  $\mathsf{erFO}_{\mathrm{cnt}}^3$  that cannot be expressed by any formula in  $\mathsf{rFO}_{\mathrm{cnt}}^3$ .
Proof. Consider the following two knowledge graphs  $G_{1}$  and  $G_{2}$  over  $R = \{r\}$  and colors  $\mathcal{C} = \{\text{Green}\}$ . The graph  $G_{1}$  has two nodes  $u$  and  $v$  and one edge  $r(u, v)$ . The graph  $G_{2}$  has three nodes  $u, v$  and  $w$  and one edge  $r(u, v)$  (hence  $w$  is an isolated node). In both graphs, all pairs are colored Green.
We show first that for every formula  $\varphi(x, y)$  in  $\mathsf{rFO}_{\mathrm{cnt}}^3$ , it is the case that  $G_1, u, n \models \varphi$  iff  $G_2, u, n \models \varphi$ , for  $n \in \{u, v\}$ . We proceed by induction on the formula. For  $\varphi(x, y) = \operatorname{Green}(x, y)$ , we have