SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[18] Lecheng Kong, Yixin Chen, and Muhan Zhang. Geodesic graph neural network for efficient graph representation learning. In NeurIPS, 2022.
[19] Carsten Lutz, Ulrike Sattler, and Frank Wolter. Modal logic and the two-variable fragment. In CSL, 2001.
[20] Christian Meilicke, Manuel Fink, Yanjie Wang, Daniel Ruffinelli, Rainer Gemulla, and Heiner Stuckenschmidt. Fine-grained evaluation of rule- and embedding-based systems for knowledge graph completion. In IWSW, 2018.
[21] Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and Leman go neural: Higher-order graph neural networks. In AAAI, 2019.
[22] Christopher Morris, Gaurav Rattan, and Petra Mutzel. Weisfeiler and leman go sparse: Towards scalable higher-order graph embeddings. In NeurIPS, 2020.
[23] Martin Otto. Graded modal logic and counting bisimulation. In arXiv, 2019.
[24] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: online learning of social representations. In KDD, 2014.
[25] Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. Drum: End-to-end differentiable rule mining on knowledge graphs. In NIPS, 2019.
[26] Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In SDM, 2021.
[27] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 2009.
[28] Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In ESWC, 2018.
[29] Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR, 2019.
[30] Komal K. Teru, Etienne G. Denis, and William L. Hamilton. Inductive relation prediction by subgraph reasoning. In ICML, 2020.
[31] Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In CVSC, 2015.
[32] Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition-based multi-relational graph convolutional networks. In ICLR, 2020.
[33] Hongwei Wang, Hongyu Ren, and Jure Leskovec. Relational message passing for knowledge graph completion. In KDD, 2021.
[34] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[35] Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. In NeurIPS, 2017.
[36] Jiaxuan You, Jonathan M. Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In AAAI, 2021.
[37] Donghan Yu, Yiming Yang, Ruohong Zhang, and Yuexin Wu. Generalized multi-relational graph convolution network. In ICML Workshop on Graph Representation Learning and Beyond, 2020.
[38] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In NeurIPS, 2018.
[39] Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. In NeurIPS, 2021.
[40] Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford networks: A general graph neural network framework for link prediction. In NeurIPS, 2021.
A Proofs of technical statements
A.1 Expressive power of R-MPNNs
The expressive power of R-MPNNs has been recently characterized in terms of a relational variant of the Weisfeiler-Leman test [4]. We define the relational local 1-WL test $^2$ , which we denote by  $\mathsf{rwl}_1$ . Let  $G = (V, E, R, c)$  be a knowledge graph. For each  $t \geq 0$ , the test updates the coloring as follows:
$$
\mathsf {r w l} _ {1} ^ {(0)} (v) = c (v),
$$
$$
\left. \mathsf {r w l} _ {1} ^ {(t + 1)} (v) = \tau \big (\mathsf {r w l} _ {1} ^ {(t)} (v), \{\{((\mathsf {r w l} _ {1} ^ {(t)} (v), r) | w \in \mathcal {N} _ {r} (v), r \in R \} \}, \right.
$$
where  $\tau$  injectively maps the above pair to a unique color, which has not been used in previous iterations. Hence,  $r\mathsf{w}|_{1}^{(t)}$  defines a node invariant for all  $t \geq 0$ .
The following result generalizes results from Barceló et al. [4].
Theorem A.1. Let  $G = (V, E, R, c)$  be a knowledge graph.
1. For all initial feature maps  $\pmb{x}$  with  $c \equiv x$ , all R-MPNNs with  $T$  layers, and  $0 \leq t \leq T$ , it holds that  $\mathsf{rw}|_1^{(t)} \preceq \pmb{h}^{(t)}$ .
2. For all  $T \geq 0$  and history function  $f$ , there is an initial feature map  $\mathbf{x}$  with  $c \equiv \mathbf{x}$  and an R-MPNN without global readout with  $T$  layers and history function  $f$ , such that for all  $0 \leq t \leq T$  we have  $\mathsf{rwI}_1^{(t)} \equiv \mathbf{h}^{(t)}$ .
Intuitively, item (1) states that the relational local 1-WL algorithm upper bounds the power of any R-MPNN  $\mathcal{A}$ : if the test cannot distinguish two nodes, then  $\mathcal{A}$  cannot either. On the other hand, item (2) states that R-MPNNs can be as expressive as  $\mathrm{rwl}_1$ : for any time limit  $T$ , there is an R-MPNN that simulates  $T$  iterations of the test. This holds even without a global readout component.
Remark A.2. A direct corollary of Theorem A.1 is that R-MPNNs have the same expressive power as  $\mathsf{rwl}_1$ , independently of their history function.
In order to prove Theorem A.1, we define a variant of  $\mathsf{rw}l_{1}$  as follows. For a history function  $f:\mathbb{N}\to \mathbb{N}$  (recall  $f$  is non-decreasing and  $f(t)\leq t$ ), and given a knowledge graph  $G = (V,E,R,c)$ , we define the  $\mathsf{rw}l_{1,f}$  test via the following update rules:
$$
\mathsf {r w l} _ {1, f} ^ {(0)} (v) = c (v)
$$
$$
\mathsf {r w l} _ {1, f} ^ {(t + 1)} (v) = \tau \big (\mathsf {r w l} _ {1, f} ^ {(f (t))} (v), \{\{\left(\mathsf {r w l} _ {1, f} ^ {(t)} (v), r) \right\rvert   w \in \mathcal {N} _ {r} (v), r \in R \} \} \big),
$$
where  $\tau$  injectively maps the above pair to a unique color, which has not been used in previous iterations. Note that  $\mathsf{rw}l_{1}$  corresponds to  $\mathsf{rw}l_{1,id}$  for the identity function  $id(t) = t$ .
We have that  $\mathsf{rw}|_{1,f}^{(t)}$  is always a refinement of  $\mathsf{rw}|_{1,f}^{(t - 1)}$ . Note that this is trivial for  $f(t) = t$  but not for arbitrary history functions.
Proposition A.3. Let  $G = (V, E, R, c)$  be a knowledge graph and  $f$  be a history function. Then for all  $t \geq 0$ , we have  $\mathsf{rw}|_{1,f}^{(t+1)} \preceq \mathsf{rw}|_{1,f}^{(t)}$ .
Proof. We proceed by induction on  $t$ . For  $t = 0$ , note that  $\mathsf{rw}|_{1,f}^{(1)}(u) = \mathsf{rw}|_{1,f}^{(1)}(v)$  implies  $\mathsf{rw}|_{1,f}^{(f(0))}(u) = \mathsf{rw}|_{1,f}^{(f(0))}(v)$  and  $f(0) = 0$ . For the inductive case, suppose  $\mathsf{rw}|_{1,f}^{(t + 1)}(u) = \mathsf{rw}|_{1,f}^{(t + 1)}(v)$  for  $t \geq 1$  and  $u, v \in V$ . By injectivity of  $\tau$ , we have that:
$$
\operatorname {r w l} _ {1, f} ^ {(f (t))} (u) = \operatorname {r w l} _ {1, f} ^ {(f (t))} (v)
$$
$$
\{\{\left(\mathsf {r w} | _ {1, f} ^ {(t)} (w), r\right) \mid w \in \mathcal {N} _ {r} (u), r \in R \} \} = \{\{\left(\mathsf {r w} | _ {1, f} ^ {(t)} (w), r\right) \mid w \in \mathcal {N} _ {r} (v), r \in R \} \}.
$$
2This test over single-relation graphs is often called color refinement [13]. In Barceló et al. [4] it is also called multi-relational 1-WL.
By inductive hypothesis and the facts that  $f(t - 1) \leq f(t)$  (as  $f$  is non-decreasing) and  $\mathsf{rwl}_{1,f}^{(f(t))}(u) = \mathsf{rwl}_{1,f}^{(f(t))}(v)$ , we obtain  $\mathsf{rwl}_{1,f}^{(f(t - 1))}(u) = \mathsf{rwl}_{1,f}^{(f(t - 1))}(v)$ . On the other hand, by inductive hypothesis, we obtain that
$$
\{\{(r w | _ {1, f} ^ {(t - 1)} (w), r) \mid w \in \mathcal {N} _ {r} (u), r \in R \} \} = \{\{(r w | _ {1, f} ^ {(t - 1)} (w), r) \mid w \in \mathcal {N} _ {r} (v), r \in R \} \}.
$$
We conclude that  $\mathsf{rw} \big|_{1,f}^{(t)}(u) = \mathsf{rw} \big|_{1,f}^{(t)}(v)$ .
As it turns out,  $\mathsf{rw}|_{1,f}^{(t)}$  defines the same coloring independently of  $f$ :
Proposition A.4. Let  $G = (V, E, R, c)$  be a knowledge graph and  $f, f'$  be history functions. Then for all  $t \geq 0$ , we have that  $\mathsf{rw}|_{1,f}^{(t)} \equiv \mathsf{rw}|_{1,f'}^{(t)}$ .
Proof. We apply induction on  $t$ . For  $t = 0$ , we have  $\mathsf{rw}|_{1,f}^{(0)} \equiv c \equiv \mathsf{rw}|_{1,f'}^{(0)}$ . For the inductive case, suppose  $\mathsf{rw}|_{1,f}^{(t)}(u) = \mathsf{rw}|_{1,f}^{(t)}(v)$ , for  $t \geq 1$  and  $u, v \in V$ . Since  $f'(t - 1) \leq t - 1$  and by Proposition A.3, we have that  $\mathsf{rw}|_{1,f}^{(f'(t - 1))}(u) = \mathsf{rw}|_{1,f}^{(f'(t - 1))}(v)$ . The inductive hypothesis implies that  $\mathsf{rw}|_{1,f'}^{(f'(t - 1))}(u) = \mathsf{rw}|_{1,f'}^{(f'(t - 1))}(v)$ . On the other hand, by injectivity of  $\tau$  we have
The inductive hypothesis implies that
$$
\{\{(r w | _ {1, f ^ {\prime}} ^ {(t - 1)} (w), r) \mid w \in \mathcal {N} _ {r} (u), r \in R \} \} = \{\{(r w | _ {1, f ^ {\prime}} ^ {(t - 1)} (w), r) \mid w \in \mathcal {N} _ {r} (v), r \in R \} \}.
$$
Summing up, we have that  $\mathsf{rw} |_{1,f'}^{(t)}(u) = \mathsf{rw} |_{1,f'}^{(t)}(v)$ , and hence  $\mathsf{rw} |_{1,f}^{(t)} \preceq \mathsf{rw} |_{1,f'}^{(t)}$ . The case  $\mathsf{rw} |_{1,f'}^{(t)} \preceq \mathsf{rw} |_{1,f}^{(t)}$  follows by symmetry.
Now we are ready to prove Theorem A.1.
We start with item (1). Take an initial feature map  $\pmb{x}$  with  $c \equiv x$ , an R-MPNN with  $T$  layers, and history function  $f$ . It suffices to show that  $\mathsf{rw}|_{1,f}^{(t)} \preceq \pmb{h}^{(t)}$ , for all  $0 \leq t \leq T$ . Indeed, by Proposition A.4, we have  $\mathsf{rw}|_{1,f}^{(t)} \equiv \mathsf{rw}|_{1,id}^{(t)} \equiv \mathsf{rw}|_{1}^{(t)}$ , where  $id$  is the identity function  $id(t) = t$ , and hence the result follows. We apply induction on  $t$ . The case  $t = 0$  follows directly as  $\mathsf{rw}|_{1,f}^{(0)} \equiv c \equiv x \equiv h^{(0)}$ . For the inductive case, assume  $\mathsf{rw}|_{1,f}^{(t)}(u) = \mathsf{rw}|_{1,f}^{(t)}(v)$  for  $t \geq 1$  and  $u, v \in V$ . By injectivity of  $\tau$  we have:
$$
\operatorname {r w l} _ {1, f} ^ {(f (t - 1))} (u) = \operatorname {r w l} _ {1, f} ^ {(f (t - 1))} (v)
$$
$$
\{\{\left(\mathsf {r w l} _ {1, f} ^ {(t - 1)} (w), r\right) \mid w \in \mathcal {N} _ {r} (u), r \in R \} \} = \{\{\left(\mathsf {r w l} _ {1, f} ^ {(t - 1)} (w), r\right) \mid w \in \mathcal {N} _ {r} (v), r \in R \} \}.
$$
By inductive hypothesis we have (recall  $f(t - 1) \leq t - 1$ ):
$$
\boldsymbol {h} _ {u} ^ {(f (t - 1))} = \boldsymbol {h} _ {v} ^ {(f (t - 1))}
$$
$$
\{\left\{\boldsymbol {h} _ {w} ^ {(t - 1)} \mid w \in \mathcal {N} _ {r} (u) \right\} \} = \{\left\{\boldsymbol {h} _ {w} ^ {(t - 1)} \mid w \in \mathcal {N} _ {r} (v) \right\} \quad \text {f o r e a c h} r \in R.
$$
This implies that  $\{\{\mathrm{MSG}_r(\pmb{h}_w^{(t - 1)})\mid w\in \mathcal{N}_r(u)\} \} = \{\{\mathrm{MSG}_r(\pmb{h}_w^{(t - 1)})\mid w\in \mathcal{N}_r(v)\} \}$ , for each  $r\in R$  and hence:
$$
\{\left\{\operatorname {M s g} _ {r} \left(\boldsymbol {h} _ {w} ^ {(t - 1)}\right) \mid w \in \mathcal {N} _ {r} (u), r \in R \right\} \} = \{\left\{\operatorname {M s g} _ {r} \left(\boldsymbol {h} _ {w} ^ {(t - 1)}\right) \mid w \in \mathcal {N} _ {r} (v), r \in R \right\}.
$$
$$
\begin{array}{l} \boldsymbol {h} _ {u} ^ {(t)} = \operatorname {U P D} \left( \right.\boldsymbol {h} _ {u} ^ {(f (t - 1))}, \operatorname {A G G} \left( \right.\left\{ \right.\left\{\mathbf {M S G} _ {r} \left(\boldsymbol {h} _ {w} ^ {(t - 1)}\right) \mid w \in \mathcal {N} _ {r} (u), r \in R \right\}\left. \right), \operatorname {R E A D} \left( \right.\left\{ \right.\left\{\boldsymbol {h} _ {w} ^ {(t)} \mid w \in V \right\}\left. \right)\left. \right)\left. \right) \\ = \operatorname {U P D} \left( \right.\boldsymbol {h} _ {v} ^ {(f (t - 1))}, \operatorname {A G G} \left( \right.\left\{ \right.\left\{\operatorname {M S G} _ {r} \left(\boldsymbol {h} _ {w} ^ {(t - 1)}\right) \mid w \in \mathcal {N} _ {r} (v), r \in R \right\}\left. \right), \operatorname {R E A D} \left( \right.\left\{ \right.\left\{\boldsymbol {h} _ {w} ^ {(t)} \mid w \in V \right\}\left. \right)\left. \right)\left. \right) \\ = \boldsymbol {h} _ {v} ^ {(t)}. \\ \end{array}
$$
For item (2), we refine the proof of Theorem 2 from Barceló et al. [4], which is based on ideas from Morris et al. [21]. In comparison with Barceló et al. [4], in our case, we have arbitrary adjacency matrices for each relation type, not only symmetric ones, and arbitrary history functions, not only the identity. However, the arguments still apply. Moreover, the most important difference is that here we aim for a model of R-MPNN without global readout that uses a single parameter matrix, instead of two parameter matrices as in Barceló et al. [4] (one for self-representations and the other for neighbors representations). This makes the simulation of  $\mathsf{rw}l_{1}$  more challenging.
We use models of R-MPNNs without global readout of the following form:
$$
\pmb {h} _ {v} ^ {(t + 1)} = \mathrm {s i g n} \left(\pmb {W} ^ {(t)} \big (\pmb {h} _ {v} ^ {(f (t))} + \sum_ {r \in R} \sum_ {w \in \mathcal {N} _ {r} (v)} \alpha_ {r} \pmb {h} _ {w} ^ {(t)} \big) - \pmb {b}\right),
$$
where  $\pmb{W}^{(t)}$  is a parameter matrix and  $\pmb{b}$  is the bias term (we shall use the all-ones vector  $\pmb{b} = \mathbf{1}$ ). As message function  $\mathrm{Mg}_r$  we use vector scaling, that is,  $\mathrm{Mg}_r(\pmb{h}) = \alpha_r\pmb{h}$ , where  $\alpha_{r}$  is a parameter of the model. For the non-linearity, we use the sign function sign. We note that the proof also works for the ReLU function, following arguments from Corollary 16 in Morris et al. [21].
For a matrix  $\mathbf{B}$ , we denote by  $\mathbf{B}_i$  its  $i$ -th column. Let  $n = |V|$  and without loss of generality assume  $V = \{1, \ldots, n\}$ . We will write features maps  $\mathbf{h}: V \to \mathbb{R}^d$  for  $G = (V, E, R, c)$  also as matrices  $\mathbf{H} \in \mathbb{R}^{d \times n}$ , where the column  $\mathbf{H}_v$  corresponds to the  $d$ -dimensional feature vector for  $v$ . Then we can also write our R-MPNN model in matrix form:
$$
\boldsymbol {H} ^ {(t + 1)} = \operatorname {s i g n} \left(\boldsymbol {W} ^ {(t)} \left(\boldsymbol {H} ^ {(f (t))} + \sum_ {r \in R} \alpha_ {r} \boldsymbol {H} ^ {(t)} \boldsymbol {A} _ {r}\right) - \boldsymbol {J}\right),
$$
where  $\mathbf{A}_r$  is the adjacency matrix of  $G$  for relation type  $r \in R$  and  $\mathbf{J}$  is the all-ones matrix of appropriate dimensions.
Let  $Fts$  be the following  $n \times n$  matrix:
$$
\boldsymbol {F} t s = \left[ \begin{array}{c c c c c} - 1 & - 1 & \dots & - 1 & - 1 \\ 1 & - 1 & \ddots & & - 1 \\ \vdots & \ddots & \ddots & \ddots & \vdots \\ 1 & & \ddots & - 1 & - 1 \\ 1 & 1 & \dots & 1 & - 1 \end{array} \right]
$$
That is,  $(\mathbf{Fts})_{ij} = -1$  if  $j \geq i$ , and  $(\mathbf{Fts})_{ij} = 1$  otherwise. Note that the columns of  $\mathbf{Fts}$  are linearly independent. We shall use the columns of  $\mathbf{Fts}$  as node features in our simulation.
The following lemma is an adaptation of Lemma 9 from Morris et al. [21].
Lemma A.5. Let  $\pmb{B} \in \mathbb{N}^{n \times p}$  be a matrix such that  $p \leq n$ , and all the columns are pairwise distinct and different from the all-zeros column. Then there is a matrix  $\pmb{X} \in \mathbb{R}^{n \times n}$  such that the matrix  $\mathrm{sign}(\pmb{X}\pmb{B} - \pmb{J}) \in \{-1, 1\}^{n \times p}$  is precisely the sub-matrix of  $\pmb{F}$ s given by its first  $p$  columns.
Proof. Let  $\mathbf{z} = (1, m, m^2, \ldots, m^{n-1}) \in \mathbb{N}^{1 \times n}$ , where  $m$  is the largest entry in  $\mathbf{B}$ , and  $\mathbf{b} = \mathbf{z}\mathbf{B} \in \mathbb{N}^{1 \times p}$ . By construction, the entries of  $\mathbf{b}$  are positive and pairwise distinct. Without loss of generality, we assume that  $\mathbf{b} = (b_1, b_2, \ldots, b_p)$  for  $b_1 > b_2 > \cdots > b_p > 0$ . As the  $b_i$  are ordered, we can choose numbers  $x_1, \ldots, x_p \in \mathbb{R}$  such that  $b_i \cdot x_j < 1$  if  $i \geq j$ , and  $b_i \cdot x_j > 1$  if  $i < j$ , for all  $i, j \in \{1, \ldots, p\}$ . Let  $\mathbf{x} = (x_1, \ldots, x_p, 2/b_p, \ldots, 2/b_p)^T \in \mathbb{R}^{n \times 1}$ . Note that  $(2/b_p) \cdot b_i > 1$ , for all  $i \in \{1, \ldots, p\}$ . Then  $\operatorname{sign}(\mathbf{x}\mathbf{b} - \mathbf{J})$  is precisely the sub-matrix of  $Fts$  given by its first  $p$  columns. We can choose  $X = x\mathbf{z} \in \mathbb{R}^{n \times n}$ .
Now we are ready to show item (2). Let  $f$  be any history function and  $T \geq 0$ . It suffices to show that there is a feature map  $\pmb{x}$  with  $c \equiv \pmb{x}$  and an R-MPNN without global readout with  $T$  layers and history function  $f$  such that  $\mathsf{rw}|_{1,f}^{(t)} \equiv \pmb{h}^{(t)}$ , for all  $0 \leq t \leq T$ . Indeed, by Proposition A.4, we have  $\mathsf{rw}|_{1,f}^{(t)} \equiv \mathsf{rw}|_{1,id}^{(t)} \equiv \mathsf{rw}|_1^{(t)}$ , where  $id$  is the identity function  $id(t) = t$ , and then the result follows. We conclude item (2) by showing the following lemma:
Lemma A.6. There is a feature map  $\pmb{h}^{(0)}: V \to \mathbb{R}^n$ , and for all  $0 \leq t < T$ , there is a feature map  $\pmb{h}^{(t+1)}: V \to \mathbb{R}^n$ , a matrix  $\pmb{W}^{(t)} \in \mathbb{R}^{n \times n}$  and scaling factors  $\alpha_r^{(t)} \in \mathbb{R}$ , for each  $r \in R$ , such that:
$\pmb{h}^{(t)} \equiv \mathsf{rw}|_{1,f}^{(t)}$
- The columns of  $\mathbf{H}^{(t)} \in \mathbb{R}^{n \times n}$  are columns of  $\mathbf{Fts}$  (recall  $\mathbf{H}^{(t)}$  is the matrix representation of  $\mathbf{h}^{(t)})$ ).
$\pmb{H}^{(t + 1)} = \mathrm{sign}\left(\pmb{W}^{(t)}\big(\pmb{H}^{(f(t))} + \sum_{r\in R}\alpha_r^{(t)}\pmb{H}^{(t)}\pmb{A}_r\big) - \pmb {J}\right).$
Proof. We proceed by induction on  $t$ . Suppose that the node coloring  $\mathsf{rw}l_{1,f}^{(0)} \equiv c$  uses colors  $1, \ldots, p$ , for  $p \leq n$ . Then we choose  $h^{(0)}$  (this is the initial feature map  $x$  in the statement of item (2)) such that  $h_v^{(0)} = Fts_{c(v)}$ , that is,  $h_v^{(0)}$  is the  $c(v)$ -th column of  $Fts$ . We have that  $h^{(0)}$  satisfies the required conditions.
For the inductive case, assume that  $\pmb{h}^{(t^{\prime})} \equiv \mathsf{rw}\mathsf{l}_{1,f}^{(t^{\prime})}$  and the columns of  $H^{(t^{\prime})}$  are columns of  $Fts$ , for all  $0 \leq t' \leq t < T$ . We need to find  $\pmb{h}^{(t + 1)}, \pmb{W}^{(t)}$  and  $\alpha_r^{(t)}$  satisfying the conditions. Let  $M \in \mathbb{R}^{n \times n}$  be the matrix inverse of  $Fts$ . If  $H_v^{(t')}$  is the  $i$ -th column of  $Fts$ , we say that  $v$  has color  $i$  at iteration  $t'$ . Observe that for all  $0 \leq t' \leq t$ , we have
$$
(M H ^ {(t ^ {\prime})}) _ {i v} = \left\{ \begin{array}{l l} 1 & \text {i f v h a s c o l o r i a t i t e r a t i o n t ^ {\prime}} \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
In other words, the  $v$ -th column of  $\pmb{M}\pmb{H}^{(t^{\prime})}$  is simply a one-hot encoding of the color of  $v$  at iteration  $t^\prime$ . For each  $r\in R$  we have
$$
\left(\boldsymbol {M} \boldsymbol {H} ^ {(t)} \boldsymbol {A} _ {r}\right) _ {i v} = | \{w \in \mathcal {N} _ {r} (v) \mid w h a s c o l o r i a t i t e r a t i o n t h e r m a l \}.
$$
Hence the  $v$ -th column of  $\mathbf{M}\mathbf{H}^{(t)}\mathbf{A}_r$  is an encoding of the multiset of colors for the neighborhood  $\mathcal{N}_r(v)$ , at iteration  $t$ . Let  $r_1, \ldots, r_m$  be an enumeration of the relation types in  $R$ . Let  $\mathbf{D} \in \mathbb{R}^{(m + 1)n \times n}$  be the matrix obtained by horizontally concatenating the matrices  $\mathbf{M}\mathbf{H}^{(f(t))}$ ,  $\mathbf{M}\mathbf{H}^{(t)}\mathbf{A}_{r_1}, \ldots, \mathbf{M}\mathbf{H}^{(t)}\mathbf{A}_{r_m}$ . Since  $\mathbf{H}^{(f(t))} \equiv \mathrm{rw}|_{1,f}^{(f(t))}$  and  $\mathbf{H}^{(t)} \equiv \mathrm{rw}|_{1,f}^{(t)}$ , we have that  $\mathbf{D} \equiv \mathrm{rw}|_{1,f}^{(t + 1)}$ . Now note that  $\mathbf{D} \equiv \mathbf{E}$ , where
$$
\boldsymbol {E} = \boldsymbol {M} \boldsymbol {H} ^ {(f (t))} + \sum_ {i = 1} ^ {m} (n + 1) ^ {i} \boldsymbol {M} \boldsymbol {H} ^ {(t)} \boldsymbol {A} _ {r _ {i}}.
$$
Indeed,  $\pmb{E}_{iv}$  is simply the  $(n + 1)$ -base representation of the vector  $(\pmb{D}_{iv}, \pmb{D}_{(n+i)v}, \pmb{D}_{(2n+i)v}, \dots, \pmb{D}_{(mn+i)v})$ , and hence  $\pmb{E}_u = \pmb{E}_v$  if and only if  $\pmb{D}_u = \pmb{D}_v$  (note that the entries of  $\pmb{D}$  are in  $\{0, \dots, n\}$ ). In particular,  $\pmb{E} \equiv \mathsf{rw}|_{1,f}^{(t+1)}$ .
Let  $p$  be the number of distinct columns of  $\mathbf{E}$  and let  $\widetilde{\mathbf{E}}\in \mathbb{N}^{n\times p}$  be the matrix whose columns are the distinct columns of  $\mathbf{E}$  in an arbitrary but fixed order. We can apply Lemma A.5 to  $\widetilde{\mathbf{E}}$  and obtain a matrix  $\mathbf{X}\in \mathbb{R}^{n\times n}$  such that  $\mathrm{sign}(\boldsymbol {X}\widetilde{\boldsymbol{E}} -\boldsymbol {J})$  is precisely the sub-matrix of  $\boldsymbol {Fts}$  given by its first  $p$  columns. We choose  $\boldsymbol{H}^{(t + 1)} = \mathrm{sign}(\boldsymbol {X}\boldsymbol {E} - \boldsymbol {J})\in \mathbb{R}^{n\times n},\boldsymbol{W}^{(t)} = \boldsymbol {X}\boldsymbol {M}\in \mathbb{R}^{n\times n}$  and  $\alpha_{r_i}^{(t)} = (n + 1)^i$ . Note that the columns of  $\boldsymbol {H}^{(t + 1)}$  are columns of  $\boldsymbol {Fts}$ , and that  $\boldsymbol {H}^{(t + 1)}\equiv \boldsymbol {E}\equiv \mathsf{rw}|_{1,f}^{(t + 1)}$ . Finally, we have
$$
\begin{array}{l} \boldsymbol {H} ^ {(t + 1)} = \operatorname {s i g n} (\boldsymbol {X} \boldsymbol {E} - \boldsymbol {J}) \\ = \operatorname {s i g n} \left(\boldsymbol {X} \left(\boldsymbol {M} \boldsymbol {H} ^ {(f (t))} + \sum_ {i = 1} ^ {m} (n + 1) ^ {i} \boldsymbol {M} \boldsymbol {H} ^ {(t)} \boldsymbol {A} _ {r _ {i}}\right) - \boldsymbol {J}\right) \\ = \operatorname {s i g n} \left(\boldsymbol {W} ^ {(t)} \left(\boldsymbol {H} ^ {(f (t))} + \sum_ {i = 1} ^ {m} \alpha_ {r _ {i}} ^ {(t)} \boldsymbol {H} ^ {(t)} \boldsymbol {A} _ {r _ {i}}\right) - \boldsymbol {J}\right). \\ \end{array}
$$
Note that our result applies to more complex message functions such as  $\mathrm{M}\mathrm{s}\mathrm{G}_r(\boldsymbol {h}) = \boldsymbol {h}*\boldsymbol {b}_r$ , where  $*$  stands for element-wise multiplication and  $b_{r}$  is a vector parameter, and  $\mathrm{M}\mathrm{s}\mathrm{G}_r(\boldsymbol {h}) = \boldsymbol {W}_r\boldsymbol {h}$ , where  $\boldsymbol{W}_{r}$  is matrix parameter, as they can easily express vector scaling. The first case has been used for the model CompGCN [32], while the second case has been used for R-GCN [28].
A.2 Proof of Theorem 5.1
We recall the statement of the theorem:
Theorem 5.1. Let  $G = (V, E, R, \mathbf{x}, \eta)$  be a knowledge graph, where  $\mathbf{x}$  is a feature map, and  $\eta$  is a pairwise coloring satisfying target node distinguishability. Let  $q \in R$  be any query relation. Then:
1. For all  $C$ -MPNNs with  $T$  layers and initialization INIT satisfying INIT  $\equiv \eta$ , and all  $0 \leq t \leq T$ , we have  $\mathsf{rawl}_2^{(t)} \preceq h_q^{(t)}$ .
2. For all  $T \geq 0$  and history function  $f$ , there is an  $C$ -MPNN without global readout with  $T$  layers and history function  $f$  such that for all  $0 \leq t \leq T$ , we have  $\mathsf{rawl}_2^{(t)} \equiv \pmb{h}_q^{(t)}$ .
We start by showing item (1) and (2) of the theorem for the case without global readout. As we explain later, extending item (1) to include readouts is straightforward. In order to prove this particular case, we apply a reduction to R-MPNNs and the relational local 1-WL. Before doing so we need some auxiliary results.