SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The test is asymmetric: given a pair  $(u, v)$ , we only look at neighbors of  $(u, v)$  obtained by changing the second coordinate of the pair. In contrast, usual versions of (local)  $k$ -WL are symmetric as neighbors may change any coordinate. Interestingly, this test characterizes the power of C-MPNNs in terms of distinguishing pairs of nodes.
Theorem 5.1. Let  $G = (V, E, R, \mathbf{x}, \eta)$  be a knowledge graph, where  $\mathbf{x}$  is a feature map and  $\eta$  is a pairwise coloring satisfying target node distinguishability. Let  $q \in R$  be any query relation. Then:
1. For all  $C$ -MPNNs with  $T$  layers and initializations  $\mathrm{INIT}$  with  $\mathrm{INIT} \equiv \eta$ , and  $0 \leq t \leq T$ , we have  $\mathsf{raw}|_{2}^{(t)} \preceq \pmb{h}_{q}^{(t)}$ .
2. For all  $T \geq 0$  and history function  $f$ , there is a C-MPNN without global readout with  $T$  layers and history function  $f$  such that for all  $0 \leq t \leq T$ , we have  $\mathsf{raw}_{2}^{(t)} \equiv \pmb{h}_{q}^{(t)}$ .
The idea of the proof is as follows: we first show a correspondent characterization of the expressive power of R-MPNNs in terms of a relational variant of the WL test (Theorem A.1). This result generalizes results from Barceló et al. [4]. We then apply a reduction from C-MPNNs to R-MPNNs, that is, we carefully build an auxiliary knowledge graph (encoding the pairs of nodes of the original knowledge graph) to transfer our R-MPNN characterization to our sought C-MPNN characterization.
Note that the lower bound (item (2)) holds even for the basic model of C-MPNNs (without global readout) and the three proposed message functions from Section 4.1. The expressive power of C-MPNNs is independent of the history function as in any case it is matched by  $\mathsf{rawl}_2$ . This suggests that the difference between traditional message passing models using functions  $f(t) = t$  and path-based models (such as NBFNets [40]) using  $f(t) = 0$  is not relevant from a theoretical point of view.
5.2 Logical characterization
We now turn to the problem of which binary classifiers can be expressed as C-MPNNs. That is, we look at C-MPNNs that classify each pair of nodes in a knowledge graph as true or false. Following Barceló et al. [3], we study logical binary classifiers, i.e., those that can be defined in the formalism of first-order logic (FO). Briefly, a first-order formula  $\phi(x,y)$  with two free variables  $x,y$  defines a logical binary classifier that assigns value true to the pair  $(u,v)$  of nodes in knowledge graph  $G$  whenever  $G\models\phi(u,v)$ , i.e.,  $\phi$  holds in  $G$  when  $x$  is interpreted as  $u$  and  $y$  as  $v$ . A logical classifier  $\phi(x,y)$  is captured by a C-MPNN  $\mathcal{A}$  if over every knowledge graph  $G$  the pairs  $(u,v)$  of nodes that are classified as true by  $\phi$  and  $\mathcal{A}$  are the same.
A natural problem then is to understand what are the logical classifiers captured by C-MPNNs. Fix a set of relation types  $R$  and a set of pair colors  $\mathcal{C}$ . We consider knowledge graphs of the form  $G = (V, E, R, \eta)$  where  $\eta$  is a mapping assigning colors from  $\mathcal{C}$  to pairs of nodes from  $V$ . In this context, FO formulas can refer to the different relation types in  $R$  and the different pair colors in  $\mathcal{C}$ . Our first characterization is established in terms of a simple fragment of FO, which we call  $\mathrm{rFO}_{\mathrm{cnt}}^3$ , and is inductively defined as follows: First,  $a(x, y)$  for  $a \in \mathcal{C}$ , is in  $\mathrm{rFO}_{\mathrm{cnt}}^3$ . Second, if  $\varphi(x, y)$  and  $\psi(x, y)$  are in  $\mathrm{rFO}_{\mathrm{cnt}}^3$ ,  $N \geq 1$  is a positive integer, and  $r \in R$ , then the formulas
$$
\neg \varphi (x, y), \quad \varphi (x, y) \wedge \psi (x, y), \quad \exists^ {\geq N} z (\varphi (x, z) \wedge r (z, y))
$$
are also in  $\mathsf{rFO}_{\mathrm{cnt}}^3$ . Intuitively,  $a(u,v)$  holds in  $G = (V,E,R,\eta)$  if  $\eta (u,v) = a$ , and  $\exists^{\geq N}z(\varphi (u,z)\wedge r(z,v))$  holds in  $G$  if  $v$  has at least  $N$  incoming edges labeled  $r\in R$  from nodes  $w$  for which  $\varphi (u,w)$  holds in  $G$ . We use the acronym  $\mathsf{rFO}_{\mathrm{cnt}}^3$  as this logic corresponds to a restriction of FO with three variables and counting. We can show the following result which is the first of its kind in the context of knowledge graphs:
Theorem 5.2. A logical binary classifier is captured by C-MPNNs without global readout if and only if it can be expressed in  $\mathsf{rFO}_{\mathrm{cnt}}^3$
The idea of the proof is to show a logical characterization for R-MPNNs (without global readout) in terms of a variant of graded modal logic called  $rFO_{cnt}^2$  (Theorem A.11), which generalizes results from Barceló et al. [3] to the case of multiple relations. Then, as in the case of Theorem 5.1, we apply a reduction from C-MPNNs to R-MPNNs (without global readout) using an auxiliary knowledge graph and a useful translation between the logics  $rFO_{cnt}^2$  and  $rFO_{cnt}^3$ .
Interestingly, arbitrary C-MPNNs (with global readout) are strictly more powerful than C-MPNNs without global readout in capturing logical binary classifiers: they can at least capture a strict extension of  $\mathsf{rFO}_{\mathrm{cnt}}^3$ , denoted by  $\mathsf{erFO}_{\mathrm{cnt}}^3$ .
Theorem 5.3. Each logical binary classifier expressible in  $\mathsf{erFO}_{\mathrm{cnt}}^3$  can be captured by a C-MPNN.
Intuitively speaking, our logic  $\mathsf{rFO}_{\mathrm{cnt}}^{3}$  from Theorem 5.2 only allows us to navigate the graph by moving to neighbors of the "current node". The logic  $\mathsf{erFO}_{\mathrm{cnt}}^{3}$  is a simple extension that allows us to move also to non-neighbors (Proposition A.15 shows that this logic actually gives us more power). Adapting the translation from logic to GNNs (from Theorem A.11), we can easily show that C-MPNNs with global readout can capture this extended logic.
The precise definition of  $\mathsf{erFO}_{\mathrm{cnt}}^3$  together with the proofs of Theorems 5.2 and 5.3 can be found in Appendices A.3 and A.4, respectively. Let us stress that  $\mathsf{rFO}_{\mathrm{cnt}}^3$  and  $\mathsf{erFO}_{\mathrm{cnt}}^3$  correspond to some binary variants of graded modal logic [8, 19]. As in Barceló et al. [3], these connections are exploited in our proofs.
5.3 Locating  $\mathrm{rawl}_2$  in the relational WL landscape
Let us note that  $\mathsf{rawl}_2$  strictly contains  $\mathsf{rw}l_1$ , since intuitively, we can degrade the  $\mathsf{rawl}_2$  test to compute unary invariants such that it coincides with  $\mathsf{rw}l_1$ . As a result, this allows us to conclude that R-MPNNs are less powerful than C-MPNNs. The  $\mathsf{rawl}_2$  test itself is upper bounded by a known relational variant of 2-WL, namely, the relational (symmetric) local 2-WL test, denoted by  $\mathsf{rw}l_2$ . Given a knowledge
graph  $G = (V,E,R,c,\eta)$ , this test assigns pairwise colors via the following update rule:
$$
\begin{array}{l} \left. \mathsf {r w l} _ {2} ^ {(t + 1)} (u, v) = \tau \big (\mathsf {r w l} _ {2} ^ {(t)} (u, v), \{\{((\mathsf {r w l} _ {2} ^ {(t)} (w, v), r) \mid w \in \mathcal {N} _ {r} (u), r \in R) \} \}, \right. \\ \left. \left\{\left(\operatorname {r w l} _ {2} ^ {(t)} (u, w), r) \mid w \in \mathcal {N} _ {r} (v), r \in R) \right\} \right\}\right) \\ \end{array}
$$
This test and a corresponding neural architecture, for arbitrary order  $k \geq 2$ , have been recently studied in Barceló et al. [4] under the name of multi-relational local  $k$ -WL.
This helps us to locate the test  $\mathrm{rawl}_2$  within the broader WL hierarchy, but it does not align perfectly with the practical setup: one common practice in link prediction is to extend knowledge graphs with inverse relations [40, 35, 25, 32] which empirically yields stronger results and hence is used in most practical setups. However, the effect of this choice has never been quantified formally. We formally explain the benefits of this design choice, showing that it leads to provably more powerful models. The idea is to consider tests (and architectures) which are augmented with the inverse edges: we write  $\mathrm{rawl}_2^+$  and  $\mathrm{rwl}_2^+$  to denote the corresponding augmented tests and prove that it results in more expressive tests (and hence architectures) in each case, respectively. The precise tests and propositions, along with their proofs can be found in Appendix A.5. We present in Figure 2 the resulting expressiveness hierarchy for all these tests.
Figure 2: Expressiveness hierarchy:  $A \to B$  iff  $A \preceq B$ . By Proposition A.20,  $\mathsf{rw}l_{2}$  and  $\mathsf{raw}l_{2}^{+}$  are incomparable. The case of  $\mathsf{rw}l_{2}^{+} \preceq \mathsf{raw}l_{2}^{+}$  is analogous to Proposition A.18.
6 Experimental evaluation
We experiment on knowledge graph benchmarks and aim to answer the following questions: Q1. What is the impact of the history function on the model performance? In particular, do models with  $f(t) = t$  perform comparably to those with  $f(t) = 0$  as our theory suggests? Q2. How do the specific choices for aggregation and message functions affect the performance? Q3. What is the impact of the initialization function on the performance? What happens when the target identifiability property does not hold? Q4. Do C-MPNNs outperform R-MPNNs empirically? Q5. Does the use of a global readout, or a relational version affect the performance?
Datasets. We use the datasets WN18RR [31] and FB15k-237 [9], for inductive relation prediction tasks, following a standardized train-test split given in four versions [30]. We augment each fact  $r(u,v)$  with an inverse fact  $r^{-1}(v,u)$ . There are no node features for either of the datasets, and the initialization is given by the respective initialization function INIT. This allows all the proposed GNN models to be applied in the inductive setup and to better align with the corresponding relational Weisfeiler-Leman algorithms. The statistics of the datasets are reported in Table 5 of Appendix C.1. The code for experiments is reported in https://github.com/HxyScottthuang/CMPNN.
Implementation. All models use 6 layers, each with 32 hidden dimensions. The decoder function parameterizes the probability of a fact  $q(u,v)$  as  $p(v\mid u,q) = \sigma (f(\pmb{h}_{v\mid u,q}^{(T)}))$ , where  $\sigma$  is the sigmoid function, and  $f$  is a 2-layer MLP with 64 hidden dimensions. We adopted layer-normalization [2] and short-cut connection after each aggregation and before applying ReLU. For the experiments concerning the message function  $\mathrm{Msg}_r^3$ , we follow the basis decomposition for the FB15k-237 dataset with 30 basis functions for sum aggregation, and 15 for PNA aggregation. We ran the experiments for 20 epochs on 1 Tesla T4 GPU, with mild modifications to accommodate all architectures studied in this paper. We discard the edges that directly connect query node pairs to prevent overfitting. The best checkpoint for each model instance is selected based on its performance on the validation set. All hyperparameter details are reported in Table 6 of Appendix C.1.
Evaluation. We consider filtered ranking protocol [6]: for each test fact  $r(u, v)$ , we construct 50 negative samples  $r(u', v')$ , randomly replacing either the head entity or the tail entity, and we report Hits@10, the rate of correctly predicted entities appearing in the top 10 entries for each instance list prediction. We report averaged results of five independent runs for all experiments.
Table 1: Inductive relation prediction with C-MPNNs using  $\mathrm{INIT}^2$  initialization and no readout. The best results for each category are shown in bold and the second best results are underlined.
Model architectures	WN18RR	FB15k-237
AGG	MSGr	f(t)	v1	v2	v3	v4	v1	v2	v3	v4
sum	MSG1r	0	0.934	0.896	0.894	0.881	0.784	0.900	0.940	0.923
sum	MSG1r	t	0.932	0.896	0.900	0.881	0.794	0.906	0.947	0.933
sum	MSG2r	0	0.939	0.906	0.881	0.881	0.734	0.899	0.911	0.941
sum	MSG2r	t	0.937	0.906	0.865	0.884	0.728	0.883	0.929	0.931
sum	MSG3r	0	0.943	0.898	0.888	0.877	0.850	0.934	0.919	0.941
sum	MSG3r	t	0.934	0.896	0.892	0.880	0.844	0.943	0.926	0.950
PNA	MSG1r	0	0.943	0.897	0.898	0.886	0.801	0.945	0.934	0.960
PNA	MSG1r	t	0.941	0.895	0.904	0.886	0.804	0.949	0.945	0.954
PNA	MSG2r	0	0.946	0.900	0.896	0.887	0.715	0.896	0.887	0.886
PNA	MSG2r	t	0.947	0.902	0.901	0.888	0.709	0.899	0.875	0.894
PNA	MSG3r	0	0.947	0.898	0.899	0.884	0.788	0.908	0.906	0.927
PNA	MSG3r	t	0.944	0.897	0.894	0.882	0.795	0.916	0.908	0.926
6.2 Results for inductive link prediction with C-MPNN architectures
We report inductive link prediction results for different C-MPNN architectures in Table 1, all initialized with  $\mathrm{INIT}^2$ . Each row of Table 1 corresponds to a specific architecture, which allows us to compare the model components. Note that while NBFNets [40] use different message functions for different datasets, we separately report for each model architecture to specifically pinpoint the impact of each model component.
History functions (Q1). First, we note that there is no significant difference between the models with different history functions. Specifically, for any choice of aggregate and message functions, the model which sets  $f(t) = t$  performs comparably to the one which sets  $f(t) = 0$ . This supports our theoretical findings, which state that path-based message passing and traditional message passing have the same expressive power. This may appear as a subtle point, but it is important for informing future work: the strength of these architectures is fundamentally due to their ability to compute more expressive binary invariants, which holds regardless of the choice of the history function.
Message functions (Q2). We highlight that there is no significant difference between different message functions on WN18RR, which is unsurprising: WN18RR splits contain at most 11 relation types, which undermines the impact of the differences in message functions. In contrast, the results on FB15k-237 are informative in this respect:  $\mathrm{Msg}_r^2$  clearly leads to worse performance than all the other choices, which can be explained by the fact that  $\mathrm{Msg}_r^2$  utilizes fewer relation-specific parameters. Importantly,  $\mathrm{Msg}_r^3$  appears strong and robust across models. This is essentially the message function of RGCN and uses basis decomposition to regularize the parameter matrices. Architectures using  $\mathrm{Msg}_r^3$  with fewer parameters (see the appendix) can match or substantially exceed the performance of the models using  $\mathrm{Msg}_r^1$ , where the latter is the primary message function used in NBFNets. This may appear counter-intuitive since  $\mathrm{Msg}_r^3$  does not have a learnable query vector  $z_{q}$ , but this vector is nonetheless part of the model via the initialization function  $\mathrm{INIT}^2$ .
Aggregation functions (Q2). We experimented with aggregation functions sum and PNA. We do not observe significant trends on WN18RR, but PNA tends to result in slightly better-performing architectures. On FB15k-237, there seems to be an intricate interplay between aggregation and message functions. For  $\mathrm{Msg}_r^1$ , PNA appears to be a better choice than sum. On the other hand, for both  $\mathrm{Msg}_r^2$  and  $\mathrm{Msg}_r^3$ , sum aggregation is substantially better. This suggests that a sophisticated aggregation, such as PNA, may not always be necessary since it can be matched (and even outperformed) with a sum aggregation. In fact, the model with sum aggregation and  $\mathrm{Msg}_r^3$  is very closely related to RGCN and appears to be one of the best-performing models across the board. This supports our theory since, intuitively, this model can be seen as an adaptation of RGCN to compute binary invariants while keeping the choices for model components the same as RGCN.
Table 2: Results for inductive relation prediction on WN18RR and FB15k-237 using no readout, global readout and relation-specific readout, respectively. We use C-MPNN architecture with AGG = PNA, MSG =  $\mathsf{MSG}_r^1$ , and  $\mathrm{INIT} = \mathrm{INIT}^2$
Model architectures
READ	WN18RR	FB15k-237
v1	v2	v3	v4	v1	v2	v3	v4
no readout	0.941	0.895	0.904	0.886	0.804	0.949	0.945	0.954
global readout	0.946	0.890	0.881	0.872	0.799	0.897	0.942	0.864
relation-specific	0.932	0.885	0.882	0.874	0.835	0.959	0.953	0.960
We conducted further experiments to understand the differences in different initializations at Appendix C.2, confirming that the initializations that do not satisfy the target node distinguishability criteria perform significantly worse (Q3). We could not include R-MPNNs (e.g., RGCN and CompGCN) in these experiments, since these architectures are not applicable to the fully inductive setup. However, we conducted experiments on transductive link prediction in Appendix D.1, and carried out additional experiments on biomedical knowledge in Appendix D.2. In both experiments, we observed that C-MPNNs outperform R-MPNNs by a significant margin (Q4).
6.3 Empirically evaluating the effect of readout
To understand the effect of readout (Q5), we experiment with C-MPNNs using  $\mathrm{INIT}^2$ , PNA,  $\mathrm{MSG}_r^1$ , and  $f(t) = t$ . We consider three architectures: (i) no readout component, (ii) global readout component (as defined earlier), and finally (iii) relation-specific readout. The idea behind the relation-specific readout is natural: instead of summing over all node representations, we sum over the features of nodes  $w$  which have an incoming  $q$ -edge ( $w \in V_q^+$ ) or an outgoing  $q$ -edge ( $w \in V_q^-$ ), where  $q$  is the query relation, as:
$$
\operatorname {R E A D} \left( \right.\left\{ \right.\left\{\boldsymbol {h} _ {w | u, q} ^ {(t)} \mid w \in V _ {q} ^ {+} \right\}, \left\{\left\{\boldsymbol {h} _ {w | u, q} ^ {(t)} \mid w \in V _ {q} ^ {-} \right.\right\}\left. \right) = \mathbf {W} _ {1} ^ {(t)} \sum_ {w \in V _ {q} ^ {+}} \boldsymbol {h} _ {w | u, q} ^ {(t)} + \mathbf {W} _ {2} ^ {(t)} \sum_ {w \in V _ {q} ^ {-}} \boldsymbol {h} _ {w | u, q} ^ {(t)}.
$$
We report all results in Table 2. First, we observe that the addition of global readout does not lead to significant differences on WN18RR, but it degrades the model performance on FB15k-237 in two splits (v2 and v4). We attribute this to having more relation types in FB15k-237, so a readout over all nodes does not necessarily inform us about a relation-specific prediction. This is the motivation behind the relation-specific readout: we observe that it leads to significant improvements compared to using no readout (or a global readout) over all splits, leading to state-of-the-art results. In contrast, relation-specific readout does not yield improvements on the relationally sparse WN18RR. These findings suggest that the global information can be very useful in practice, assuming the graph is rich in relation types. We further conduct a synthetic experiment to validate the expressive power of using global readout: architectures without global readout cannot exceed random performance, while those using a global readout can solve the synthetic link prediction task (see Appendix E).
7 Outlook and limitations
We studied a broad general class of GNNs designed for link prediction over knowledge graphs with a focus on formally characterizing their expressive power. Our study shows that recent state-of-the-art models are at a 'sweet spot': while they compute binary invariants, they do so through some local variations of higher-dimensional WL tests. This is precisely characterized within our framework. To capture global properties, we propose readout components and show that their relational variant is useful on datasets with many relation types. This study paints a more complete picture for our overall understanding of existing models and informs future work on possible directions.
In terms of limitations, it is important to note that the presented approach has certain restrictions. For instance, it is limited to binary tasks, such as link prediction. Thus, this approach does not directly extend to higher arity tasks, e.g., link prediction on relational hypergraphs or predicting the existence of substructures of size  $k$  in a knowledge graph. When considering local or conditional variations of the  $k$ -WL algorithm, computational demands can become a significant challenge. Further research is needed to explore potential optimizations for higher-order MPNNs that can enhance their performance while preserving their expressive power.
The authors would like to acknowledge the use of the University of Oxford Advanced Research Computing (ARC) facility in carrying out this work (http://dx.doi.org/10.5281/zenodo.22558). We would also like to thank Google Cloud for kindly providing computational resources. Barceló is funded by ANID-Millennium Science Initiative Program - CodeICN17002. Romero is funded by Fondecyt grant 11200956. Barceló and Romero are funded by the National Center for Artificial Intelligence CENIA FB210017, BasalANID.
[1] Ralph Abboud, Ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. In *IJCAI*, 2021.
[2] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. In CoRR, 2016.
[3] Pablo Barceló, Egor V. Kostylev, Mikaël Monet, Jorge Pérez, Juan L. Reutter, and Juan Pablo Silva. The logical expressiveness of graph neural networks. In ICLR, 2020.
[4] Pablo Barceló, Mikhail Galkin, Christopher Morris, and Miguel Romero. Weisfeiler and leman go relational. In LoG, 2022.
[5] Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In NIPS, 2016.
[6] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, 2013.
[7] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lio, and Petar Velicković. Principal neighbourhood aggregation for graph nets. In NeurIPS, 2020.
[8] Maarten de Rijke. A note on Graded Modal Logic. In Stud Logica, 2000.
[9] Tim Dettmers, Minervini Pasquale, Stenetorp Pontus, and Sebastian Riedel. Convolutional 2D knowledge graph embeddings. In AAAI, 2018.
[10] Luis Antonio Galárraga, Christina Teflioudi, Katja Hose, and Fabian Suchanek. AMIE: Association rule mining under incomplete evidence in ontological knowledge bases. In WWW, 2013.
[11] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In ICML, 2017.
[12] Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In IJCNN, 2005.
[13] Martin Grohe. The logic of graph neural networks. In LICS, 2021.
[14] Aditya Grover and Jure Leskovec. Node2vec: Scalable feature learning for networks. In KDD, 2016.
[15] Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. In eLife, 2017.
[16] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS, 2020.
[17] Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.