Title: A Theory of Link Prediction via Relational Weisfeiler-Leman on Knowledge Graphs
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural networks

knowledge graphs

expressivity

logical characterization
Abstract: Graph neural networks are prominent models for representation learning over graph-structured data. While the capabilities and limitations of these models are well-understood for simple graphs, our understanding remains incomplete in the context of knowledge graphs. Our goal is to provide a systematic understanding of the landscape of graph neural networks for knowledge graphs pertaining to the prominent task of link prediction. Our analysis entails a unifying perspective on seemingly unrelated models and unlocks a series of other models. The expressive power of various models is characterized via a corresponding relational Weisfeiler-Leman algorithm. 
This analysis is extended to provide a precise logical characterization of the class of functions captured by a class of graph neural networks. The theoretical findings presented in this paper explain the benefits of some widely employed practical design choices, which are validated empirically.

A Theory of Link Prediction via Relational Weisfeiler-Leman on Knowledge Graphs

Abstract

Graph neural networks are prominent models for representation learning over graph-structured data. While the capabilities and limitations of these models are well-understood for simple graphs, our understanding remains incomplete in the context of knowledge graphs. Our goal is to provide a systematic understanding of the landscape of graph neural networks for knowledge graphs pertaining to the prominent task of link prediction. Our analysis entails a unifying perspective on seemingly unrelated models and unlocks a series of other models. The expressive power of various models is characterized via a corresponding relational Weisfeiler-Leman algorithm. This analysis is extended to provide a precise logical characterization of the class of functions captured by a class of graph neural networks. The theoretical findings presented in this paper explain the benefits of some widely employed practical design choices, which are validated empirically.

1 Introduction

Graph neural networks (GNNs) [27, 12] are prominent models for representation learning over graphstructured data, where the idea is to iteratively compute vector representations of nodes of an input graph through a series of invariant (resp., equivariant) transformations. While the landscape of GNNs is overwhelmingly rich, the vast majority of such models are instances of message passing neural networks [11] which are well-studied, leading to a theoretical understanding of their capabilities and limitations [34, 21]. In turn, our understanding is rather limited for GNN models dedicated to learning over knowledge graphs , which are applied in a wide range of domains.

To make our context precise, we first consider an extension of message passing neural networks with relation-specific message functions, which we call relational message passing neural networks. Two prominent examples of this framework are RGCN [28] and CompGCN [32], and their expressive power has recently been characterized through a dedicated relational Weisfeiler-Leman test [4].

While offering principled means for learning over knowledge graphs, the standard relational message passing framework is tailored for computing unary node representations and therefore models of this class are better suited for node-level tasks (e.g., node/entity classification). Actually, it is well-known that even a good node-level representation might not necessarily induce a good edge representation,

Miguel Romero

Pablo Barceló

hindering the applicability of such models for the crucial task of link prediction [39]. This has led to the design of GNN architectures specifically tailored for link prediction over knowledge graphs [30, 40, 33, 18], for which our understanding remains limited.

The goal of this paper is to offer a theory of the capabilities and limitations of a class of relational GNN architectures which compute pairwise node representations to be utilized for link prediction. Although most such architectures can be seen to be subsumed by higher-order message passing neural networks that compute pairwise representations on nodes, the inherently quadratic behavior of the latter justifies local approximations which align better with models used in practice. Of particular interest to us is Neural Bellman-Ford Networks (NBFNets) [40], which define a message passing approach inspired by the Bellman-Ford algorithm. We argue in salient detail that the crucial insight of this approach is in leveraging the idea of computing conditional pairwise-node representations, which leads to more expressive models at a more reasonable computational cost, given its local nature.

Building on this fundamental aspect, we define conditional message passing neural networks, which extend traditional ones by a conditional message passing paradigm: every node representation is conditional on a source node and a query relation, which allows for computing pairwise node representations. This framework strictly contains NBFNets and allows for a systematic treatment of various other models. Through a careful study of this framework, we can explain the conceptual differences between different models along with their respective expressive power.

Our contributions can be summarized as follows:

We introduce conditional message passing neural networks which encode representations of nodes v conditioned on a (source) node u and a query relation q , yielding pairwise node representations. We discuss the model design space, including a discussion on different initialization regimes and the presence of global readout functions in each layer of the network.

We define a relational Weisfeiler-Leman algorithm (building on similar works such as Barceló et al. [4]), and prove that conditional message passing neural networks can match the expressive power of this algorithm. This study reveals interesting insights about NBFNets, suggesting that their strong empirical performance is precisely due to the expressive power, which can be matched by other instances of this framework.

Viewing conditional message passing neural networks (with or without global readouts) as classifiers over pairs of nodes, we give logical characterizations of their expressive power based on formulas definable in some binary variants of graded modal logic [8, 19]. This provides us with a declarative and well-studied formal counterpart of C-MPNNs.

We conduct an experimental analysis to verify the impact of various model choices, particularly pertaining to initialization, history, message computation, and global readout functions. We also conduct both inductive and transductive experiments on various real-world datasets, empirically validating our theoretical findings.

2 Related work and motivation

Early GNNs for knowledge graphs are relational variations of message passing neural networks. A prototypical example is the RGCN architecture [28], which extends graph convolutional networks (GCNs) [17] with relation-specific message functions. CompGCN [32] and several other architectures [37] follow this line of work with differences in their aggregate, update, and message functions. These architectures encode unary node representations and typically rely on a pairwise decoder function to predict the likelihood of a link which is known to be suboptimal for link prediction [39]. There is a good understanding of the expressive power of these architectures [4]: we generalize these results in Section 3, since they form the basis for the rest of our results.

A different approach is given for single-relational graphs by SEAL [38], where the idea is to encode (labeled) subgraphs (instead of nodes). GraIL [30] extends this idea to knowledge graphs, and one important virtue of these models is that they are inductive even if there are no node features in the input graph. The idea is to use a form of labeling trick [39] based on pairwise shortest path distances in sampled subgraphs, but these architectures suffer from scalability issues. More recent inductive architectures integrate ideas from earlier path-based link prediction approaches [24, 14] into modern GNN architectures, resulting in proposals such as PathCon [33], Geodesic GNNs [18], and NBFNets [40]. Our study is very closely related to NBFNets which is inspired by the generalized

version of the Bellman-Ford algorithm for finding shortest paths. These architectures aggregate over relational paths by keeping track of conditional pairwise-node representations. While NBFNets can be intuitively seen as the neural counterpart of the generalized Bellman-Ford algorithm, they do not provably align with this algorithm since the 'semiring assumption' is invalidated through the use of non-linearities (which is explicit in Zhu et al. [40]). This leaves open many questions regarding the capabilities and limitations of these architectures.

In this paper, we argue that the key insight behind architectures such as NBFNets is in locally computing pairwise representations through conditioning on a source node, and this has roots in earlier works, such as ID-GNNs [36]. To formally study this, we introduce conditional message passing neural networks as a strict generalization of NBFNets [40] and related models such as NeuralLP [35], or DRUM [25]. Through this abstraction, we theoretically study the properties of a large class of models in relation to local variants of relational Weisfeiler-Leman algorithms. Broadly, our study can be seen as the relational counterpart of the expressiveness studies conducted for GNNs [34, 21, 3], particularly related to higher-order GNNs [21], which align with higher-order dimensional variants of the WL test. Our characterization relies on local versions of higher-order WL tests [22], albeit not in a relational context. This can be seen as a continuation and generalization of the results given for relational message passing neural networks [4] to a broader class of models.

3 Background

3.1 Knowledge graphs and invariants

Knowledge graphs. A knowledge graph is a tuple G = ( V, E, R, c ) , where V is a set of nodes, E ⊆ R × V × V is a set of labeled edges, or facts, R is the set of relation types and c : V → D is a node coloring. When D = R d , we also say that c is a d -dimensional feature map , and typically use x instead of c . We write r ( u, v ) to denote a labeled edge, or a fact, where r ∈ R and u, v ∈ V . The neighborhood of a node v ∈ V relative to a relation r ∈ R is defined as N r ( v ) := { u | r ( u, v ) ∈ E } .

Graph invariants. We define k -ary graph invariants following the terminology of Grohe [13], for which we first define isomorphism over knowledge graphs. An isomorphism from a knowledge graph G = ( V, E, R, c ) to a knowledge graph G ′ = ( V ′ , E ′ , R, c ′ ) is a bijection f : V → V ′ such that c ( v ) = c ′ ( f ( v )) for all v ∈ V , and r ( u, v ) ∈ E if and only if r ( f ( u ) , f ( v )) ∈ E ′ , for all r ∈ R and u, v ∈ V . A 0 -ary graph invariant is a function ξ defined on knowledge graphs such that ξ ( G ) = ξ ( G ′ ) for all isomorphic knowledge graphs G and G ′ . For k ≥ 1 , a k -ary graph invariant is a function ξ that associates with each knowledge graph G = ( V, E, R, c ) a function ξ ( G ) defined on V k such that for all knowledge graphs G and G ′ , all isomorphisms f from G to G ′ , and all k -tuples of nodes v ∈ V k , it holds that ξ ( G )( v ) = ξ ( G ′ )( f ( v )) . If k = 1 , this defines a node invariant , or unary invariant , and if k = 2 , this defines a binary invariant , which is central to our study.

Refinements. A function ξ ( G ) : V k → D refines a function ξ ′ ( G ) : V k → D , denoted as ξ ( G ) ⪯ ξ ′ ( G ) , if for all v , v ′ ∈ V k , ξ ( G )( v ) = ξ ( G )( v ′ ) implies ξ ′ ( G )( v ) = ξ ′ ( G )( v ′ ) . We call such functions equivalent , denoted as ξ ( G ) ≡ ξ ′ ( G ) , if ξ ( G ) ⪯ ξ ′ ( G ) and ξ ′ ( G ) ⪯ ξ ( G ) . A k -ary graph invariant ξ refines a k -ary graph invariant ξ ′ , if ξ ( G ) refines ξ ′ ( G ) for all knowledge graphs G .

3.2 Relational message passing neural networks

We introduce relational message passing neural networks (R-MPNNs) , which encompass several known models such as RGCN [28] and CompGCN [32]. The idea is to iteratively update the feature of a node v based on the different relation types r ∈ R and the features of the corresponding neighbors in N r ( v ) . In our most general model we also allow readout functions, that allow further updates to the feature of v by aggregating over the features of all nodes in the graph.

Let G = ( V, E, R, x ) be a knowledge graph, where x is a feature map. An R-MPNN computes a sequence of feature maps h ( t ) : V → R d ( t ) , for t ≥ 0 . For simplicity, we write h ( t ) v instead of h ( t ) ( v ) . For each node v ∈ V , the representations h ( t ) v are iteratively computed as:

where UPD, AGG, READ, and MSG r are differentiable update , aggregation , global readout , and relation-specific message functions, respectively, and f : N → N is a history function 1 , which is always non-decreasing and satisfies f ( t ) ≤ t . An R-MPNN has a fixed number of layers T ≥ 0 , and then, the final node representations are given by the map h ( T ) : V → R d ( T ) .

The use of a readout component in message passing is well-known [5] but its effect is not wellexplored in a relational context. It is of interest to us since it has been shown that standard GNNs can capture a larger class of functions with global readout [3].

An R-MPNN can be viewed as an encoder function enc that associates with each knowledge graph G a function enc ( G ) : V → R d ( T ) , which defines a node invariant corresponding to h ( T ) . The final representations can be used for node-level predictions. For link-level tasks, we use a binary decoder dec q : R d ( T ) × R d ( T ) → R , which produces a score for the likelihood of the fact q ( u, v ) , for q ∈ R .

In Appendix A.1 we provide a useful characterization of the expressive power of R-MPNNs in terms of a relational variant of the Weisfeiler-Leman test [4]. This characterization is essential for the rest of the results that we present in the paper.

4 Conditional message passing neural networks

R-MPNNs have serious limitations for the task of link prediction [39], which has led to several proposals that compute pairwise representations directly. In contrast to the case of R-MPNNs, our understanding of these architectures is limited. In this section, we introduce the framework of conditional MPNNs that offers a natural framework for the systematic study of these architectures.

Let G = ( V, E, R, x ) be a knowledge graph, where x is a feature map. A conditional message passing neural network (C-MPNN) iteratively computes pairwise representations, relative to a fixed query q ∈ R and a fixed node u ∈ V , as follows:

where INIT, UPD, AGG, READ, and MSG r are differentiable initialization , update , aggregation , global readout , and relation-specific message functions, respectively, and f is the history function. We denote by h ( t ) q : V × V → R d ( t ) the function h ( t ) q ( u, v ) := h ( t ) v | u,q , and denote z q to be a learnable vector representing the query q ∈ R . A C-MPNN has a fixed number of layers T ≥ 0 , and the final pair representations are given by h ( T ) q . We sometimes write C-MPNNs without global readout to refer to the class of models which do not use a readout component.

Intuitively, C-MPNNs condition on a source node u in order to compute representations of ( u, v ) for all target nodes v . To further explain, we show a visualization of C-MPNNs and R-MPNNs in Figure 1 to demonstrate the differences in the forward pass. Contrary to the R-MPNN model where we carry out relational message passing first and rely on the binary decoder to compute a query fact q ( u, v ) , the C-MPNN model first initializes all node representations with the zero vector except the representation of the node u which is assigned a vector with non-zero entry. Following the initialization, we carry out relational message passing and decode the hidden state of the target node v to obtain the output, which yields the representation of v conditioned on u .

̸

Observe that C-MPNNs compute binary invariants, provided that the initialization INIT is a binary invariant. To ensure that the resulting model computes pairwise representations, we require INIT ( u, v, q ) to be a nontrivial function in the sense that it needs to satisfy target node distinguishability : for all q ∈ R and v = u ∈ V , it holds that INIT ( u, u, q ) = INIT ( u, v, q ) .

̸

This is very closely related to the Labeling Trick proposed by Zhang et al. [39], which is an initialization method aiming to differentiate a set { u, v } of target nodes from the remaining nodes in a graph. However, the Labeling Trick only applies when both the source u and target nodes v are labeled. Recent state-of-the-art models such as ID-GNN [36] and NBFNet [40] utilize a similar method, but only with the source node u labeled differently in initialization. Our definition captures precisely this, and we offer a theoretical analysis of the capabilities of the aforementioned architectures accordingly.

u

v

q

G

h

u

h

v

Message Passing

dec

q

(

,

)

→

R

z

q

0

INIT

(

u, v, q

)

h

v

|

u,q

Message Passing

dec

(

)

→

R

Encoder

Decoder

R-MPNN

C-MPNN

One alternative is to directly learn pairwise representations following similar ideas to those of higherorder GNNs, but these algorithms are not scalable. Architectures such as NBFNets represent a trade-off between computational complexity and expressivity. The advantage of learning conditional representations h v | u,q is to be able to learn such representations in parallel for all v ∈ V , amortizing the computational overhead; see Zhu et al. [40] for a discussion. We have also carried out a runtime analysis comparison among different classes of models in Appendix B.

4.1 Design space and basic model architectures

To specify a C-MPNN architecture, we need to specify the functions INIT, AGG, MSG r , f , and READ. In the following, we consider three initialization functions { INIT 1 , INIT 2 , INIT 3 } , two aggregation functions { sum , PNA } , three message functions { MSG 1 r , MSG 2 r , MSG 3 r } , two history functions { f ( t ) = t, f ( t ) = 0 } , and either sum global readout or no readout term.

Initialization. We consider the following natural variants for initialization:

where ∗ represents element-wise multiplication, the function 1 u = v ( v ) is the indicator function which returns the all-ones vector 1 if u = v and the all-zeros vector 0 otherwise with corresponding size.

Clearly, both INIT 1 and INIT 2 satisfy target node distinguishability assumption if we assume z q has no zero entry, where INIT 2 , in addition, allows query-specific initialization to be considered. Suppose we further relax the condition to be target node distinguishability in expectation . Then, INIT 3 can also distinguish between each conditioned node u given the same query vector z q by adding an error vector ϵ u sampled from N (0 , 1) to the conditioned node's initialization.

Aggregation. We consider sum aggregation and Principal Neighborhood Aggregation (PNA) [7].

Message. We consider the following variations of message functions:

where W ( t ) r are relation-specific transformations. MSG 1 r computes a query-dependent message, whereas MSG 2 r and MSG 3 r are analogous to message functions of CompGCN and RGCN, respectively.

History. In addition, we can set f , which intuitively is the function that determines the history of node embeddings to be considered. By setting f ( t ) = t , we obtain a standard message-passing algorithm where the update function considers the representation of the node in the previous iteration. We can alternatively set f ( t ) = 0 , in which case we obtain (a generalization of) NBFNets.

Readout. We consider a standard readout which sums the representations and applies a linear transformation on the resulting representations. Alternatively, we consider the special case, where we omit this component (we discuss a dedicated readout operation in our empirical analysis later).

A simple architecture. Consider a basic C-MPNN architecture with global readout, which, for a query relation q ∈ R and a fixed node u , updates the representations as:

where W ( t ) 0 , W ( t ) 1 are linear transformations followed by a non-linearity σ .

5 Characterizing the expressive power

5.1 A relational Weisfeiler-Leman characterization

̸

To analyze the expressive power of C-MPNNs, we introduce the relational asymmetric local 2 -WL , denoted by rawl 2 . In this case, we work with knowledge graphs of the form G = ( V, E, R, c, η ) , where η : V × V → D is a pairwise coloring . We say that η satisfies target node distinguishability if η ( u, u ) = η ( u, v ) for all u = v ∈ V . The notions of isomorphism and invariants extend to this context in a natural way. For each t ≥ 0 , we update the coloring as:

̸

where τ injectively maps the above pair to a unique color, which has not been used in previous iterations. Observe that rawl ( t ) 2 defines a binary invariant, for all t ≥ 0 .

The test is asymmetric: given a pair ( u, v ) , we only look at neighbors of ( u, v ) obtained by changing the second coordinate of the pair. In contrast, usual versions of (local) k -WL are symmetric as neighbors may change any coordinate. Interestingly, this test characterizes the power of C-MPNNs in terms of distinguishing pairs of nodes.

Theorem 5.1. Let G = ( V, E, R, x , η ) be a knowledge graph, where x is a feature map and η is a pairwise coloring satisfying target node distinguishability. Let q ∈ R be any query relation. Then:

For all C-MPNNs with T layers and initializations INIT with INIT ≡ η , and 0 ≤ t ≤ T , we have rawl ( t ) 2 ⪯ h ( t ) q .

For all T ≥ 0 and history function f , there is a C-MPNN without global readout with T layers and history function f such that for all 0 ≤ t ≤ T , we have rawl ( t ) 2 ≡ h ( t ) q .

The idea of the proof is as follows: we first show a correspondent characterization of the expressive power of R-MPNNs in terms of a relational variant of the WL test (Theorem A.1). This result generalizes results from Barceló et al. [4]. We then apply a reduction from C-MPNNs to R-MPNNs, that is, we carefully build an auxiliary knowledge graph (encoding the pairs of nodes of the original knowledge graph) to transfer our R-MPNN characterization to our sought C-MPNN characterization.

Note that the lower bound (item (2)) holds even for the basic model of C-MPNNs (without global readout) and the three proposed message functions from Section 4.1. The expressive power of CMPNNs is independent of the history function as in any case it is matched by rawl 2 . This suggests that the difference between traditional message passing models using functions f ( t ) = t and path-based models (such as NBFNets [40]) using f ( t ) = 0 is not relevant from a theoretical point of view.

5.2 Logical characterization

We now turn to the problem of which binary classifiers can be expressed as C-MPNNs. That is, we look at C-MPNNs that classify each pair of nodes in a knowledge graph as true or false . Following Barceló et al. [3], we study logical binary classifiers, i.e., those that can be defined in the formalism of first-order logic (FO). Briefly, a first-order formula ϕ ( x, y ) with two free variables x, y defines a logical binary classifier that assigns value true to the pair ( u, v ) of nodes in knowledge graph G whenever G | = ϕ ( u, v ) , i.e., ϕ holds in G when x is interpreted as u and y as v . A logical classifier ϕ ( x, y ) is captured by a C-MPNN A if over every knowledge graph G the pairs ( u, v ) of nodes that are classified as true by ϕ and A are the same.

A natural problem then is to understand what are the logical classifiers captured by C-MPNNs. Fix a set of relation types R and a set of pair colors C . We consider knowledge graphs of the form G = ( V, E, R, η ) where η is a mapping assigning colors from C to pairs of nodes from V . In this context, FO formulas can refer to the different relation types in R and the different pair colors in C . Our first characterization is established in terms of a simple fragment of FO, which we call rFO 3 cnt , and is inductively defined as follows: First, a ( x, y ) for a ∈ C , is in rFO 3 cnt . Second, if φ ( x, y ) and ψ ( x, y ) are in rFO 3 cnt , N ≥ 1 is a positive integer, and r ∈ R , then the formulas

are also in rFO 3 cnt . Intuitively, a ( u, v ) holds in G = ( V, E, R, η ) if η ( u, v ) = a , and ∃ ≥ N z ( φ ( u, z ) ∧ r ( z, v )) holds in G if v has at least N incoming edges labeled r ∈ R from nodes w for which φ ( u, w ) holds in G . We use the acronym rFO 3 cnt as this logic corresponds to a restriction of FO with three variables and counting. We can show the following result which is the first of its kind in the context of knowledge graphs:

Theorem 5.2. A logical binary classifier is captured by C-MPNNs without global readout if and only if it can be expressed in rFO 3 cnt .

The idea of the proof is to show a logical characterization for R-MPNNs (without global readout) in terms of a variant of graded modal logic called rFO 2 cnt (Theorem A.11), which generalizes results from Barceló et al. [3] to the case of multiple relations. Then, as in the case of Theorem 5.1, we apply a reduction from C-MPNNs to R-MPNNs (without global readout) using an auxiliary knowledge graph and a useful translation between the logics rFO 2 cnt and rFO 3 cnt .

Interestingly, arbitrary C-MPNNs (with global readout) are strictly more powerful than C-MPNNs without global readout in capturing logical binary classifiers: they can at least capture a strict extension of rFO 3 cnt , denoted by erFO 3 cnt .

Theorem 5.3. Each logical binary classifier expressible in erFO 3 cnt can be captured by a C-MPNN.

Intuitively speaking, our logic rFO 3 cnt from Theorem 5.2 only allows us to navigate the graph by moving to neighbors of the 'current node'. The logic erFO 3 cnt is a simple extension that allows us to move also to non-neighbors (Proposition A.15 shows that this logic actually gives us more power). Adapting the translation from logic to GNNs (from Theorem A.11), we can easily show that C-MPNNs with global readout can capture this extended logic.

The precise definition of erFO 3 cnt together with the proofs of Theorems 5.2 and 5.3 can be found in Appendices A.3 and A.4, respectively. Let us stress that rFO 3 cnt and erFO 3 cnt correspond to some binary variants of graded modal logic [8, 19]. As in Barceló et al. [3], these connections are exploited in our proofs.

5.3 Locating rawl 2 in the relational WL landscape

Let us note that rawl 2 strictly contains rwl 1 , since intuitively, we can degrade the rawl 2 test to compute unary invariants such that it coincides with rwl 1 . As a result, this allows us to conclude that R-MPNNs are less powerful than C-MPNNs. The rawl 2 test itself is upper bounded by a known relational variant of 2 -WL, namely, the relational (symmetric) local 2-WL test, denoted by rwl 2 . Given a knowledge

graph G = ( V, E, R, c, η ) , this test assigns pairwise colors via the following update rule:

This test and a corresponding neural architecture, for arbitrary order k ≥ 2 , have been recently studied in Barceló et al. [4] under the name of multi-relational local k -WL .

This helps us to locate the test rawl 2 within the broader WL hierarchy, but it does not align perfectly with the practical setup: one common practice in link prediction is to extend knowledge graphs with inverse relations [40, 35, 25, 32] which empirically yields stronger results and hence is used in most practical setups. However, the effect of this choice has never been quantified formally. We formally explain the benefits of this design choice, showing that it leads to provably more powerful models. The idea is to consider tests (and architectures) which are augmented with the inverse edges: we write rawl + 2 and rwl + 2 to denote the corresponding augmented tests and prove that it results in more expressive tests (and hence architectures) in each case, respectively. The precise tests and propositions, along with their proofs can be found in Appendix A.5. We present in Figure 2 the resulting expressiveness hierarchy for all these tests.

6 Experimental evaluation

We experiment on knowledge graph benchmarks and aim to answer the following questions: Q1. What is the impact of the history function on the model performance? In particular, do models with f ( t ) = t perform comparably to those with f ( t ) = 0 as our theory suggests? Q2. How do the specific choices for aggregation and message functions affect the performance? Q3. What is the impact of the initialization function on the performance? What happens when the target identifiability property does not hold? Q4. Do C-MPNNs outperform R-MPNNs empirically? Q5. Does the use of a global readout, or a relational version affect the performance?

6.1 Experimental setup

Datasets. We use the datasets WN18RR [31] and FB15k-237 [9], for inductive relation prediction tasks, following a standardized train-test split given in four versions [30]. We augment each fact r ( u, v ) with an inverse fact r -1 ( v, u ) . There are no node features for either of the datasets, and the initialization is given by the respective initialization function INIT. This allows all the proposed GNN models to be applied in the inductive setup and to better align with the corresponding relational Weisfeiler-Leman algorithms. The statistics of the datasets are reported in Table 5 of Appendix C.1. The code for experiments is reported in https://github.com/HxyScotthuang/CMPNN .

Implementation. All models use 6 layers, each with 32 hidden dimensions. The decoder function parameterizes the probability of a fact q ( u, v ) as p ( v | u, q ) = σ ( f ( h ( T ) v | u,q )) , where σ is the sigmoid function, and f is a 2-layer MLP with 64 hidden dimensions. We adopted layer-normalization [2] and short-cut connection after each aggregation and before applying ReLU. For the experiments concerning the message function MSG 3 r , we follow the basis decomposition for the FB15k-237 dataset with 30 basis functions for sum aggregation, and 15 for PNA aggregation. We ran the experiments for 20 epochs on 1 Tesla T4 GPU. with mild modifications to accommodate all architectures studied in this paper. We discard the edges that directly connect query node pairs to prevent overfitting. The best checkpoint for each model instance is selected based on its performance on the validation set. All hyperparameter details are reported in Table 6 of Appendix C.1.

Evaluation. We consider filtered ranking protocol [6]: for each test fact r ( u, v ) , we construct 50 negative samples r ( u ′ , v ′ ) , randomly replacing either the head entity or the tail entity, and we report Hits@10, the rate of correctly predicted entities appearing in the top 10 entries for each instance list prediction. We report averaged results of five independent runs for all experiments.

6.2 Results for inductive link prediction with C-MPNN architectures

We report inductive link prediction results for different C-MPNN architectures in Table 1, all initialized with INIT 2 . Each row of Table 1 corresponds to a specific architecture, which allows us to compare the model components. Note that while NBFNets [40] use different message functions for different datasets, we separately report for each model architecture to specifically pinpoint the impact of each model component.

History functions (Q1). First, we note that there is no significant difference between the models with different history functions. Specifically, for any choice of aggregate and message functions, the model which sets f ( t ) = t performs comparably to the one which sets f ( t ) = 0 . This supports our theoretical findings, which state that path-based message passing and traditional message passing have the same expressive power. This may appear as a subtle point, but it is important for informing future work: the strength of these architectures is fundamentally due to their ability to compute more expressive binary invariants , which holds regardless of the choice of the history function.

Message functions (Q2). We highlight that there is no significant difference between different message functions on WN18RR, which is unsurprising: WN18RR splits contain at most 11 relation types, which undermines the impact of the differences in message functions. In contrast, the results on FB15k-237 are informative in this respect: MSG 2 r clearly leads to worse performance than all the other choices, which can be explained by the fact that MSG 2 r utilizes fewer relation-specific parameters. Importantly, MSG 3 r appears strong and robust across models. This is essentially the message function of RGCN and uses basis decomposition to regularize the parameter matrices. Architectures using MSG 3 r with fewer parameters (see the appendix) can match or substantially exceed the performance of the models using MSG 1 r , where the latter is the primary message function used in NBFNets. This may appear counter-intuitive since MSG 3 r does not have a learnable query vector z q , but this vector is nonetheless part of the model via the initialization function INIT 2 .

Aggregation functions (Q2). We experimented with aggregation functions sum and PNA . We do not observe significant trends on WN18RR, but PNA tends to result in slightly better-performing architectures. On FB15k-237, there seems to be an intricate interplay between aggregation and message functions. For MSG 1 r , PNA appears to be a better choice than sum . On the other hand, for both MSG 2 r and MSG 3 r , sum aggregation is substantially better. This suggests that a sophisticated aggregation, such as PNA , may not always be necessary since it can be matched (and even outperformed) with a sum aggregation. In fact, the model with sum aggregation and MSG 3 r is very closely related to RGCN and appears to be one of the best-performing models across the board. This supports our theory since, intuitively, this model can be seen as an adaptation of RGCN to compute binary invariants while keeping the choices for model components the same as RGCN.

We conducted further experiments to understand the differences in different initializations at Appendix C.2, confirming that the initializations that do not satisfy the target node distinguishability criteria perform significantly worse ( Q3 ). We could not include R-MPNNs (e.g., RGCN and CompGCN) in these experiments, since these architectures are not applicable to the fully inductive setup. However, we conducted experiments on transductive link prediction in Appendix D.1, and carried out additional experiments on biomedical knowledge in Appendix D.2. In both experiments, we observed that C-MPNNs outperform R-MPNNs by a significant margin ( Q4 ).

6.3 Empirically evaluating the effect of readout

To understand the effect of readout ( Q5 ), we experiment with C-MPNNs using INIT 2 , PNA, MSG 1 r , and f ( t ) = t . We consider three architectures: (i) no readout component, (ii) global readout component (as defined earlier), and finally (iii) relation-specific readout. The idea behind the relation-specific readout is natural: instead of summing over all node representations, we sum over the features of nodes w which have an incoming q -edge ( w ∈ V + q ) or an outgoing q -edge ( w ∈ V -q ), where q is the query relation, as:

We report all results in Table 2. First, we observe that the addition of global readout does not lead to significant differences on WN18RR, but it degrades the model performance on FB15k-237 in two splits (v2 and v4). We attribute this to having more relation types in FB15k-237, so a readout over all nodes does not necessarily inform us about a relation-specific prediction. This is the motivation behind the relation-specific readout: we observe that it leads to significant improvements compared to using no readout (or a global readout) over all splits, leading to state-of-the art results. In contrast, relation-specific readout does not yield improvements on the relationally sparse WN18RR. These findings suggest that the global information can be very useful in practice, assuming the graph is rich in relation types. We further conduct a synthetic experiment to validate the expressive power of using global readout: architectures without global readout cannot exceed random performance, while those using a global readout can solve the synthetic link prediction task (see Appendix E).

7 Outlook and limitations

We studied a broad general class of GNNs designed for link prediction over knowledge graphs with a focus on formally characterizing their expressive power. Our study shows that recent state-of-the-art models are at a 'sweet spot': while they compute binary invariants, they do so through some local variations of higher-dimensional WL tests. This is precisely characterized within our framework. To capture global properties, we propose readout components and show that their relational variant is useful on datasets with many relation types. This study paints a more complete picture for our overall understanding of existing models and informs future work on possible directions.

In terms of limitations, it is important to note that the presented approach has certain restrictions. For instance, it is limited to binary tasks, such as link prediction. Thus, this approach does not directly extend to higher arity tasks, e.g., link prediction on relational hypergraphs or predicting the existence of substructures of size k in a knowledge graph. When considering local or conditional variations of the k -WL algorithm, computational demands can become a significant challenge. Further research is needed to explore potential optimizations for higher-order MPNNs that can enhance their performance while preserving their expressive power.

References

Ralph Abboud, ˙ Ismail ˙ Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. In IJCAI , 2021.

Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. In CoRR , 2016.

Pablo Barceló, Egor V. Kostylev, Mikaël Monet, Jorge Pérez, Juan L. Reutter, and Juan Pablo Silva. The logical expressiveness of graph neural networks. In ICLR , 2020.

Pablo Barceló, Mikhail Galkin, Christopher Morris, and Miguel Romero. Weisfeiler and leman go relational. In LoG , 2022.

Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In NIPS , 2016.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS , 2013.

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veliˇ ckovi´ c. Principal neighbourhood aggregation for graph nets. In NeurIPS , 2020.

Maarten de Rijke. A note on Graded Modal Logic. In Stud Logica , 2000.

Tim Dettmers, Minervini Pasquale, Stenetorp Pontus, and Sebastian Riedel. Convolutional 2D knowledge graph embeddings. In AAAI , 2018.

Luis Antonio Galárraga, Christina Teflioudi, Katja Hose, and Fabian Suchanek. AMIE: Association rule mining under incomplete evidence in ontological knowledge bases. In WWW , 2013.

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In ICML , 2017.

Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In IJCNN , 2005.

Martin Grohe. The logic of graph neural networks. In LICS , 2021.

Aditya Grover and Jure Leskovec. Node2vec: Scalable feature learning for networks. In KDD , 2016.

Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. In eLife , 2017.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS , 2020.

Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

Lecheng Kong, Yixin Chen, and Muhan Zhang. Geodesic graph neural network for efficient graph representation learning. In NeurIPS , 2022.

Carsten Lutz, Ulrike Sattler, and Frank Wolter. Modal logic and the two-variable fragment. In CSL , 2001.

Christian Meilicke, Manuel Fink, Yanjie Wang, Daniel Ruffinelli, Rainer Gemulla, and Heiner Stuckenschmidt. Fine-grained evaluation of rule- and embedding-based systems for knowledge graph completion. In IWSW , 2018.

Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and Leman go neural: Higher-order graph neural networks. In AAAI , 2019.

Christopher Morris, Gaurav Rattan, and Petra Mutzel. Weisfeiler and leman go sparse: Towards scalable higher-order graph embeddings. In NeurIPS , 2020.

Martin Otto. Graded modal logic and counting bisimulation. In arXiv , 2019.

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: online learning of social representations. In KDD , 2014.

Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. Drum: End-to-end differentiable rule mining on knowledge graphs. In NIPS , 2019.

Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In SDM , 2021.

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks , 2009.

Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In ESWC , 2018.

Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR , 2019.

Komal K. Teru, Etienne G. Denis, and William L. Hamilton. Inductive relation prediction by subgraph reasoning. In ICML , 2020.

Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In CVSC , 2015.

Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition-based multi-relational graph convolutional networks. In ICLR , 2020.

Hongwei Wang, Hongyu Ren, and Jure Leskovec. Relational message passing for knowledge graph completion. In KDD , 2021.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR , 2019.

Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. In NeurIPS , 2017.

Jiaxuan You, Jonathan M. Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In AAAI , 2021.

Donghan Yu, Yiming Yang, Ruohong Zhang, and Yuexin Wu. Generalized multi-relational graph convolution network. In ICML Workshop on Graph Representation Learning and Beyond , 2020.

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In NeurIPS , 2018.

Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. In NeurIPS , 2021.

Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford networks: A general graph neural network framework for link prediction. In NeurIPS , 2021.

A Proofs of technical statements

A.1 Expressive power of R-MPNNs

The expressive power of R-MPNNs has been recently characterized in terms of a relational variant of the Weisfeiler-Leman test [4]. We define the relational local 1 -WL test 2 , which we denote by rwl 1 . Let G = ( V, E, R, c ) be a knowledge graph. For each t ≥ 0 , the test updates the coloring as follows:

where τ injectively maps the above pair to a unique color, which has not been used in previous iterations. Hence, rwl ( t ) 1 defines a node invariant for all t ≥ 0 .

The following result generalizes results from Barceló et al. [4].

Theorem A.1. Let G = ( V, E, R, c ) be a knowledge graph.

For all initial feature maps x with c ≡ x , all R-MPNNs with T layers, and 0 ≤ t ≤ T , it holds that rwl ( t ) 1 ⪯ h ( t ) .

For all T ≥ 0 and history function f , there is an initial feature map x with c ≡ x and an RMPNN without global readout with T layers and history function f , such that for all 0 ≤ t ≤ T we have rwl ( t ) 1 ≡ h ( t ) .

Intuitively, item (1) states that the relational local 1 -WL algorithm upper bounds the power of any R-MPNN A : if the test cannot distinguish two nodes, then A cannot either. On the other hand, item (2) states that R-MPNNs can be as expressive as rwl 1 : for any time limit T , there is an R-MPNN that simulates T iterations of the test. This holds even without a global readout component.

Remark A.2 . A direct corollary of Theorem A.1 is that R-MPNNs have the same expressive power as rwl 1 , independently of their history function.

In order to prove Theorem A.1, we define a variant of rwl 1 as follows. For a history function f : N → N (recall f is non-decreasing and f ( t ) ≤ t ), and given a knowledge graph G = ( V, E, R, c ) , we define the rwl 1 ,f test via the following update rules:

where τ injectively maps the above pair to a unique color, which has not been used in previous iterations. Note that rwl 1 corresponds to rwl 1 ,id for the identity function id ( t ) = t .

We have that rwl ( t ) 1 ,f is always a refinement of rwl ( t -1) 1 ,f . Note that this is trivial for f ( t ) = t but not for arbitrary history functions.

Proposition A.3. Let G = ( V, E, R, c ) be a knowledge graph and f be a history function. Then for all t ≥ 0 , we have rwl ( t +1) 1 ,f ⪯ rwl ( t ) 1 ,f .

Proof. We proceed by induction on t . For t = 0 , note that rwl (1) 1 ,f ( u ) = rwl (1) 1 ,f ( v ) implies rwl ( f (0)) 1 ,f ( u ) = rwl ( f (0)) 1 ,f ( v ) and f (0) = 0 . For the inductive case, suppose rwl ( t +1) 1 ,f ( u ) = rwl ( t +1) 1 ,f ( v ) , for t ≥ 1 and u, v ∈ V . By injectivity of τ , we have that:

By inductive hypothesis and the facts that f ( t -1) ≤ f ( t ) (as f is non-decreasing) and rwl ( f ( t )) 1 ,f ( u ) = rwl ( f ( t )) 1 ,f ( v ) , we obtain rwl ( f ( t -1)) 1 ,f ( u ) = rwl ( f ( t -1)) 1 ,f ( v ) . On the other hand, by inductive hypothesis, we obtain that

We conclude that rwl ( t ) 1 ,f ( u ) = rwl ( t ) 1 ,f ( v )

As it turns out, rwl ( t ) 1 ,f defines the same coloring independently of f :

Proposition A.4. Let G = ( V, E, R, c ) be a knowledge graph and f, f ′ be history functions. Then for all t ≥ 0 , we have that rwl ( t ) 1 ,f ≡ rwl ( t ) 1 ,f ′ .

Proof. We apply induction on t . For t = 0 , we have rwl (0) 1 ,f ≡ c ≡ rwl (0) 1 ,f ′ . For the inductive case, suppose rwl ( t ) 1 ,f ( u ) = rwl ( t ) 1 ,f ( v ) , for t ≥ 1 and u, v ∈ V . Since f ′ ( t -1) ≤ t -1 and by Proposition A.3, we have that rwl ( f ′ ( t -1)) 1 ,f ( u ) = rwl ( f ′ ( t -1)) 1 ,f ( v ) . The inductive hypothesis implies that rwl ( f ′ ( t -1)) 1 ,f ′ ( u ) = rwl ( f ′ ( t -1)) 1 ,f ′ ( v ) . On the other hand, by injectivity of τ we have

The inductive hypothesis implies that

Summing up, we have that rwl ( t ) 1 ,f ′ ( u ) = rwl ( t ) 1 ,f ′ ( v ) , and hence rwl ( t ) 1 ,f ⪯ rwl ( t ) 1 ,f ′ . The case rwl ( t ) 1 ,f ′ ⪯ rwl ( t ) 1 ,f follows by symmetry.

Now we are ready to prove Theorem A.1.

We start with item (1). Take an initial feature map x with c ≡ x , an R-MPNN with T layers, and history function f . It suffices to show that rwl ( t ) 1 ,f ⪯ h ( t ) , for all 0 ≤ t ≤ T . Indeed, by Proposition A.4, we have rwl ( t ) 1 ,f ≡ rwl ( t ) 1 ,id ≡ rwl ( t ) 1 , where id is the identity function id ( t ) = t , and hence the result follows. We apply induction on t . The case t = 0 follows directly as rwl (0) 1 ,f ≡ c ≡ x ≡ h (0) . For the inductive case, assume rwl ( t ) 1 ,f ( u ) = rwl ( t ) 1 ,f ( v ) for t ≥ 1 and u, v ∈ V . By injectivity of τ we have:

By inductive hypothesis we have (recall f ( t -1) ≤ t -1 ):

This implies that { { MSG r ( h ( t -1) w ) | w ∈ N r ( u ) } } = { { MSG r ( h ( t -1) w ) | w ∈ N r ( v ) } } , for each r ∈ R , and hence:

We conclude that

For item (2), we refine the proof of Theorem 2 from Barceló et al. [4], which is based on ideas from Morris et al. [21]. In comparison with Barceló et al. [4], in our case, we have arbitrary adjacency matrices for each relation type, not only symmetric ones, and arbitrary history functions, not only the identity. However, the arguments still apply. Moreover, the most important difference is that here we aim for a model of R-MPNN without global readout that uses a single parameter matrix, instead of two parameter matrices as in Barceló et al. [4] (one for self-representations and the other for neighbors representations). This makes the simulation of rwl 1 more challenging.

We use models of R-MPNNs without global readout of the following form:

where W ( t ) is a parameter matrix and b is the bias term (we shall use the all-ones vector b = 1 ). As message function MSG r we use vector scaling , that is, MSG r ( h ) = α r h , where α r is a parameter of the model. For the non-linearity, we use the sign function sign . We note that the proof also works for the ReLU function, following arguments from Corollary 16 in Morris et al. [21].

For a matrix B , we denote by B i its i -th column. Let n = | V | and without loss of generality assume V = { 1 , . . . , n } . We will write features maps h : V → R d for G = ( V, E, R, c ) also as matrices H ∈ R d × n , where the column H v corresponds to the d -dimensional feature vector for v . Then we can also write our R-MPNN model in matrix form:

where A r is the adjacency matrix of G for relation type r ∈ R and J is the all-ones matrix of appropriate dimensions.

Let Fts be the following n × n matrix:

That is, ( Fts ) ij = -1 if j ≥ i , and ( Fts ) ij = 1 otherwise. Note that the columns of Fts are linearly independent. We shall use the columns of Fts as node features in our simulation.

The following lemma is an adaptation of Lemma 9 from Morris et al. [21].

Lemma A.5. Let B ∈ N n × p be a matrix such that p ≤ n , and all the columns are pairwise distinct and different from the all-zeros column. Then there is a matrix X ∈ R n × n such that the matrix sign( XB -J ) ∈ {-1 , 1 } n × p is precisely the sub-matrix of Fts given by its first p columns.

Proof. Let z = (1 , m, m 2 , . . . , m n -1 ) ∈ N 1 × n , where m is the largest entry in B , and b = zB ∈ N 1 × p . By construction, the entries of b are positive and pairwise distinct. Without loss of generality, we assume that b = ( b 1 , b 2 , . . . , b p ) for b 1 > b 2 > · · · > b p > 0 . As the b i are ordered, we can choose numbers x 1 , . . . , x p ∈ R such that b i · x j < 1 if i ≥ j , and b i · x j > 1 if i < j , for all i, j ∈ { 1 , . . . , p } . Let x = ( x 1 , . . . , x p , 2 /b p , . . . , 2 /b p ) T ∈ R n × 1 . Note that (2 /b p ) · b i > 1 , for all i ∈ { 1 , . . . , p } . Then sign( xb -J ) is precisely the sub-matrix of Fts given by its first p columns. We can choose X = xz ∈ R n × n .

Now we are ready to show item (2). Let f be any history function and T ≥ 0 . It suffices to show that there is a feature map x with c ≡ x and an R-MPNN without global readout with T layers and history function f such that rwl ( t ) 1 ,f ≡ h ( t ) , for all 0 ≤ t ≤ T . Indeed, by Proposition A.4, we have rwl ( t ) 1 ,f ≡ rwl ( t ) 1 ,id ≡ rwl ( t ) 1 , where id is the identity function id ( t ) = t , and then the result follows. We conclude item (2) by showing the following lemma:

Lemma A.6. There is a feature map h (0) : V → R n , and for all 0 ≤ t < T , there is a feature map h ( t +1) : V → R n , a matrix W ( t ) ∈ R n × n and scaling factors α ( t ) r ∈ R , for each r ∈ R , such that:

h ( t ) ≡ rwl ( t ) 1 ,f .

The columns of H ( t ) ∈ R n × n are columns of Fts (recall H ( t ) is the matrix representation of h ( t ) ).

Proof. We proceed by induction on t . Suppose that the node coloring rwl (0) 1 ,f ≡ c uses colors 1 , . . . , p , for p ≤ n . Then we choose h (0) (this is the initial feature map x in the statement of item (2)) such that h (0) v = Fts c ( v ) , that is, h (0) v is the c ( v ) -th column of Fts . We have that h (0) satisfies the required conditions.

For the inductive case, assume that h ( t ′ ) ≡ rwl ( t ′ ) 1 ,f and the columns of H ( t ′ ) are columns of Fts , for all 0 ≤ t ′ ≤ t < T . We need to find h ( t +1) , W ( t ) and α ( t ) r satisfying the conditions. Let M ∈ R n × n be the matrix inverse of Fts . If H ( t ′ ) v is the i -th column of Fts , we say that v has color i at iteration t ′ . Observe that for all 0 ≤ t ′ ≤ t , we have

In other words, the v -th column of MH ( t ′ ) is simply a one-hot encoding of the color of v at iteration t ′ . For each r ∈ R we have

Hence the v -th column of MH ( t ) A r is an encoding of the multiset of colors for the neighborhood N r ( v ) , at iteration t . Let r 1 , . . . , r m be an enumeration of the relation types in R . Let D ∈ R ( m +1) n × n be the matrix obtained by horizontally concatenating the matrices MH ( f ( t )) , MH ( t ) A r 1 , . . . , MH ( t ) A r m . Since H ( f ( t )) ≡ rwl ( f ( t )) 1 ,f and H ( t ) ≡ rwl ( t ) 1 ,f , we have that D ≡ rwl ( t +1) 1 ,f . Now note that D ≡ E , where

Indeed, E iv is simply the ( n + 1) -base representation of the vector ( D iv , D ( n + i ) v , D (2 n + i ) v . . . , D ( mn + i ) v ) , and hence E u = E v if and only if D u = D v (note that the entries of D are in { 0 , . . . , n } ). In particular, E ≡ rwl ( t +1) 1 ,f .

Let p be the number of distinct columns of E and let ˜ E ∈ N n × p be the matrix whose columns are the distinct columns of E in an arbitrary but fixed order. We can apply Lemma A.5 to ˜ E and obtain a matrix X ∈ R n × n such that sign( X ˜ E -J ) is precisely the sub-matrix of Fts given by its first p columns. We choose H ( t +1) = sign( XE -J ) ∈ R n × n , W ( t ) = XM ∈ R n × n and α ( t ) r i = ( n +1) i . Note that the columns of H ( t +1) are columns of Fts , and that H ( t +1) ≡ E ≡ rwl ( t +1) 1 ,f . Finally, we have

Note that our result applies to more complex message functions such as MSG r ( h ) = h ∗ b r , where ∗ stands for element-wise multiplication and b r is a vector parameter, and MSG r ( h ) = W r h , where W r is matrix parameter, as they can easily express vector scaling. The first case has been used for the model CompGCN [32], while the second case has been used for R-GCN [28].

A.2 Proof of Theorem 5.1

We recall the statement of the theorem:

Theorem 5.1. Let G = ( V, E, R, x , η ) be a knowledge graph, where x is a feature map, and η is a pairwise coloring satisfying target node distinguishability. Let q ∈ R be any query relation. Then:

For all C-MPNNs with T layers and initialization INIT satisfying INIT ≡ η , and all 0 ≤ t ≤ T , we have rawl ( t ) 2 ⪯ h ( t ) q .

For all T ≥ 0 and history function f , there is an C-MPNN without global readout with T layers and history function f such that for all 0 ≤ t ≤ T , we have rawl ( t ) 2 ≡ h ( t ) q .

We start by showing item (1) and (2) of the theorem for the case without global readout. As we explain later, extending item (1) to include readouts is straightforward. In order to prove this particular case, we apply a reduction to R-MPNNs and the relational local 1 -WL. Before doing so we need some auxiliary results.

̸

Let G = ( V, E, R, c, η ) be a knowledge graph where η is a pairwise coloring. We denote by G 2 the knowledge graph G 2 = ( V × V, E ′ , R, c η ) where E ′ = { r (( u, w ) , ( u, v )) | r ( w,v ) ∈ E,r ∈ R } and c η is the node coloring c η (( u, v )) = η ( u, v ) . Note that the coloring c is irrelevant in the construction. Intuitively, G 2 encodes the adjacency relation between pairs of nodes of G used in C-MPNNs. We stress that G and G 2 have the same relation type set R . If A is a C-MPNN and B is an R-MPNN, we write h ( t ) A ,G ( u, v ) := h ( t ) q ( u, v ) and h ( t ) B ,G 2 (( u, v )) := h ( t ) (( u, v )) for the features computed by A and B over G and G 2 , respectively. We sometimes write N r H ( v ) to emphasize that the neighborhood is taken over the knowledge graph H . Finally, we say that an initial feature map y for G 2 satisfies target node distinguishability if y (( u, u )) = y (( u, v )) for all u = v .

̸

We have the following equivalence between C-MPNNs and R-MPNNs without global readout:

Proposition A.7. Let G = ( V, E, R, x , η ) 3 be a knowledge graph where x is a feature map, and η is a pairwise coloring. Let q ∈ R be any query relation. Then:

For every C-MPNN without global readout A with T layers, there is an initial feature map y for G 2 an R-MPNN without global readout B with T layers such that for all 0 ≤ t ≤ T and u, v ∈ V , we have h ( t ) A ,G ( u, v ) = h ( t ) B ,G 2 (( u, v )) .

For every initial feature map y for G 2 satisfying target node distinguishability and every RMPNN without global readout B with T layers, there is a C-MPNN without global readout A with T layers such that for all 0 ≤ t ≤ T and u, v ∈ V , we have h ( t ) A ,G ( u, v ) = h ( t ) B ,G 2 (( u, v )) .

Proof. We start with item (1). The sought R-MPNN B has the same history function and the same message, aggregation, and update functions as A for all the T layers. The initial feature map y is defined as y (( u, v )) = INIT ( u, v, q ) , where INIT is the initialization function of A .

We show the equivalence by induction on t . For t = 0 , we have h (0) A ( u, v ) = INIT ( u, v, q ) = y (( u, v )) = h (0) B (( u, v )) . For the inductive case, take u, v ∈ V . We have h ( t +1) A ( u, v ) = UPD ( h ( f ( t )) A ( u, v ) , AGG ( { { MSG r ( h ( t ) A ( u, w ) , z q ) | w ∈ N G r ( v ) , r ∈ R } } ) ) = UPD ( h ( f ( t )) B (( u, v )) , AGG ( { { MSG r ( h ( t ) B (( u, w )) , z q ) | ( u, w ) ∈ N G 2 r (( u, v )) , r ∈ R } } ) ) = h ( t +1) B (( u, v )) .

For item (2), we take A to have the same history function and the same message, aggregation, and update functions than B , for all the T layers, and initialization function INIT such that INIT ( u, v, q ) = y (( u, v )) . The argument for the equivalence is the same as item (1).

Regarding WL algorithms, we have a similar equivalence:

Proposition A.8. Let G = ( V, E, R, c, η ) be a knowledge graph where η is a pairwise coloring. For all t ≥ 0 and u, v ∈ V , we have that rawl ( t ) 2 ( u, v ) computed over G coincides with rwl ( t ) 1 (( u, v )) computed over G 2 = ( V × V, E ′ , R, c η ) .

Proof. For t = 0 , we have rawl (0) 2 ( u, v ) = η ( u, v ) = c η (( u, v )) = rwl (0) 1 (( u, v )) . For the inductive case, we have

Now we are ready to prove Theorem 5.1.

For G = ( V, E, R, x , η ) , we consider G 2 = ( V × V, E ′ , R, c η ) . We start with item (1). Let A be a C-MPNN without global readout with T layers and initialization INIT satisfying INIT ≡ η and let 0 ≤ t ≤ T . Let y be an initial feature map for G 2 and B be an R-MPNN without global readout with T layers as in Proposition A.7, item (1). Note that y ≡ c η as y (( u, v )) = INIT ( u, v, q ) . We can apply Theorem A.1, item (1) to G 2 , y and B and conclude that rwl ( t ) 1 ⪯ h ( t ) B ,G 2 . This implies that rawl ( t ) 1 ⪯ h ( t ) A ,G .

For item (2), let T ≥ 0 and f be a history function. We can apply Theorem A.1, item (2), to G 2 to obtain an initial feature map y with y ≡ c η and an R-MPNN without global readout B with T layers and history function f such that for all 0 ≤ t ≤ T , we have rwl ( t ) 1 ≡ h ( t ) B ,G 2 . Note that y satisfies target node distinguishability since η does. Let A be the C-MPNN without global readout obtained from Proposition A.7, item (2). We have that rawl ( t ) 2 ≡ h ( t ) A ,G as required.

Remark A.9 . Note that item (2) holds for the basic model of C-MPNNs (without global readout), presented in Section 4.1, with the three proposed message functions MSG 1 r , MSG 2 r , MSG 3 r , as they can express vector scaling.

Finally, note that the extension of item (1) in Theorem 5.1 to the case with global readouts is straightforward. Indeed, we can apply exactly the same argument we used for the case without global readout. As for two pairs of the form ( u, v ) and ( u, v ′ ) the global readout vectors are identical (to READ ( { { h ( t ) w | u,q | w ∈ V } } ) ), this argument still works.

A.3 Proof of Theorem 5.2

We recall the statement of the theorem:

Theorem 5.2. A logical binary classifier is captured by C-MPNNs without global readout if and only if it can be expressed in rFO 3 cnt .

The proof is a reduction to the one-dimensional case, that is, R-MPNNs without global readout and a logic denoted by rFO 2 cnt . So in order to show Theorem 5.2 we need some definitions and auxiliary results.

Fix a set of relation types R and a set of node colors C . We consider knowledge graphs of the form G = ( V, E, R, c ) where c is a node coloring assigning colors from C . In this context, logical formulas can refer to relation types from R and node colors from C . Following Barceló et al. [3], a logical (node) classifier is a unary formula expressible in first-order logic (FO), classifying each node u on a knowledge graph G according to whether the formula holds or not for u over G .

We define a fragment rFO 2 cnt of FO as follows. A rFO 2 cnt formula is either a ( x ) for a ∈ C , or one of the following, where φ and ψ are rFO 2 cnt formulas, N ≥ 1 is a positive integer and r ∈ R :

We remark that rFO 2 cnt is actually the fragment of FO used in Barceló et al. [3] to characterize GNNs, adapted to multiple relations. It is well-known that rFO 2 cnt is equivalent to graded modal logic [8].

The following proposition provides useful translations from rFO 2 cnt to rFO 3 cnt and vice versa. Recall from Section A.2, that given a knowledge graph G = ( V, E, R, η ) where η is a pairwise coloring, we define the knowledge graph G 2 = ( V × V, E ′ , R, c η ) where E ′ = { r (( u, w ) , ( u, v )) | r ( w,v ) ∈ E,r ∈ R } and c η is the node coloring c η (( u, v )) = η ( u, v ) .

Proposition A.10. We have the following:

For all rFO 3 cnt formula φ ( x, y ) , there is a formula ˜ φ ( x ) in rFO 2 cnt such that for all knowledge graph G = ( V, E, R, η ) , we have G,u,v | = φ if and only if G 2 , ( u, v ) | = ˜ φ .

For all formula φ ( x ) in rFO 2 cnt , there is a rFO 3 cnt formula ˜ φ ( x, y ) such that for all knowledge graph G = ( V, E, R, η ) , we have G,u,v | = ˜ φ if and only if G 2 , ( u, v ) | = φ .

Proof. We start with item (1). We define ˜ φ ( x ) by induction on the formula φ ( x, y ) :

If φ ( x, y ) = a ( x, y ) for color a , then ˜ φ ( x ) = a ( x ) .

If φ ( x, y ) = ¬ ψ ( x, y ) , then ˜ φ ( x ) = ¬ ˜ ψ ( x ) .

If φ ( x, y ) = φ 1 ( x, y ) ∧ φ 2 ( x, y ) , then ˜ φ ( x ) = ˜ φ 1 ( x ) ∧ ˜ φ 2 ( x ) .

If φ ( x, y ) = ∃ ≥ N z ( ψ ( x, z ) ∧ r ( z, y )) then ˜ φ ( x ) = ∃ ≥ N y ( ˜ ψ ( y ) ∧ r ( y, x )) .

Fix G = ( V, E, R, η ) and G 2 = ( V × V, E ′ , R, c η ) . We show by induction on the formula φ that G,u,v | = φ if and only if G 2 , ( u, v ) | = ˜ φ .

For the base case, that is, case (1) above, we have that φ ( x, y ) = a ( x, y ) and hence G,u,v | = φ iff η ( u, v ) = a iff c η (( u, v )) = a iff G 2 , ( u, v ) | = ˜ φ .

Now we consider the inductive case. For case (2) above, we have φ ( x, y ) = ¬ ψ ( x, y ) . Then G,u,v | = φ iff G,u,v ̸| = ψ iff G 2 , ( u, v ) ̸| = ˜ ψ iff G 2 , ( u, v ) | = ˜ φ .

For case (3), we have φ ( x, y ) = φ 1 ( x, y ) ∧ φ 2 ( x, y ) . Then G,u,v | = φ iff G,u,v | = φ 1 and G,u,v | = φ 2 iff G 2 , ( u, v ) | = ˜ φ 1 and G 2 , ( u, v ) | = ˜ φ 2 iff G 2 , ( u, v ) | = ˜ φ .

Finally, for case (4), we have φ ( x, y ) = ∃ ≥ N z ( ψ ( x, z ) ∧ r ( z, y )) . Assume G,u,v | = φ , then there exist at least N nodes w ∈ V such that G,u,w | = ψ and r ( w,v ) ∈ E . By the definition of G 2 , there exist at least N nodes in G 2 of the form ( u, w ) such that G 2 , ( u, w ) | = ˜ ψ and r (( u, w ) , ( u, v )) ∈ E ′ . It follows that G 2 , ( u, v ) | = ˜ φ . On the other hand, suppose G 2 , ( u, v ) | = ˜ φ . Then there exist at least N nodes ( o, o ′ ) in G 2 such that G, ( o, o ′ ) | = ˜ ψ and r (( o, o ′ ) , ( u, v )) ∈ E ′ . By definition of G 2 each ( o, o ′ ) must be of the form ( o, o ′ ) = ( u, w ) for some w ∈ V such that r ( w,v ) . Then there are at least N nodes w ∈ V such that G,u,w | = ψ and r ( w,v ) ∈ E . It follows that G,u,v | = φ .

Item (2) is similar. We define ˜ φ ( x, y ) by induction on the formula φ ( x ) :

If φ ( x ) = a ( x ) for color a , then ˜ φ ( x, y ) = a ( x, y ) .

If φ ( x ) = ¬ ψ ( x ) , then ˜ φ ( x, y ) = ¬ ˜ ψ ( x, y ) .

If φ ( x ) = φ 1 ( x ) ∧ φ 2 ( x ) , then ˜ φ ( x, y ) = ˜ φ 1 ( x, y ) ∧ ˜ φ 2 ( x, y ) .

If φ ( x ) = ∃ ≥ N y ( ψ ( y ) ∧ r ( y, x )) then ˜ φ ( x, y ) = ∃ ≥ N z ( ˜ ψ ( x, z ) ∧ r ( z, y )) .

Following the same inductive argument from item (1), we obtain that G,u,v | = ˜ φ if and only if G 2 , ( u, v ) | = φ .

The following theorem is an adaptation of Theorem 4.2 from Barceló et al. [3]. The main difference with Barceló et al. [3] is that here we need to handle multiple relation types.

Theorem A.11. A logical classifier is captured by R-MPNNs without global readout if and only if it can be expressed in rFO 2 cnt .

Proof. We start with the backward direction. Let φ ( x ) be a formula in rFO 2 cnt for relation types R and node colors C . Let φ 1 , . . . , φ L be an enumeration of the subformulas of φ such that if φ i is a subformula of φ j , then i ≤ j . In particular, φ L = φ . We shall define an R-MPNN without global readout B φ with L layers computing L -dimensional features in each layer. The idea is that at layer ℓ ∈ { 1 , . . . , L } , the ℓ -th component of the feature h ( ℓ ) v is computed correctly and corresponds to 1 if φ ℓ is satisfied in node v , and 0 otherwise. We add an additional final layer that simply outputs the last component of the feature vector.

We use models of R-MPNNs of the following form:

where W ∈ R L × L is a parameter matrix and b ∈ R L is the bias term. As message function MSG r we use MSG r ( h ) = W r h , where W r ∈ R L × L is a parameter matrix . For the non-linearity σ we use the truncated ReLU function σ ( x ) = min(max(0 , x ) , 1) . The ℓ -th row of W and W r , and the ℓ -th entry of b are defined as follows (omitted entries are 0 ):

If φ ℓ ( x ) = a ( x ) for a color a ∈ C , then W ℓℓ = 1 .

If φ ℓ ( x ) = ¬ φ k ( x ) then W ℓk = -1 , and b ℓ = 1 .

If φ ℓ ( x ) = φ j ( x ) ∧ φ k ( x ) then W ℓj = 1 , W ℓk = 1 and b ℓ = -1 .

If φ ℓ ( x ) = ∃ ≥ N y ( φ k ( y ) ∧ r ( y, x )) then ( W r ) ℓk = 1 and b ℓ = -N +1 .

Let G = ( V, E, R, c ) be a knowledge graph with node colors from C . In order to apply B φ to G , we choose initial L -dimensional features h (0) v such that ( h (0) v ) ℓ = 1 if φ ℓ = a ( x ) and a is the color of v , and ( h (0) v ) ℓ = 0 otherwise. In other words, the L -dimensional initial feature h (0) v is a one-hot encoding of the color of v . It follows from the same arguments than Proposition 4.1 in Barceló et al. [3] that for all ℓ ∈ { 1 , . . . , L } we have ( h ( t ) v ) ℓ = 1 if G,v | = φ ℓ and ( h ( t ) v ) ℓ = 0 otherwise, for all v ∈ V and t ∈ { ℓ, . . . , L } . In particular, after L layers, B φ calculates h ( L ) v such that ( h ( L ) v ) L = 1 if G,v | = φ and ( h ( L ) v ) L = 0 otherwise. As layer L +1 extracts the L -th component of the feature vector, the result follows.

For the forward direction, we follow the strategy of Theorem 4.2 from Barceló et al. [3]. Given a knowledge graph G = ( V, E, R, c ) and a number L ∈ N , we define the unravelling of v ∈ V at depth L , denoted Unr L G ( v ) is the knowledge graph having:

A node ( v, u 1 , . . . , u i ) for each directed path u i , . . . , u 1 , v in G of length i ≤ L .

For each r ∈ R , a fact r (( v, u 1 , . . . , u i ) , ( v, u 1 , . . . , u i -1 )) for all facts r ( u i , u i -1 ) ∈ E (here u 0 := v ).

Each node ( v, u 1 , . . . , u i ) is colored with c ( u i ) , that is, the same color as u i .

Note that the notion of directed path is defined in the obvious way, as for directed graphs but ignoring relation types. Note also that Unr L G ( v ) is a tree in the sense that the underlying undirected graph is a tree.

The following proposition is a trivial adaptation of Observation C.3 from Barceló et al. [3]. We write Unr L G ( v ) ≃ Unr L G ′ ( v ′ ) if there exists an isomorphism f from Unr L G ( v ) to Unr L G ′ ( v ′ ) such that f ( v ) = v ′ .

Proposition A.12. Let G and G ′ be two knowledge graphs and v and v ′ be nodes in G and G ′ , respectively. Then, for all L ∈ N , we have that rwl ( L ) 1 ( v ) on G coincides with rwl ( L ) 1 ( v ′ ) on G ′ if and only if Unr L G ( v ) ≃ Unr L G ′ ( v ′ ) .

As a consequence of Theorem A.1, we obtain:

Proposition A.13. Let G and G ′ be two knowledge graphs and v and v ′ be nodes in G and G ′ , respectively, such that Unr L G ( v ) ≃ Unr L G ′ ( v ′ ) for all L ∈ N . Then for any R-MPNN without global readout with T layers, we have that h ( T ) v on G coincides with h ( T ) v ′ on G ′ .

Finally, the following theorem follows from Theorem C.5 in Barceló et al. [3], which in turn follows from Theorem 2.2 in Otto [23]. The key observation here is that the results from Otto [23] are actually presented for multi-modal logics, that is, multiple relation types.

Theorem A.14. [23] Let α be a unary FO formula over knowledge graphs. If α is not equivalent to a rFO 2 cnt formula, then there exist two knowledge graphs G and G ′ , and two nodes v in G and v ′ in G ′ such that Unr L G ( v ) ≃ Unr L G ′ ( v ′ ) for all L ∈ N and such that G,v | = α but G ′ , v ′ ̸| = α .

Now we are ready to obtain the forward direction of the theorem. Suppose that a logical classifier α is captured by an R-MPNN without global readout B with T layers, and assume by contradiction that α is not equivalent to a rFO 2 cnt formula. Then we can apply Theorem A.14 and obtain two knowledge graphs G and G ′ , and two nodes v in G and v ′ in G ′ such that Unr L G ( v ) ≃ Unr L G ′ ( v ′ ) for all L ∈ N and such that G,v | = α but G ′ , v ′ ̸| = α . Applying Proposition A.13, we have that h ( T ) v on G coincides with h ( T ) v ′ on G ′ , and hence B classifies either both v and v ′ as true over G and G ′ , respectively, or both as false . This is a contradiction.

Now Theorem 5.2 follows easily. Let α be a logical binary classifier and suppose it is captured by a C-MPNN without global readout A . By Proposition A.7,item (1), we know that A can be simulated by a R-MPNN without global readout B over G 2 . In turn, we can apply Theorem A.11 and obtain a formula φ in rFO 2 cnt equivalent to B . Finally, we can apply the translation in Proposition A.10, item (2), and obtain a corresponding formula ˜ φ in rFO 3 cnt . We claim that ˜ φ captures A . Let G be a knowledge graph and u, v two nodes. We have that G,u,v | = ˜ φ iff G 2 , ( u, v ) | = φ iff B classifies ( u, v ) as true over G 2 iff A classifies ( u, v ) as true over G .

The other direction is obtained analogously following the reverse translations.

A.4 Proof of Theorem 5.3

We recall the statement of the theorem:

Theorem 5.3. Each logical binary classifier expressible in erFO 3 cnt can be captured by a C-MPNN.

We start by formally defining the logic erFO 3 cnt , which is a simple extension of rFO 3 cnt from Section 5.2. Fix a set of relation types R and a set of pair colors C . Recall we consider knowledge graphs of the form G = ( V, E, R, η ) where η is a mapping assigning colors from C to pairs of nodes from V . The logic erFO 3 cnt contains only binary formulas and it is defined inductively as follows: First, a ( x, y ) for a ∈ C , is in erFO 3 cnt . Second, if φ ( x, y ) and ψ ( x, y ) are in erFO 3 cnt , N ≥ 1 is a positive integer, and r ∈ R , then the formulas

are also in erFO 3 cnt . As expected, a ( u, v ) holds in G = ( V, E, R, η ) if η ( u, v ) = a , and ∃ ≥ N z ( φ ( u, z ) ∧ ℓ ( z, v )) holds in G , for ℓ ∈ R ∪ {¬ r | r ∈ R } , if there are at least N nodes w ∈ V for which φ ( u, w ) and ℓ ( w,v ) hold in G .

Intuitively, erFO 3 cnt extends rFO 3 cnt with negated modalities , that is, one can check for non-neighbors of a node. As it turns out, erFO 3 cnt is strictly more expressive than rFO 3 cnt .

Proposition A.15. There is a formula in erFO 3 cnt that cannot be expressed by any formula in rFO 3 cnt .

Proof. Consider the following two knowledge graphs G 1 and G 2 over R = { r } and colors C = { Green } . The graph G 1 has two nodes u and v and one edge r ( u, v ) . The graph G 2 has three nodes u , v and w and one edge r ( u, v ) (hence w is an isolated node). In both graphs, all pairs are colored Green.

We show first that for every formula φ ( x, y ) in rFO 3 cnt , it is the case that G 1 , u, n | = φ iff G 2 , u, n | = φ , for n ∈ { u, v } . We proceed by induction on the formula. For φ ( x, y ) = Green ( x, y ) , we have

that G 1 , u, n | = φ and G 2 , u, n | = φ and hence we are done. Suppose now that φ ( x, y ) = ¬ ψ ( x, y ) . Assume G 1 , u, n | = φ for n ∈ { u, v } . Then G 1 , u, n ̸| = ψ , and by inductive hypothesis, we have G 2 , u, n ̸| = ψ and then G 2 , u, n | = φ . The other direction is analogous. Assume now that φ ( x, y ) = ψ 1 ( x, y ) ∧ ψ 2 ( x, y ) . Suppose G 1 , u, n | = φ for n ∈ { u, v } . Then G 1 , u, n | = ψ 1 and G 1 , u, n | = ψ 2 , and by inductive hypothesis, G 2 , u, n | = ψ 1 and G 2 , u, n | = ψ 2 . Then G 2 , u, n | = φ . The other direction is analogous. Finally, suppose that φ ( x, y ) = ∃ ≥ N z ( ψ ( x, z ) ∧ r ( z, y )) . Assume that G 1 , u, n | = φ for n ∈ { u, v } . Then there exist at least N nodes w ∈ N r ( n ) in G 1 such that G 1 , u, w | = ψ . Since N r ( u ) = ∅ , we have n = v . As N r ( v ) = { u } , we have w = u . Since the neighborhood N r ( v ) is the same in G 1 and G 2 , and by the inductive hypothesis, we have that there are at least N nodes w ∈ N r ( v ) in G 2 such that G 2 , u, w | = ψ . This implies that G 2 , u, v | = φ . The other direction is analogous.

Now consider the erFO 3 cnt formula φ ( x, y ) = ∃ ≥ 2 z ( Green ( x, z ) ∧ ¬ r ( z, y )) . We claim that there is no rFO 3 cnt formula equivalent to φ ( x, y ) . By contradiction, suppose we have such an equivalent formula φ ′ . As shown above, we have G 1 , u, v | = φ ′ iff G 2 , u, v | = φ ′ . On the other hand, by definition, we have that G 1 , u, v ̸| = φ and G 2 , u, v | = φ . This is a contradiction.

Now we are ready to prove Theorem 5.3. We follow the same strategy as in the proof of the backward direction of Theorem A.11.

Let φ ( x, y ) be a formula in erFO 3 cnt , for relation types R and pair colors C . Let φ 1 , . . . , φ L be an enumeration of the subformulas of φ such that if φ i is a subformula of φ j , then i ≤ j . In particular, φ L = φ . We will construct a C-MPNN A φ with L layers computing L -dimensional features in each layer. At layer ℓ ∈ { 1 , . . . , L } , the ℓ -th component of the feature h ( ℓ ) v | u,q will correspond to 1 if φ ℓ is satisfied on ( u, v ) , and 0 otherwise. The query relation q plays no role in the construction, that is, for any possible q ∈ R the output of A φ is the same. Hence, for simplicity, in the remaining of the proof we shall write h ( t ) v | u instead of h ( t ) v | u,q . We add an additional final layer that simply outputs the last component of the feature vector.

We use models of C-MPNNs of the following form:

where W 0 , W 1 ∈ R L × L are parameter matrices and b ∈ R L is the bias term. As message function MSG r we use MSG r ( h ) = W r h , where W r ∈ R L × L is a parameter matrix . For the non-linearity σ we use the truncated ReLU function σ ( x ) = min(max(0 , x ) , 1) . The ℓ -th row of W 0 , W 1 and W r , and the ℓ -th entry of b are defined as follows (omitted entries are 0 ):

If φ ℓ ( x, y ) = a ( x, y ) for a color a ∈ C , then ( W 0 ) ℓℓ = 1 .

If φ ℓ ( x, y ) = ¬ φ k ( x, y ) then ( W 0 ) ℓk = -1 , and b ℓ = 1 .

If φ ℓ ( x, y ) = φ j ( x, y ) ∧ φ k ( x, y ) then ( W 0 ) ℓj = 1 , ( W 0 ) ℓk = 1 and b ℓ = -1 .

If φ ℓ ( x, y ) = ∃ ≥ N z ( φ k ( x, z ) ∧ r ( z, y )) then ( W r ) ℓk = 1 and b ℓ = -N +1 .

If φ ℓ ( x, y ) = ∃ ≥ N z ( φ k ( x, z ) ∧¬ r ( z, y )) then ( W r ) ℓk = -1 , ( W 1 ) ℓk = 1 and b ℓ = -N +1 .

Let G = ( V, E, R, η ) be a knowledge graph with pair colors from C . In order to apply A φ to G , we choose the initialization INIT such that L -dimensional initial features h (0) v | u satisfy ( h (0) v | u ) ℓ = 1 if φ ℓ = a ( x, y ) and η ( u, v ) = a and ( h (0) v | u ) ℓ = 0 otherwise. That is, the L -dimensional initial feature h (0) v | u is a one-hot encoding of the color of ( u, v ) . Using the same arguments as in the proof of Theorem A.11, we have ( h ( t ) v | u ) ℓ = 1 if G,u,v | = φ ℓ and ( h ( t ) v | u ) ℓ = 0 otherwise, for all u, v ∈ V and t ∈ { ℓ, . . . , L } . In particular, after L layers, A φ calculates h ( L ) v | u such that ( h ( L ) v | u ) L = 1 if G,u,v | = φ and ( h ( L ) v | u ) L = 0 otherwise. As layer L +1 extracts the L -th component of the feature vector, the result follows.

Remark A.16 . We note that in Barceló et al. [3], it is shown that a more expressive graded modal logic (with more expressive modalities) can be captured by R-MPNNs in the context of single-relation

graphs. It is by no means obvious that an adaption of this logic to the multi-relational case is captured by our C-MPNNs. We leave as an open problem to find more expressive logics that can be captured by C-MPNNs.

A.5 Proofs of the results from Section 5.3

First, recall the definition of rawl 2 . Given a knowledge graph G = ( V, E, R, c, η ) , we have

Note that rawl 2 , and hence C-MPNNs, are one-directional : the neighborhood N r ( v ) only considers facts in one direction, in this case, from neighbors to v . Hence, a natural extension is to consider bi-directional neighborhoods. Fortunately, we can define this extension by simply applying the same test rawl 1 to knowledge graphs extended with inverse relations. We formalize this below.

For a test T , we sometimes write T ( G,u,v ) , or T ( G,v ) in case of unary tests, to emphasize that the test is applied over G , and T ( G ) for the node/pairwise coloring given by the test. Let G = ( V, E, R, c, η ) be a knowledge graph. We define its augmented knowledge graph to be G + = ( V, E + , R + , c, η ) , where R + is the disjoint union of R and { r -| r ∈ R } , and

̸

We define the augmented relational asymmetric local 2-WL test on G , denoted by rawl + 2 , as

for all t ≥ 0 and u, v ∈ V . As we show below, rawl + 2 is strictly more powerful than rawl 2 .

Proposition A.17. The following statements hold:

For all t ≥ 0 and all knowledge graphs G , we have rawl + 2 ( t ) ( G ) ⪯ rawl 2 ( t ) ( G ) .

̸

There is a knowledge graph G and nodes u, v, u ′ , v ′ such that rawl + 2 (1) ( G,u,v ) = rawl + 2 (1) ( G,u ′ , v ′ ) but rawl 2 ( t ) ( G,u,v ) = rawl 2 ( t ) ( G,u ′ , v ′ ) for all t ≥ 1 .

Proof. To prove rawl + 2 ( G ) ⪯ rawl 2 ( G ) first, we consider induction on iteration t . The base case for t = 0 is trivial by the assumption, so it is enough to consider the inductive step. We need to show that for some k ,

By the definition of rawl + 2 ( k +1) ( u, v ) , rawl + 2 ( k +1) ( u ′ , v ′ ) and the injectivity of τ , it holds that

Because all inverse relations r -are newly introduced, so it is impossible to be mixed with r , we can split the second equation into the following equations:

By the inductive hypothesis and unpacking the first equation, we can further imply that

Thus, By definition of rawl ( k +1) 2 ( u, v ) , rawl ( k +1) 2 ( u ′ , v ′ ) it holds that

For the counterexample, we consider a relational graph with two types of relation G ′ = ( V ′ , E ′ , R ′ , c, η ) such that V ′ = { u, v, v ′ } , E ′ = { r 1 ( v, u ) , r 2 ( v ′ , u ) } , and R ′ = { r 1 , r 2 } . Let the initial pairwise labeling η for node pairs ( u, v ) and ( u ′ , v ) satisfy rawl + 2 (0) ( u, v ) = rawl + 2 (0) ( u, v ′ ) and rawl (0) 2 ( u, v ) = rawl (0) 2 ( u, v ′ ) . For such graph G ′ , we consider node pair ( u, v ) and ( u, v ′ ) . For rawl ( t ) 2 where t ≥ 0 , we show by induction that rawl ( t ) 2 ( u, v ) = rawl ( t ) 2 ( u, v ′ ) . The base case is trivial by assumption. The inductive step shows that by the inductive hypothesis,

On the other hand, we have

̸

We can also extend C-MPNNs with bi-directionality in an obvious way, obtaining augmented CMPNNs. By applying Theorem 5.1 to the augmented graph G + , we obtain the equivalence between augmented C-MPNNs and the test rawl + 2 directly. In turn, Proposition A.17 implies that augmented C-MPNNs are strictly more powerful that C-MPNNs in distinguishing nodes in a graph.

Recall the definition of rwl 2 . Given a knowledge graph G = ( V, E, R, c, η ) , we have

We define the augmented version rwl + 2 in an obvious way, that is,

for all t ≥ 0 and u, v ∈ V .

We will drop the notation of G during the proof when the context is clear for simplicity.

Proposition A.18. For all t ≥ 0 and all knowledge graph G , let rwl (0) 2 ( G ) ≡ rawl (0) 2 ( G ) . The following statements hold:

For every t > 0 , it holds that rwl ( t ) 2 ( G ) ⪯ rawl ( t ) 2 ( G ) .

̸

There is a knowledge graph G and pair of nodes ( u, v ) and ( u ′ , v ′ ) such that rawl ( t ) 2 ( G,u,v ) = rawl ( t ) 2 ( G,u ′ , v ′ ) for all t ≥ 0 but rwl (1) 2 ( G,u,v ) = rwl (1) 2 ( G,u ′ , v ′ ) .

Proof. First, we show rwl ( t ) 2 ( G ) ⪯ rawl ( t ) 2 ( G ) and proceed by induction on t . The base case for t = 0 is trivial by assumption. For the inductive step, given that we have rwl ( k +1) 2 ( u, v ) = rwl ( k +1) 2 ( u ′ , v ′ ) , by the definition of rwl ( k +1) 2 ( u, v ) and rwl ( k +1) 2 ( u ′ , v ′ ) , and as τ is injective, it holds that

Now, by the inductive hypothesis we have rawl ( k ) 2 ( u, v ) = rawl ( k ) 2 ( u ′ , v ′ ) . We can further transform the last equation by applying the inductive hypothesis again after unpacking the multiset. This results in

On the other hand, we have

̸

Proposition A.19. For all t ≥ 0 and all knowledge graph G , let rwl + 2 (0) ( G ) ≡ rwl (0) 2 ( G ) . The following statements hold:

For every t > 0 , rwl + 2 ( t ) ( G ) ⪯ rwl ( t ) 2 ( G )

There is a knowledge graph G ′′ and pair of nodes ( u, v ) and ( u ′ , v ′ ) such that rwl ( t ) 2 ( G ′′ , u, v ) = rwl ( t ) 2 ( G ′′ , u ′ , v ′ ) for all t ≥ 0 but rwl +(1) 2 ( G ′′ , u, v ) = rwl +(1) 2 ( G ′′ , u ′ , v ′ ) .

̸

Proof. As before, we first prove rwl + 2 ( G ) ⪯ rwl 2 ( G ) by induction on iteration t . The base case for t = 0 is trivial by the assumption. By the inductive hypothesis,

Figure 3: Graph G ′ as the counter-example in Proposition A.18. It is also shown in Proposition A.20 to prove rawl + 2 ( t ) ( u, v ) = rawl + 2 ( t ) ( u ′ , v ′ ) for all t ≥ 0 but rwl (1) 2 ( u, v ) = rwl (1) 2 ( u ′ , v ′ ) .

̸

̸

Figure 4: Two counterexamples shown in Proposition A.17 and A.19. The left graph G ′ is to show rawl ( t ) 2 ( u, v ) = rawl ( t ) 2 ( u ′ , v ′ ) for all t ≥ 0 but rawl +(1) 2 ( u, v ) = rawl +(1) 2 ( u ′ , v ′ ) , whereas the right graph G ′′ is to show rwl ( t ) 2 ( u, v ) = rwl ( t ) 2 ( u ′ , v ′ ) for all t ≥ 0 but rwl +(1) 2 ( u, v ) = rwl +(1) 2 ( u ′ , v ′ ) .

̸

Thus, it holds that rawl ( k +1) 2 ( u, v ) = rawl ( k +1) 2 ( u ′ , v ′ ) by definition of rawl ( k +1) 2 ( u, v ) and rawl ( k +1) 2 ( u ′ , v ′ ) .

For counter-example, we show the case for t ≥ 0 . Consider a relational graph G ′ = ( V ′ , E ′ , R, c, η ) such that V ′ = { u, u ′ , v, x } , E ′ = { r ( x, u ′ ) } , and R = { r } with the initial labeling η for node pairs ( u, v ) and ( u ′ , v ) satisfies rwl (0) 2 ( u, v ) = rwl (0) 2 ( u ′ , v ) and rawl (0) 2 ( u, v ) = rawl (0) 2 ( u ′ , v ) . For such graph G ′ , we consider node pair ( u, v ) and ( u ′ , v ) . For rawl ( t ) 2 where t ≥ 0 , we show by induction that rawl ( t ) 2 ( u, v ) = rawl ( t ) 2 ( u ′ , v ) . The base case is trivial by assumption. The inductive step shows that by the inductive hypothesis:

for some k . Thus, assuming rwl + 2 ( k +1) ( u, v ) = rwl +( k +1) 2 ( u ′ , v ′ ) , by the definition of rwl + 2 ( k +1) ( u, v ) and rwl +( k +1) 2 ( u ′ , v ′ ) and by the injectivity of τ , it holds that

By the similar argument in proving rawl + 2 ⪯ rawl 2 , we can split the multiset into two equations by decomposing N + r ( u ) = N r ( u ) ∪ N r -( u ) and N + r ( v ) = N r ( v ) ∪ N r -( v ) . Thus, we have

By the inductive hypothesis and unpacking the first and the third equations, we can further imply that

This would results in rwl ( k +1) 2 ( u, v ) = rwl ( k +1) 2 ( u ′ , v ′ ) by the definition of rwl ( k +1) 2 ( u, v ) and rwl ( k +1) 2 ( u ′ , v ′ )

For the counter-example, we consider a relational graph G ′′ = ( V ′′ , E ′′ , R ′′ , c, η ) such that V ′′ = { u, u ′ , v, v ′ , x, x ′ } , E ′′ = { r 1 ( u, x ) , r 2 ( u ′ , x ′ ) } , and R ′′ = { r 1 , r 2 } . For such graph G ′′ , we set the initial labeling η for node pairs ( u, v ) and ( u ′ , v ′ ) to satisfy rwl + 2 (0) ( u, v ) = rwl + 2 (0) ( u ′ , v ′ ) and rwl (0) 2 ( u, v ) = rwl (0) 2 ( u ′ , v ′ ) . For such graph G ′′ , we consider node pair ( u, v ) and ( u ′ , v ′ ) .

Consider rwl ( t ) 2 where t ≥ 0 , we show by induction that rwl ( t ) 2 ( u, v ) = rwl ( t ) 2 ( u ′ , v ′ ) . The base case is trivial by assumption. The inductive step shows that by inductive hypothesis,

On the other hand, we have

̸

Proposition A.20. For all t ≥ 0 and all knowledge graph G , let rwl (0) 2 ( G ) ≡ rawl + 2 (0) ( G ) , then the following statement holds:

̸

There is a knowledge graph G and pair of nodes ( u, v ) and ( u ′ , v ′ ) such that rwl 2 ( t ) ( G,u,v ) = rwl 2 ( t ) ( G,u ′ , v ′ ) for all t ≥ 0 but rawl + 2 (1) ( G,u,v ) = rawl + 2 (1) ( G,u ′ , v ′ ) .

̸

There is a knowledge graph G ′ and pair of nodes ( u, v ) and ( u ′ , v ′ ) such that rawl + 2 ( t ) ( G ′ , u, v ) = rawl + 2 ( t ) ( G ′ , u ′ , v ′ ) for all t ≥ 0 but rwl (1) 2 ( G ′ , u, v ) = rwl (1) 2 ( G ′ , u ′ , v ′ ) .

Proof. To show rwl 2 ( G ) does not refine rawl + 2 ( G ) , we consider a relational graph G = ( V, E, R, c, η ) such that V = { u, u ′ , v, v ′ , x, x ′ } , E = { r 1 ( v, x ) , r 2 ( v ′ , x ′ ) } , and R = { r 1 , r 2 } . We let the initial

̸

Figure 5: Counterexample shown in Proposition A.20 to prove rwl 2 ( t ) ( u, v ) = rwl 2 ( t ) ( u ′ , v ′ ) for all t ≥ 0 but rawl + 2 (1) ( u, v ) = rawl + 2 (1) ( u ′ , v ′ ) .

labeling η for node pairs ( u, v ) and ( u ′ , v ′ ) to satisfy rwl (0) 2 ( u, v ) = rwl (0) 2 ( u ′ , v ′ ) and rwl (0) 2 ( u, v ) = rwl (0) 2 ( u ′ , v ′ ) . For such graph G , we consider node pair ( u, v ) and ( u ′ , v ′ ) . For rwl ( t ) 2 where t ≥ 0 , we show by induction that rwl ( t ) 2 ( u, v ) = rwl ( t ) 2 ( u ′ , v ′ ) . The base case is trivial by assumption. The inductive step shows that by the inductive hypothesis,

On the other hand, we have

̸

Finally, to show rawl + 2 does not refine rwl 2 , we demonstrate the case for t ≥ 0 . Consider a relational graph G ′ = ( V ′ , E ′ , R ′ , c, η ) such that V ′ = { u, u ′ , v, x } , E ′ = { r ( x, u ′ ) } , and R ′ = { r } . We let the initial labeling η for node pairs ( u, v ) and ( u ′ , v ) satisfy rwl (0) 2 ( u, v ) = rwl (0) 2 ( u ′ , v ) and rawl + 2 (0) ( u, v ) = rawl + 2 (0) ( u ′ , v ) . For such graph G ′ , we consider node pair ( u, v ) and ( u ′ , v ) . For rawl + 2 ( t ) where t ≥ 0 , and show by induction that rawl + 2 ( t ) ( u, v ) = rawl + 2 ( t ) ( u ′ , v ) . The base case is trivial by assumption. The inductive step shows that by the inductive hypothesis,

On the other hand, we have

̸

Note that the counter-example graph G ′ here is identical to the one in Proposition A.18.

B Runtime analysis

In this section, we present the asymptotic time complexity of R-MPNN, C-MPNN, and a labelingtrick-based relation prediction model GraIL [30] in Table 3, taken from Zhu et al. [40]. We also consider 2 -RN, the neural architecture from Barceló et al. [4] corresponding to rwl 2 to showcase an example of higher-order GNNs. We present both the complexity of a single forward pass of the model as well as the amortized complexity of a single query to better reflect the practical use cases and the advantages stemming from parallelizability. To avoid confusion, we take CompGCN [32] as the example of R-MPNN, and basic C-MPNN, since specific architecture choices will impact the asymptotic complexity of the model.

Notation. Given a knowledge graph G = ( V, E, R, c ) , we have that | V | , | E | , | R | represents the size of vertices, edges, and relation types, respectively. d is the hidden dimension of the model, and T is the number of layers in the model.

Runtime of C-MPNNs. Let us note that C-MPNNs subsume NBFNets. Indeed, if we consider C-MPNNs without readout and set the history function as f ( t ) = 0 , we obtain (a slight generalization of) NBFNets, as stated in the paper. In terms of the runtime, C-MPNNs and NBFNets are comparable: even though the readout component of C-MPNNs incurs a linear overhead, this is dominated by other factors. Thus, as shown in Lemma 2 of Zhu et al. [40], the total complexity for a forward pass is O ( T ( | E | d + | V | d 2 )) to compute | V | queries at the same time, resulting an amortized complexity of O ( T ( | E | d | V | + d 2 )) . See Zhu et al. [40] for detailed discussion and proof.

Comparison with R-MPNNs. As directly carrying out relational message passing in R-MPNN for each triplet is costly, a common way to carry out link prediction task is to first compute all node representations and then use a binary decoder to perform link prediction. This would result in O ( T ( | E | d + | V | d 2 )) for a complete forward pass. Consequently, we obtain | R || V | 2 triplets, and since passing through binary decoder has a complexity of O ( d ) , we have that the amortized complexity for each query is O ( T ( | E | d | R || V | 2 + d 2 | R || V | + d )) . Although the complexity of a forward pass for R-MPNN and C-MPNN are the same, we compute | R || V | 2 queries with R-MPNN but only | V | queries with C-MPNN, resulting difference in amortized complexity of a single query. However, as discussed in Section 4, this comes at the cost of expressivity.

Comparison with the architectures using labeling trick. GraIL is an architecture using labeling trick and we focus on this to make the comparison concrete. In terms of runtime, architectures that rely on the labeling trick are typically slower, since they need to label both the source and the target node, as we state in Section 4. This yields worse runtime complexity for these models, and particularly for GraIL 4 , with an amortized complexity of a query being the same as the one for a forward pass, that is, O ( T ( | E | d 2 + | V | d 2 )) . In C-MPNNs, we only label the source node u , which allows parallel comparison for all queries in the form q ( u, ?) : a single forward pass would compute all hidden representations of these queries. With the labeling trick, | V | forward passes need to be carried out as each time, both source u and target v need to be specified. See Zhu et al. [40] for detailed discussion and proof.

Comparison with higher-order GNNs. Regarding higher-order GNNs, we take 2 -RN, the neural architecture from Barceló et al. [4] corresponding to rwl 2 . These higher-order models require O (( | V || E | d + | V | 2 d 2 )) in each layer (assuming each message function takes O ( d ) as before) to update all pairwise representations in a single forward pass, which is computationally prohibitive in practice. This makes the study of the amortized complexity obselete, but we provide an analysis for the sake of completeness. Similar to R-MPNN, we need an additional query-specific unary decoder for the task of link prediction on knowledge graphs, resulting in O ( d ) complexity overhead. The amortized complexity of a single query is thus O ( T ( | E | d | R || V | + d 2 | R | + d )) .

C Experiments on inductive link prediction

C.1 Details of the experiments reported in Section 6

In this section, we report the details of the experiments reported in the body of this paper. Specifically, we report the performance of some baseline models (Table 4), dataset statistics (Table 5), hyperparameters (Table 6), and the number of trainable parameters for each model variation (Table 7).

All of the model variations minimize the negative log-likelihood of positive and negative facts. We follow the partial completeness assumption [10] by randomly corrupting the head entity or the tail entity to generate the negative samples. We parameterize the conditional probability of a fact q ( u, v ) by p ( v | u, q ) = σ ( f ( h ( T ) v | u,q )) , where σ is the sigmoid function and f is 2-layer MLP. Following RotatE [29], we adopt self-adversarial negative sampling by sampling negative triples from the following distribution with α as the adversarial temperature :

where k is the number of negative samples for one positive sample and ( u ′ i , q, v ′ i ) is the i -th negative sample. Finally, w i is the weight for the i -th negative sample, given by

C.2 Experiments for evaluating the effect of initialization functions

Initialization functions (Q3). We argued that the initialization function INIT ( u, v, q ) needs to satisfy the property of target node distinguishability to compute binary invariants. To validate the impact of different initialization regimes, we conduct a further experiment which is reported in Table 8, with the same experiment settings as in Section 6. In addition to the initialization functions INIT 1 , INIT 2 , INIT 3 defined in Section 4.1, we also experiment with a simple function INIT 0 which assigns 0 to all nodes. As expected, using INIT 0 initialization, results in a very sharp decrease in model performance in WN18RR, but less so in FB15k-237. Intuitively, the model suffers more in WN18RR since there are much fewer relations, and it is harder to distinguish node pairs without an initialization designed to achieve this. Perhaps one of the simplest functions satisfying target node distinguishability criteria is INIT 1 = 1 u = v ∗ 1 , which pays no respect to the target query relation. Empirically, INIT 1 achieves strong results, showing that even the simplest function ensuring this property could boost the model performance. Interestingly, the performance of models using INIT 1 match or exceed models using INIT 2 , even though the latter additionally has a relation-specific learnable query vector. Note, however, that this shall not undermine the role of the learnable query relation: integrating the learnable query vector either in the initialization function or in the message computation function seems to suffice.

Random initialization (Q3). The idea of random node initialization is known to lead to more expressive models [26, 1]. Inspired by this, we can incorporate varying degrees of randomization to the initialization, satisfying the target node distinguishability property in expectation. The key advantage is that the resulting models are still inductive and achieve a nontrivial expressiveness gain over alternatives. In this case, the models with INIT 3 perform closer to the models with INIT 2 , but we do not see a particular advantage on these benchmarks.

D Experiments on transductive link prediction

We further conducted transductive link prediction experiments to empirically validate that C-MPNNs are more expressive than R-MPNNs via the abstraction given by rawl 2 and rwl 1 , respectively (Q4) .

D.1 Experiments on WN18RR and FB15k-237

Datasets. We use two benchmark datasets for transductive link prediction experiments, namely WN18RR [31] and FB15k-237 [9], with the provided standardized train-test split. Similar to the inductive relation prediction experiments, we augment the fact r ( u, v ) with its inverse fact r -1 ( v, u ) . The detailed data statistics are shown in Table 10.

Implementation. For basic C-MPNN without readout (denoted as C-MPNNbasic in Table 9), all hyper-parameters used are reported in Table 11, and we adopt layer-normalization [2] and short-cut connection after each aggregation and before applying ReLU. We also discard the edges that directly connect query node pairs to prevent overfitting. For RGCN, we use two layers, each with a hidden dimension of 100 . We consider the same basis-decomposition trick with a basis number equal to 30 . In addition, we train the model with a learning rate of 0 . 1 and a dropout rate of 0 . 2 , using Adam optimizer for 10 , 000 epochs. For CompGCN, we use three layers, each with hidden dimension 200 , and adopt distmult scoring function as well as element-wise multiplication as a composition function. During the training process, we consider a learning rate of 0 . 001 and a dropout rate of 0 . 1 , using Adam optimizer for 500 epochs. Additionally, since there are no node features for both of the datasets, we initialize node representations using learnable embeddings with Xavier initialization for RGCN and CompGCN. We take the results of NeuralLP from Yang et al. [35], and the results of DRUM from Sadeghian et al. [25], respectively. The best checkpoint for each model instance is selected based on its performance on the validation set. All experiments are performed on one NVIDIA V100 32GB GPU.

Evaluation. We consider filtered ranking protocol [6] with 1 negative sample per positive triplet, and report Mean Rank(MR), Mean Reciprocal Rank(MRR), and Hits@10 for each model. We also report averaged results of five independent runs for all experiments.

Results. From Table 9, it is evident that models under the class of C-MPNN consistently outperform R-MPNN models across both datasets and all evaluation metrics, with only one exception on FB15k237 with NeuralLP. This aligns with our theoretical understanding, as C-MPNN models are inherently more expressive than R-MPNNs. In particular, the C-MPNNbasic model stands out by achieving the lowest MR and the highest values for both MRR and Hits@10, surpassing all other models by a significant margin, underscoring its efficiency and robustness in transductive knowledge graph completion tasks.

D.2 Experiments on biomedical datasets

To compare R-MPNNs and C-MPNNs on large-scale graphs, we carried out additional transductive knowledge graph completion experiments on biomedical datasets: Hetionet [15] and ogbl-biokg [16].

Datasets. Hetionet [15] is a large biomedical knowledge graph that integrates data from 29 different public databases, representing various biological connections and associations, and ogbl-biokg is a large-scale biomedical knowledge graph dataset, developed as a part of the Open Graph Benchmark (OGB) [16] suite. We have used the provided standardized train-test split for both datasets and similarly augmented every fact r ( u, v ) with its inverse relation r -1 ( u, v ) . We present the detailed dataset statistics in Table 13.

Implementation. We study the basic C-MPNN without readout, referred to as C-MPNN in the Table 12, the epoch number is set to 1 due to the datasets being larger and denser, leading to longer execution times for C-MPNN. All hyperparameters used for C-MPNN are reported in Table 14, and we adopt layer-normalization [2] and short-cut connection after each aggregation and before applying ReLU. We also discard the edges that directly connect query node pairs. For RGCN, we consider a batch size of 65536 , a learning rate of 0 . 001 , and 100 epochs for both datasets. On Hetionet, we use a hidden dimension size of 200 , and a learning rate of 0 . 001 across 4 layers. On ogbl-biokg, we consider a dimension size of 500 , and a dropout rate of 0 . 2 across 2 layers. For CompGCN on Hetionet, the parameters include a batch size of 65536 , 100 epochs, a dimension size of 200 , a learning rate of 0 . 01 , and a single layer. We adopt distmult scoring function and element-wise multiplication as a composition function. On ogbl-biokg, CompGCN results cannot be reproduced to the best of our effort due to an out-of-memory (OOM) error. The best checkpoint for each model instance is selected based on its performance on the validation set. All experiments are performed on one NVIDIA V100 32GB GPU. In addition, since there are no node features for both of the datasets, we initialize node representations using learnable embeddings with Xavier initialization for RGCN and CompGCN.

Evaluation. We consider filtered ranking protocol [6], but restrict our ranking to entities of the same type. With each positive triplet, we use 32 number of negative samples and report the averaged results of Mean Reciprocal Rank(MRR), Hits@1, and Hits@10 over five independent runs for each model.

Results. From Table 12, it is evident that C-MPNN outperforms R-MPNN on both datasets by a large margin, despite the challenges posed by the large and dense nature of biomedical knowledge graphs. These results are reassuring as they have further consolidated our theory, showing that the expressivity gain of C-MPNNs compared to R-MPNNs has a significant impact on real-world biomedical datasets.

E Evaluating the power of global readout on a synthetic dataset

We constructed a synthetic experiment to showcase the power of global readout ( Q5 ).

Dataset. We proposed a synthetic dataset TRI-SQR, which consists of multiple pairs of knowledge graphs in form ( G 1 , G 2 ) s.t. G 1 = ( V 1 , E 1 , R, c 1 ) , G 2 = ( V 2 , E 2 , R, c 2 ) where R = { r 0 , r 1 , r 2 } . For each pair, we constructed as follows:

Generate an Erdos-Renyi graph G init with 5 nodes and a random probability p . We randomly select one of the nodes as the source node u .

G 1 is constructed by disjoint union G init with two triangles, one with edges relation of r 1 , and the other with r 2 . The target query is r 3 ( u, v ) for all v in a triangle with an edge relation of r 1 .

Similarly, G 2 is constructed by disjoint union another copy of G init with two squares, one with edges relation of r 1 , and the other with r 2 . The target query is r 3 ( u, v ) for all v in a square with an edge relation of r 2 .

One example of such a pair can be shown in Figure 6. We generate 100 graph pairs and assign 70 pairs as the training set and the remaining 30 pairs as the testing set ( 140 training graphs and 60 testing graphs). In total, there are 490 training triplets and 210 testing triplets.

Objective. For all node v / ∈ G init, we want a higher score for all the links r 0 ( u, v ) if either v is in a triangle consists of r 1 relation or v is in a square consists of r 2 relation, and a lower score otherwise. For negative sampling, we choose counterpart triplets for each graph, that is, we take r 0 ( u, v ) for all v in a triangle with an edge relation of r 2 in G 1 and in a square with an edge relation of r 1 in G 2 .

Model architectures. We have considered two model architectures, namely C-MPNNs with INIT = INIT 2 , AGG = sum , MSG = MSG 2 r , and f ( t ) = t without sum global readout:

u

•

•

•

•

•

•

•

•

•

•

G

1

u

•

•

•

•

•

•

•

•

•

•

•

•

G

2

r

1

r

1

r

2

r

2

r

2

r

1

r

2

r

1

r

1

r

2

r

1

r

2

r

2

r

0

r

0

r

0

r

1

r

2

r

1

r

2

r

1

r

2

r

2

r

1

r

1

r

1

r

1

r

2

r

2

r

2

r

2

r

0

r

0

r

0

r

0

and the same C-MPNN architecture with sum global readout:

Design. We claim that C-MPNNs with sum readout can correctly predict all testing triplets, whereas C-MPNNs without sum readout will fail to learn this pattern and achieve 50% as random guessing. Theoretically, the TRI-SQR dataset is designed in such a way that any R-MPNN model assigns identical node representations to nodes in triangles and squares with the same relation type. Consequently, any C-MPNN model without global readout will be unable to determine which graph between G 1 and G 2 in the graph pair is being predicted, making it challenging to learn the conditional representation h u | v,q . However, we anticipate that a C-MPNN model with sum readout can differentiate between G 1 and G 2 in each pair, as it can access global information like the total number of nodes in the graph. This allows it to accurately identify the graph being predicted in the graph pair, even when the representations of the triangle and square with the same relation are identical. As a result, it can learn the target rules and achieve 100% accuracy.

Experiment details. We configure each model variant with four layers and 32 hidden dimensions per layer. We set a learning rate of 1e-4 for both models and train them for 500 epochs. Empirically, we find that C-MPNN with global sum readout achieves 100% accuracy, while C-MPNN without global sum readout reaches 50% accuracy as random guesses, which is consistent with our theoretical expectations.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on graph neural networks (GNNs) for link prediction on knowledge graphs. It introduces the concept of conditional message-passing neural networks, which compute pairwise node representations based on a source node and a query relation. The paper analyzes the expressive power of these networks with a relational Weisfeiler-Leman algorithm and a logical characterization. Experimental analysis is conducted to validate the theoretical findings.\n\nSTRENGTHS:\n1. The paper offers a systematic understanding of conditional message-passing neural networks for link prediction on knowledge graphs. \n\n2. The paper provides insights into the expressive power of conditional message-passing neural networks over the general message-passing neural networks. \n\n3. The paper conducts experimental analysis to empirically validate the theoretical findings. It explores the impact of different model choices, e.g., initialization, global readout functions, providing practical insights into the performance of graph neural networks for link prediction on knowledge graphs.\n\nWEAKNESSES:\n1.  The title and some lines in the introduction seem to be a little bit overclaimed. For instance, the title named a theory of link prediction seems to be claimed on the general link prediction task while the main discussion in the main paper is about the Knowledge Graph link prediction. besides, it is hard for me to understand lines 32-36. I could see from this paragraph that the paper focuses on the NBFNet-based model with conditional message passing. Nonetheless, it seems not so clear to me what higher-order messaging passing neural network computing the pairwise representation on the node. I would like to suggest authors to strengthen the focus of the paper at the beginning and then detail the difference between the RGCN, and CompGCN models with NBFNet. \n2.  The second bullet of the contribution part at line 52 claims that the paper designed a relational Weisfeiler-Lemma algorithm. It leads to the potential misleading that the paper is the first to propose the relational Weisfeiler-Lemma algorithm. Nonetheless, as discussed in line 97 in related work, the relationship is already included in the discussion in 'Weisfeiler and Leman go relational'. I would like to suggest the authors for a clear comparison with the related work on existing relational Weisfeiler-Lemma algorithms. \n3.  The intuition for logical characterization is not so clear. It seems to be hard for me to understand the logical characterization. How it contributes to the expressiveness is not so clear. I would like to suggest the author have more discussion on motivation and why the contribution is significant.\n4.  Lack of explanation on why only focuses on the inductive Link Prediction setting. Despite the NBFNet shows strong ability typically inductive setting. Nonetheless, the algorithm seems not so relevant in the transductive or inductive setting. I still think the experiments on both transductive and inductive settings should be taken into consideration.\n\nQUESTIONS:\nSee in Weakness\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates graph neural networks (GNNs) for link prediction in knowledge graphs. The authors propose a GNN that generalizes several existing techniques based on the labeling trick and NBFNets and investigate its expressive power by relating it to the Weisfeiler-Leman method and giving a logical characterization. Different design choices of the architecture are explored experimentally.\n\nSTRENGTHS:\n* A highly relevant problem with various applications is addressed and neural architectures for related tasks are thoroughly analyzed, giving new theoretical insights. As the number of papers published in this area grows rapidly, it is of utmost importance to explain the advantage of practical design choices and to work out the benefits of specific architectures. The paper is a solid contribution in this direction.\n* The authors give a good overview of related work, including several recent results, and place their work well in the context of existing approaches. The paper continues recent work and generalizes it, which is a valuable contribution to fostering future work.\n* The paper is well-written and enjoyable to read; the presentation is on a high level.\n\nWEAKNESSES:\n* The experimental comparison focuses on the design choices of the proposed architecture but does not compare to existing works. This would be desirable, in particular, since higher-order GNNs subsume some of the techniques (as stated in the paper). The trade-off between efficiency and expressivity appears to be crucial but is neither investigated theoretically in terms of time complexity nor experimentally in terms of measured runtimes.\n* The presentation of conditional messages passing remains abstract. It would be helpful to come up with an illustrative example, if possible.\n\nMinor comments:\n* l176: $\\epsilon_u$ should be explained\n* l186: $\\mathbf{z}_q$ as argument of MSG should be just $q$ following the notation introduced earlier \n* l370: \"Thus, thus\"\n\nQUESTIONS:\n* How does the approach compare to NBFNets, architecture using the labeling trick, and higher-order WL-type GNNs regarding (i) accuracy and (ii) running time?\n\nLIMITATIONS:\nLimitations are addressed sufficiently.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors note that while GNNs are understood well in the context of simple graphs, there is a lack of comprehensive understanding when it comes to knowledge graphs. This study aims to systematically explore the use of GNNs in knowledge graphs, specifically in relation to the task of link prediction. The research includes a unifying view on different, seemingly unrelated models, potentially revealing new ones. The study evaluates the expressive power of different models using the relational Weisfeiler-Leman algorithm. The theoretical findings help clarify the advantages of some commonly used practical design choices, and these theories are supported by empirical validation.\n\nSTRENGTHS:\nI have read the appendix of the paper. The technical details are clearly articulated, and the proof statements are clear and persuasive.\n\nThe authors introduce conditional message-passing neural networks and define a relational Weisfeiler-Leman algorithm to substantiate that these neural networks can match the expressive power of this algorithm.\n\nThe initialization progress is interesting.\n\nWEAKNESSES:\nIn the background section, the author first mentioned G = (V, E, R, c) while change it to G = (V, E, R, x) later. I know the author says it usually should be x instead of c. But can we directly use G = (V, E, R, x) at first to prevent confusion?\n\nMore GNN baseline method should be involved in the experiments.\n\nThe proof process is commendable, but could it be possible to draw a conclusion from it and incorporate it into the main text instead of the appendix?\n\nQUESTIONS:\nCan the author add more baselines?\n\nAlthough the authors have already provided two datasets, each with four different versions, I think this is adequate, so I have not listed it as a weakness. However, it would be good if the authors could include one or two additional datasets.\n\nLIMITATIONS:\nAs the author mentioned, the method is limited to binary task such as link prediction.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper explores the expressive power of several GNNs designed for link prediction in knowledge graphs. The authors propose a conditional message passing framework with various designs for each component, a generalization of NBFNets. They also prove that the proposed framework can match the expressive power of a relational Weisfeiler-Leman algorithm. The experimental results demonstrate the impact of different model choices.

STRENGTHS:
1. The paper provides a theoretical understanding of some relational GNNs for knowledge graphs, such as NBFNets.
2. The proposed C-MPNN is reasonable, generalizing GNNs that can compute pairwise representations.
3. The authors study the effect of different model architectures, adding depth to the exploration of GNNs.

WEAKNESSES:
1. Using relational asymmetric local 2-WL (rawl2) to measure the expressive power of C-MPNN is a little awkward. The definition of rawl2 is quite similar to C-MPNN, leading to potential confusion.
2. The experiments do not verify some theoretical findings, focusing mainly on different choices of model design. Certain theoretical discoveries, such as logical characterization, are not confirmed.
3. The experiments are primarily conducted on two datasets with differing data splits, leading to inconsistencies. More datasets and deeper analysis may be necessary to solidify the findings.

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-31