Title: Calibrate and Boost Logical Expressiveness of GNN Over Multi-Relational and Temporal Graphs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Knowledge Graphs

First-Order Logic

Temporal Knowledge Graph

Graph Neural Networks
Abstract: As a powerful framework for graph representation learning, Graph Neural Networks (GNNs) have garnered significant attention in recent years. However, to the best of our knowledge, there has been no formal analysis of the logical expressiveness of GNNs as Boolean node classifiers over multi-relational graphs, where each edge carries a specific relation type. In this paper, we investigate $\mathcal{FOC}_2$, a fragment of first-order logic with two variables and counting quantifiers. On the negative side, we demonstrate that the R$^2$-GNN architecture, which extends the local message passing GNN by incorporating global readout, fails to capture $\mathcal{FOC}_2$ classifiers in the general case. Nevertheless, on the positive side, we establish that R$^2$-GNNs models are equivalent to $\mathcal{FOC}_2$ classifiers under certain restricted yet reasonable scenarios. To address the limitations of R$^2$-GNNs regarding expressiveness, we propose a simple graph transformation technique, akin to a preprocessing step, which can be executed in linear time. This transformation enables R$^2$-GNNs to effectively capture any $\mathcal{FOC}_2$ classifiers when applied to the "transformed" input graph. Moreover, we extend our analysis of expressiveness and graph transformation to temporal graphs, exploring several temporal GNN architectures and providing an expressiveness hierarchy for them. To validate our findings, we implement R$^2$-GNNs and the graph transformation technique and conduct empirical tests in node classification tasks against various well-known GNN architectures that support multi-relational or temporal graphs. Our experimental results consistently demonstrate that R$^2$-GNN with the graph transformation outperforms the baseline methods on both synthetic and real-world datasets

Calibrate and Boost Logical Expressiveness of GNN Over Multi-Relational and Temporal Graphs

Abstract

As a powerful framework for graph representation learning, Graph Neural Networks (GNNs) have garnered significant attention in recent years. However, to the best of our knowledge, there has been no formal analysis of the logical expressiveness of GNNs as Boolean node classifiers over multi-relational graphs, where each edge carries a specific relation type. In this paper, we investigate FOC 2 , a fragment of first-order logic with two variables and counting quantifiers. On the negative side, we demonstrate that the R 2 -GNN architecture, which extends the local message passing GNN by incorporating global readout, fails to capture FOC 2 classifiers in the general case. Nevertheless, on the positive side, we establish that R 2 -GNN models are equivalent to FOC 2 classifiers under certain restricted yet reasonable scenarios. To address the limitations of R 2 -GNN regarding expressiveness, we propose a simple graph transformation technique, akin to a preprocessing step, which can be executed in linear time. This transformation enables R 2 -GNN to effectively capture any FOC 2 classifiers when applied to the "transformed" input graph. Moreover, we extend our analysis of expressiveness and graph transformation to temporal graphs, exploring several temporal GNN architectures and providing an expressiveness hierarchy for them. To validate our findings, we implement R 2 -GNN and the graph transformation technique and conduct empirical tests in node classification tasks against various well-known GNN architectures that support multi-relational or temporal graphs. Our experimental results consistently demonstrate that R 2 -GNN with the graph transformation outperform the baseline methods on both synthetic and real-world datasets. The code is available at https://github.com/hdmmblz/multi-graph .

1 Introduction

Graph Neural Networks (GNNs) have become a standard paradigm for learning with graph structured data, such as knowledge graphs Park et al. [2019], Tena Cucala et al. [2021], Wang et al. [2023] and molecules Hao et al. [2020], Gasteiger et al. [2021], Guo et al. [2021]. GNNs take as input a graph where each node is labelled by a feature vector, and then they recursively update the feature vector of each node by processing a subset of the feature vectors from the previous layer. For example, many GNNs update a node's feature vector by combining its value in the previous layer with the output of some aggregation function applied to its neighbours ' feature vectors in the previous layer; in this case, after k iterations, a node's feature vector can capture structural information about the node's k -hop neighborhood. GNNs have proved to be very efficient in many applications like knowledge graph completion and recommender systems. Most previous work on GNNs mainly revolves around finding

GNN architectures (e.g. using different aggregation functions or graph-level pooling schemes) which offer good empirical performance Kipf and Welling [2016], Xu et al. [2018], Corso et al. [2020]. The theoretical properties of different architectures, however, are not yet well understood.

In Xu et al. [2018], the authors first proposed a theoretical framework to analyze the expressive power of GNNs by establishing a close connection between GNNs and the Weisfeiler-Lehman (1-WL) test for checking graph isomorphism. Similarly, Geerts and Reutter [2022] provides an elegant way to easily obtain bounds on the separation power of GNNs in terms of the Weisfeiler-Leman (k-WL) tests. However, the characterization in terms of the Weisfeiler-Lehman test only calibrates distinguishing ability. It cannot answer which Boolean node classifier can be expressed by GNNs . To this end, Barceló et al. [2020] consider a class of GNNs named ACR-GNNs proposed in Battaglia et al. [2018], where the update function uses a 'global' aggregation of the features of all nodes in the graph in addition to the typical aggregation of feature vectors of neighbour nodes. Then, the authors of the paper prove that in the single-relational 2 scenario, ACR-GNNs can capture every Boolean node classifier expressible in the logic FOC 2 .

However, most knowledge graphs need multiple relation types. For example, in a family tree, there are multiple different relation types such as "father" and "spouse". In this paper, we consider the abstraction of a widely used GNN architecture called R-GCN Schlichtkrull et al. [2018], which is applicable to multi-relational graphs. Following Barceló et al. [2020], we define R 2 -GNN as a generalization of R-GCN by adding readout functions to the neighborhood aggregation scheme. We show that although adding readout functions enables GNNs to aggregate information of isolated nodes that can not be collected by the neighborhood-based aggregation mechanism, R 2 -GNN are still unable to capture all Boolean node classifiers expressible as formulas in logic FOC 2 in multi-relational scenarios if applied 'directly' to the input. This leaves us with the following questions: (1) Are there reasonable and practical sub-classes of multi-relational graphs for which FOC 2 can be captured by R 2 -GNN? (2) Is there some simple way to encode input graphs, so that all FOC 2 node classifiers can be captured by R 2 -GNN for all multi-relational graphs?

In this paper, we provide answers to the above questions. Moreover, we show that our theoretical findings also transfer to temporal knowledge graphs, which are studied extensively in Park et al. [2022] and Gao and Ribeiro [2022]. In particular, we leverage the findings from Gao and Ribeiro [2022] which shows that a temporal graph can be transformed into an 'equivalent' static multirelational graph. Consequently, our results, originally formulated for static multi-relational graphs, naturally extend to the domain of temporal knowledge graphs. Our contributions are as follows:

We calibrate the logic expressiveness of R 2 -GNN as node classifiers over different subclasses of multi-relational graphs.

In light of some negative results about the expressiveness of R 2 -GNN found in the multirelational scenario, there is a compelling need to boost the power of R 2 -GNN. To address this challenge, we propose a graph transformation and show that such a transformation enables R 2 -GNN to capture each classifier expressible as a FOC 2 formula in all multi-relational graphs.

We expand the scope of expressiveness results and graph transformation from static multirelational graphs to temporal settings. Within this context, we propose several temporal GNN architectures and subject them to a comparative analysis with frameworks outlined in Gao and Ribeiro [2022]. Ultimately, we derive an expressiveness hierarchy.

2 Preliminaries

2.1 Multi-relational Graphs

A multi-relational graph is a 4 -tuple G = ( V, E , P 1 , P 2 ) , where V , P 1 , P 2 are finite sets of nodes , types and relations (a.k.a, unary/binary predicates) 3 , respectively, and E is a set of triples of the form ( v 1 , p 2 , v 2 ) or ( v, type, p 1 ) , where p 1 ∈ P 1 , p 2 ∈ P 2 , v 1 , v 2 , v ∈ V , and type is a special symbol.

Next, given arbitrary (but fixed) finite sets P 1 and P 2 of unary and binary predicates, respectively, we define the following three kinds of graph classes:

a universal graph class can be any set of graphs of the form ( V, E , P 1 , P 2 ) .

a bounded graph class is a universal graph class for which there exists n ∈ N such that each graph in the class has no more than n nodes;

a simple graph class is a universal graph class where for each graph ( V, E , P 1 , P 2 ) in the class, and for each pair of nodes v 1 , v 2 ∈ V , there exists at most one triple in E of the form ( v 1 ,p 2 ,v 2 ) , where p 2 ∈ P 2 .

We typically use symbols G u , G b , and G s to denote universal, bounded, and simple graph classes, respectively.

Definition 1. For a given graph class over predicates P 1 and P 2 , a Boolean node classifier is a function C such that for each graph G = ( V, E , P 1 , P 2 ) in that graph class, and each v ∈ V , C classifies v as true or false .

2.2 Graph Neural Networks

Node Encoding We leverage a GNN as a Boolean node classifier for multi-relational graphs, which cannot be directly processed by GNN architectures, requiring graphs where each node is labelled by an initial feature vector. Therefore, we require some form of encoding to map a multi-relational graph to a suitable input for a GNN. Such an encoding should keep graph permutation invariance Geerts and Reutter [2021] since we don't want a GNN to have different outputs for isomorphic graphs. Inspired by Liu et al. [2021] for a multi-relational graph G = ( V, E , P 1 ,P 2 ) and an ordering p 1 , p 2 , · · · , p k of the predicates in P 1 , we define an initialization function I ( · ) which maps each node v ∈ V to a Boolean feature vector I ( v ) = x v with a fixed dimension | P 1 | , where the i th component of the vector is set to 1 if and only if the node v is of the type p i , that is, ( x v ) i = 1 if and only if ( v, type, p i ) ∈ E . If P 1 is an empty set, we specify that each node has a 1-dimension feature vector whose value is 1 . Clearly, this encoding is permutation invariant.

R-GNN R-GCN Schlichtkrull et al. [2018] is a widely-used GNN architecture that can be applied to multi-relational graphs. By allowing different aggregation and combination functions, we extend

R-GCN to a more general form which we call R-GNN. Formally, let { { A ( i ) j } | P 2 | j =1 } L i =1 and { C ( i ) } L i =1

be two sets of aggregation and combination functions. An R-GNN computes vectors x ( i ) v for every node v of the multi-relational graph G = ( V, E , P 1 ,P 2 ) on each layer i , via the recursive formula

where x (0) v is the initial feature vector as encoded by I ( · ) , { {·} } denotes a multiset , ( · ) | P 2 | j =1 denotes a tuple of size | P 2 | , N G,j ( v ) denotes the neighbours of v via a binary relation p j ∈ P 2 , that is, nodes w ∈ V such that ( v,p j ,w ) ∈ E .

R 2 -GNN R 2 -GNN extends R-GNN by specifying readout functions { R ( i ) } L i =1 , which aggregates the feature vectors of all the nodes in a graph. The vector x ( i ) v of each node v in G on each layer i , is computed by the following formula

Every layer in an R 2 -GNN first computes the aggregation over all the nodes in G ; then, for every node v , it computes the aggregation over the neighbors of v ; and finally, it combines the features of v with the two aggregation vectors; the result of this operation is the new feature vector for v . Please note that an R-GNN can be seen as a special type of R 2 -GNN where the combination function simply ignores the output of the readout function.

It is worth noting that R-GNN as well as R 2 -GNN is not a specific model architecture; it is a framework that contains a bunch of different GNN architectures. In the paper, we mentioned it's generalized from R-GCN (Schlichtkrull et al. [2018]), but our primary objective is to establish a comprehensive framework that serves as an abstraction of most Message-Passing GNNs (MPGNN).

In the definitions (Equations (1) and (2)), the functions can be set as any functions, such as matrix multiplications or QKV-attentions. Most commonly used GNN such as R-GCN (Schlichtkrull et al. [2018]) and R-GAT (Busbridge et al. [2019]) are captured (upper-bounded) within our R-GNN frameworks. Other related works, such as (Barceló et al. [2020], Huang et al. [2023], Qiu et al. [2023]) also use intrinsically the same framework as our R-GNN/R 2 -GNN, which has been widely adopted and studied within the GNN community. We believe that analyzing these frameworks can yield common insights applicable to numerous existing GNNs

GNN-based Boolean node classifier In order to translate the output of a GNN to a Boolean value, we apply a Boolean classification function CLS : R d →{ true,false } , where d is the dimension of the feature vectors x L v . Hence, a Boolean node classifier based on an R 2 -GNN M proceeds in three steps: (1) encode the input multi-relational graph G as described above, (2) apply the R 2 -GNN, and (3) apply CLS to the output of the R 2 -GNN. This produces a true or false value for each node of G . In what follows, we abuse the language and represent a family of GNN-based Boolean node classifiers by the name of the corresponding GNN architecture; for example, R 2 -GNN is the set of all R 2 -GNN-based Boolean node classifiers.

2.3 Logic FOC 2 Formulas

In this paper, we focus on the logic FOC 2 , a fragment of first-order logic that only allows formulas with at most two variables, but in turn permits to use counting quantifiers . Formally, given two finite sets P 1 and P 2 of unary and binary predicates, respectively, a FOC 2 formula φ is inductively defined according to the following grammar:

where x/y in the above rules can be replaced by one another. But please note that x and y are the only variable names we are allowed to use (Though we can reuse these two names). In particular, a FOC 2 formula φ with exactly one free variable x represents a Boolean node classifier for multi-relational graphs as follows: a node v is assigned to true iff the formula φ v obtained by substituting x by v is satisfied by the (logical) model represented by the multi-relational graph. Similarly as the GNN-based Boolean node classifiers, in what follows, we abuse the language and represent the family of FOC 2 Boolean node classifiers by its name FOC 2 .

2.4 Inclusion and Equality Relationships

In this paper, we will mainly talk about inclusion/non-inclusion/equality/strict-inclusion relationships between different node classifier families on certain graph classes. To avoid ambiguity, we give formal definitions of these relationships here. These definitions are all quite natural.

Definition 2. For any two sets of node classifier A,B , and graph class G , We say:

A ⊆ B on G , iff for any node classifier a ∈ A , there exists some node classifier b ∈ B such that for all graph G ∈ G and v ∈ V ( G ) , it satisfies a ( G,v ) = b ( G,v ) (Namely, a and b evaluate the same for all instances in G ). It implies B is more expressive than A on G .

A ⊈ B on G , iff the above condition in item 1 doesn't hold.

A ⊊ B on G , iff A ⊆ B but B ⊈ A . It implies B is strictly more expressive than A on G .

A = B on G , iff A ⊆ B and B ⊆ A . It implies A and B has the same expressivity on G .

3 Related Work

The relationship between first-order logic and the Weisfeiler-Lehman test was initially established by Cai et al. [1989]. Subsequently, more recent works such as Xu et al. [2018], have connected the Weisfeiler-Lehman test with expressivity of GNN. This line of research has been followed by numerous studies, including Maron et al. [2020], which explore the distinguishability of GNNs using the Weisfeiler-Lehman test technique. In particular, Barceló et al. [2020] introduced the calibration of logical expressivity in GNN-based classifiers and proposed a connection between FOC 2 and R 2 -GNN in single-relational scenario. This led to the emergence of related works, such as Huang et al. [2023], Geerts and Reutter [2021], and Qiu et al. [2023], all of which delve into the logical expressivity of GNNs. Moreover, the theoretical analysis provided in Gao and Ribeiro [2022] has inspired us to extend our results to temporal graph scenarios.

G

1

:

a

b

p

1

p

2

d

c

p

1

p

2

G

2

:

a

b

c

d

p

1

p

2

p

1

p

2

Node classifier: φ ( x ) := ∃ ≥ 1 y ( p 1 ( x,y ) ∧ p 2 ( x,y )) .

4 Logic expressiveness of R 2 -GNN in multi-relational graphs

Our analysis begins with the observation that certain Boolean classifiers can be represented as FOC 2 formulas, but remain beyond the expressiveness of any R 2 -GNN (and consequently, any R-GNN or R-GCN). An illustrative example of this distinction is provided in Figure 1. In this example, we make the assumption that P 1 is empty, thereby ensuring that all nodes in both G 1 and G 2 possess identical initial feature vectors. Additionally, P 2 is defined to comprise precisely two relations, namely, p 1 and p 2 . It is evident that no R 2 -GNN can distinguish the node a in G 1 from node a in G 2 - that is, when an R 2 -GNN performs the neighbour-based aggregation, it cannot distinguish whether the p 1 -neighbour of a and the p 2 -neighbour of a are the same. Moreover, the global readout aggregation cannot help in distinguishing those nodes because all nodes have the same feature vector.

We proceed to formalize this intuition and, in the reverse direction, offer a corresponding result. We demonstrate that there exist Boolean classifiers that fall within the scope of R 2 -GNN but elude capture by any FOC 2 formula.

Proposition 3. FOC 2 ̸⊆ R 2 -GNN and R 2 -GNN ̸⊆ FOC 2 on some universal graph class.

We prove Proposition 3 in the Appendix. Here, we give some intuition about the proof. The first result is proved using the example shown in Figure 1, which we have already discussed. To show R 2 -GNN ̸⊆ FOC 2 , we construct a classifier c which classifies a node into true iff the node has a larger number of r 1 -type neighbors than that of r 2 -type neighbors. We can prove that we can easily construct an R 2 -GNN to capture c . However, for FOC 2 , this cannot be done, since we can only use counting quantifiers expressing that there exist at most or at least a specific number of neighbours connected via a particular relation, but our target classifier requires comparing indefinite numbers of neighbours via two relations. Thus, we proceed by contradiction, assume that there exists a FOC 2 classifier equivalent to c , and then find two large enough graphs with nodes that cannot be distinguished by the classifier (but can be distinguished by c ).

In some real-world applications, it is often possible to find an upper bound on the size of any possible input graph or to ensure that any input graph will contain at most one relation between every two nodes. For this reason, we next present restricted but positive&practical expressiveness results on bounded and simple graph classes.

Theorem 4. FOC 2 ⊆ R 2 -GNN on any simple graph class, and FOC 2 ⊊ R 2 -GNN on some simple graph class.

The key idea of the construction is that we will first transform the FOC 2 formula into a new form which we call relation-specified FOC 2 (an equivalent form to FOC 2 , see more details in our Appendix), and then we are able to construct an equivalent R 2 -GNN inductively over the parser tree of the transformed formula.

Having Theorem 4, one may wonder about the inclusion relationship of R 2 -GNN and FOC 2 in the backward direction. In Proposition 3, we showed that for arbitrary universal graph classes, this inclusion relationship fails. However, given a bounded graph class, we can show that for each R 2 -GNN Boolean node classifier, one can write an equivalent FOC 2 classifier. An intuition about why this is the case is that all graphs in a bounded graph class will have at most n constants, for some known n ∈ N , so for each R 2 -GNN classifier, we can construct an equivalent FOC 2 classifier with a finite number of sub-formulas to recover the features obtained at different layers of R 2 -GNN.

Theorem 5. R 2 -GNN ⊆ FOC 2 on any bounded graph class, and R 2 -GNN ⊊ FOC 2 on some bounded graph class.

G

a

b

p

1

p

2

G

′

F

a

ab

b

ba

p

1

p

2

aux2

aux1

aux1

Combining Theorem 4 and Theorem 5, we have the following corollary.

Corollary 5.1. R 2 -GNN = FOC 2 on any bounded simple graph class.

At Last, one may be curious about the complexity of logical classifier in Theorem 5. Here we can give a rather loose bound as follows:

Theorem 6. For any bounded graph class G b . Suppose any G ∈ G b has no more than N nodes, and G b has unary predicate set P 1 and relation (binary predicate) set P 2 . Let m 1 := | P 1 | ,m 2 := | P 2 | , then for any node classifier c , suppose c can be represented as an R 2 -GNN with depth (layer number) L , then by Theorem 5 there is a FOC 2 classifier φ equivalent to c over G b , and the following hold:

The quantifier depth of φ is no more than L .

The size of φ (quantified by the number of nodes of φ 's parse tree) is no more than 2 2 f ( L ) , where f ( L ) := 2 2 2( N +1) f ( L -1) ,f (0) = O (2 2 2( m 1 + m 2 ) ) .

The key idea of Theorem 6 is the following: First, by Lemma 27 in our appendix, the combination of ALL FOC 2 logical classifiers with quantifier depth no more than L can already distinguish accepting and rejecting instances of c . Then by Proposition 26 (This is a key point of this bound; please refer to our appendix), We know the number of intrinsically different bounded-depth FOC 2 classifiers is finite, so we only need to get an upper bound on this number. Finally, we can get the desired bound by iteratively using the fact that a boolean combination of a set of formulas can be always written as DNF (disjunctive normal form). The tower of power of two comes from L rounds of DNF enumerations. Although the bound seems scary, it is a rather loose bound. We give a detailed proof of Theorem 6 in the appendix along with the proof of Theorem 5.

5 R 2 -GNN capture FOC 2 over transformed multi-relational graphs

As we pointed out in the previous section, one of the reasons why R 2 -GNN cannot capture FOC 2 classifiers over arbitrary universal graph classes is that in multi-relational graphs, they cannot distinguish whether information about having a neighbour connected via a particular relation comes from the same neighbour node or different neighbour nodes. Towards solving this problem, we propose a graph transformation F (see Definition 7), which enables R 2 -GNN to capture all FOC 2 classifiers on multi-relational graphs. Similar transformation operations have also been used and proved to be an effective way to encode multi-relational graphs in previous studies, e.g., MGNNs Tena Cucala et al. [2021], Indigo Liu et al. [2021] and Time-then-Graph Gao and Ribeiro [2022].

Definition 7. Given a multigraph G = ( V, E , P 1 , P 2 ) , the transformation F will map G to another graph F ( G ) = ( V ′ , E ′ , P ′ 1 , P ′ 2 ) with changes described as follows:

for any two nodes a, b ∈ V , if there exists at least one relation p ∈ P 2 between a and b , we add two new nodes ab and ba to V ′ .

we add a new unary predicate { primal } and two new binary predicates { aux1 , aux2 }. Hence, F ( P 1 ) := P ′ 1 = P 1 ∪ { primal } , and F ( P 2 ) := P ′ 2 = P 2 ∪ { aux1 , aux2 } . For each node v ′ ∈ V ′ , primal ( v ′ ) = 1 iff v ′ is also in V ; otherwise, primal ( v ′ ) = 0 ;

for each triplet of the form ( a,p 2 , b ) in E , we add to E ′ four new triples: ( ab , aux1 , a ) , ( ba , aux1 , b ) and ( ab , aux2 , ba ) as well as ( ab ,p 2 , ba ) .

An example is in Figure 2. We can see that after applying the graph transformation , we need to execute two more hops to propagate information from node a to node b . However, now we are able to distinguish whether the information about different relations comes from the same node or different nodes. This transformation can be implemented and stored in linear time/space complexity O ( | V | + |E| ) , which is very efficient.

Definition 8. Given a classifier C and a transformation function F , we define C ◦ F to be a new classifier, an extension of C with an additional transformation operation on the input graph.

With graph transformation F , we get a more powerful class of classifiers than R 2 -GNN. We analyze the logical expressiveness of R 2 -GNN ◦ F in multi-relational graphs, which means first transform a graph G to F ( G ) and then run an R 2 -GNN on F ( G ) . We will see in the following that this transformation F boosts the logical expressiveness of R 2 -GNN prominently.

Theorem 9. R 2 -GNN ⊆ R 2 -GNN ◦ F on any universal graph class.

Theorem 10. FOC 2 ⊆ R 2 -GNN ◦ F on any universal graph class.

Theorem 9 demonstrates that R 2 -GNN with graph transformation F have more expressiveness than R 2 -GNN; and Theorem 10 shows the connection between FOC 2 and R 2 -GNN equipped with graph transformation F . We depict their relations in Figure 3. Theorem 9

R

2

-GNN

◦

F

R

2

-GNN

FOC

2

is a natural result since no information is lost in the process of transformation, while Theorem 10 is an extension on Theorem 4, whose formal proofs can be found in the Appendix. As for the backward direction, we have the result shown in Theorem 11.

Theorem 11. R 2 -GNN ◦ F ⊆ FOC 2 on any bounded graph class.

The proof of the theorem is relatively straightforward based on previous results: by Theorem 5, it follows that R 2 -GNN ◦ F ⊆ FOC 2 ◦ F on any bounded graph class. Then, it suffices to prove FOC 2 ◦ F ⊆ FOC 2 , which we do by using induction over the quantifier depth.

By combining Theorem 10 and Theorem 11, we obtain Corollary 11.1, stating that FOC 2 and R 2 -GNN ◦ F have the same expressiveness with respect to bounded graph classes. Corollary 11.1 does not hold for arbitrary universal graph classes, but our finding is nevertheless exciting because, in many real-world applications there are upper bounds over input graph size.

Corollary 11.1. R 2 -GNN ◦ F = FOC 2 on any bounded graph class.

To show the strict separation as in Figure 3, we can combine Proposition 3 and theorems 4 and 9 and Theorem 10 to directly get the following:

Corollary 11.2. R 2 -GNN ⊊ R 2 -GNN ◦ F on some universal graph class, and FOC 2 ⊊ R 2 -GNN ◦ F on some simple graph class.

One may think after transformation F , the logic FOC 2 ◦ F with new predicateds becomes stronger as well. However by a similar proof as for Theorem 10 and Lemma 28, we can actually show FOC 2 ◦ F ⊆ FOC 2 always holds, so F won't bring added power for FOC 2 . However, it indeed make R 2 -GNN strictly more expressive.

6 Temporal Graphs

As stated in Gao and Ribeiro [2022], a temporal knowledge graph, composed of multiple snapshots, can consistently undergo transformation into an equivalent static representation as a multi-relational graph. Consequently, this signifies that our theoretical results initially devised for multi-relational graphs can be extended to apply to temporal graphs, albeit through a certain manner of transfer.

Following previous work Jin et al. [2019], Pareja et al. [2020], Park et al. [2022], Gao and Ribeiro [2022], we define a temporal knowledge graph as a set of graph 'snapshots' distributed over a sequence of finite and discrete time points { 1 , 2 , . . . , T } . Formally, a temporal knowledge graph is a set G = { G 1 , · · · , G T } for some T ∈ N , where each G t is a static multi-relational graph. All these G t share the same node set and predicate set.

In a temporal knowledge graph, a relation or unary fact between two nodes might hold or disappear across the given timestamps. For example, a node a may be connected to a node b via a relation p in the first snapshot, but not in the second; in this case, we have ( a, p, b ) in G 1 not in G 2 . To keep track of which relations hold at which snapshots, we propose temporal predicates , an operation which we define in Definition 12.

R

2

-TGNN

time-and-graph

R

2

-GNN

◦

H

R

2

-GNN

◦

F

◦

H

R

2

-TGNN

◦

F

T

time-then-graph

⊊

⊊

⊊

⊈

Definition 12. Given a temporal graph G = { G 1 , · · · , G T } , where each G t is of the form ( V t , E t , P 1 , P 2 ) , temporal predicates are obtained from G by replacing, for each t ∈ { 1 , . . . , T } and each p ∈ P 2 , each triple ( v a , p, v b ) ∈ E t with ( v a , p t ,v b ) , where p t is a fresh predicate, unique for p and t . Similarly, each unary fact ( v a ,q ) ∈ E t , q ∈ P 1 should be replaced by ( v a ,q t ) .

Note that temporalising introduces T ×| P | new predicates in total. By temporalizing predicates , we assign a superscript to each predicate and use it to distinguish relations over different timestamps.

Definition 13. Given a temporal knowledge graph G = { G 1 , . . . , G T } , the collapse function H maps G to the static graph H ( G ) obtained by taking the union of graphs over all timestamps in the temporalization of G .

As we have proved in Section 5, for multi-relational graphs, R 2 -GNN with graph transformation is more powerful than the pure R 2 -GNN. Here, we transfer these theoretical findings in multi-relational graphs to the setting of temporal knowledge graphs. To be more specific, after temporalizing predicates , we apply a graph transformation to each graph snapshot.

Definition 14. We define F T to be the temporal graph transformation that takes any temporal knowledge graph as input, applies graph transformation to each snapshot and outputs. Specially, non-primal nodes, aux1 and aux2 edges added in any snapshot should be added into all snapshots.

R 2 -TGNN Gao and Ribeiro [2022] casts node representation in temporal graphs into two frameworks: time-and-graph and time-then-graph . Due to space constraints, we refer interested readers to Gao and Ribeiro [2022] for more details about the two frameworks. Here, we define a more general GNN-based framework abbreviated as R 2 -TGNN, where each R 2 -TGNN is a sequence {A t } T t =1 , where each A t is an R 2 -GNN model.Given a temporal knowledge graph G = { G 1 , . . . , G T } , where G t = ( V t , E t , P 1 , P 2 ) for each t ∈ { 1 , . . . , T } . The updating rule is as follows:

where I G t ( v ) is the one-hot initial feature vector of node v at timestamp t , and A t ( G t ,v, y t ) calculates the new feature vector of v by running the R 2 -GNN model A t on G t , but using y t as the initial feature vectors. As shown in Theorem 15, R 2 -TGNN composed with F T have the same expressiveness as time-then-graph 4 , while being more powerful than time-and-graph .

Theorem 15. time-and-graph ⊊ R 2 -TGNN ◦ F T = time-then-graph.

We also establish the validity of Theorem 16, which asserts that R 2 -TGNN with graph transformation maintains the same expressive power, whether it is applied directly to the temporal graph or to the equivalent collapsed static multi-relational graph

Theorem 16. R 2 -TGNN ◦ F T = R 2 -GNN ◦ F ◦ H

We also prove a strict inclusion that R 2 -TGNN ⊊ R 2 -TGNN ◦ H . Finally we get the following hierarchy of these frameworks as in Figure 6. the proof of Theorem 17 is in the appendix.

Theorem 17. The following hold:

R 2 -GNN ⊊ R 2 -GNN ◦ H ⊊ R 2 -TGNN ◦ F ◦ H = R 2 -TGNN ◦ F T = time-then-graph.

time-and-graph ⊊ R 2 -TGNN ◦ F T .

R 2 -TGNN ⊈ time-and-graph.

7 Experiment

We empirically verify our theoretical findings for multi-relational graphs by evaluating and comparing the testing performance of R 2 -GNN with graph transformation and less powerful GNNs (R-GNN and R 2 -GNN). We did two groups of experiments on synthetic datasets and real-world datasets, respectively. Details for datasets generation and statistical information as well as hyper-parameters can be found in the Appendix.

7.1 Synthetic Datasets

We first define three simple FOC 2 classifiers

Besides, we define another complicate FOC 2 classifier denoted as φ 4 shown as follows:

For each of them, we generate an independent dataset containing 7k multi-relational graphs of size up to 50-1000 nodes for training and 500 multi-relational graphs of size similar to the train set. We tried different configurations for the aggregation functions and evaluated the node classification performances of three temporal GNN methods (R-TGNNs, R 2 -TGNNs and R 2 -TGNNs ◦ F T ) on these datasets.

We verify our hypothesis empirically according to models' actual performances of fitting these three classifiers. Theoretically, φ 1 should be captured by all three models because the classification result of a node is decided by the information of its neighbor nodes, which can be accomplished by the general neighborhood based aggregation mechanism. φ 2 should not be captured by R-TGNN because the use of ¬ p 2 1 ( x,y ) as a guard means that the classification result of a node depends on the global information including those isolated nodes, which needs a global readout. For φ 3 and φ 4 , they should only be captured by R 2 -TGNNs ◦ F T . An intuitive explanation for this argument is that if we temporalise predicates and then collapse the temporal graph into its equivalent static multi-relational graph using H , we will encounter the same issue as in the Figure 1. Thus we can't distinguish expected nodes without graph transformation .

Results for temporal GNN methods and static GNN methods on four synthetic datasets can be found in Table 1. We can see that R 2 -GNN with graph transformation achieves the best performance. Our theoretical findings show that it is a more expressive model, and the experiments indeed suggest that the model can exploit this theoretical expressiveness advantage to produce better results. Besides, we can also see that R 2 -TGNN ◦ F ◦ H and R 2 -TGNN ◦ F T achieve almost the same performance, which is in line with Theorem 16.

7.2 Real-world Datasets

For real-world static multirelational graphs benchmarks, we used AIFB and MUTAG from Ristoski and Paulheim [2016]. Since open source datasets for the node classification on temporal knowledge graphs are rare, we only tried one dataset Brain-10 Gao and Ribeiro [2022] for temporal settings. 5

For static multi-relational graphs, we compare the performances of our methods with RGCN Schlichtkrull et al. [2018]. Note that RGCN assigns each node an index and the initial embedding of each node is initialised based on the node index, so the initialisation functional is not permutation-equivariant Chen et al. [2019a] and RGCN cannot be used to perform an isomorphism test. However, from Table 3, we can see that R 2 -GNN with graph transformation still achieves the highest accuracy while being able to be used for the graph isomorphism test. Besides, R 2 -GNN ◦ F also performs better compared with both R-GNN and R 2 -GNN. This again suggests that the extra expressive power gained by adding a graph transformation step to R 2 -GNN can be exploited by the model to obtain better results.

For temporal graphs, Gao and Ribeiro [2022] have classified existing temporal models into two categories, time-and-graph and time-then-graph , and shown that time-then-graph models have better performance. We choose five models mentioned in Gao and Ribeiro [2022] as our baseline and include the best accuracy of the dataset Brain-10 reported in Gao and Ribeiro [2022]. As we expected, R 2 -TGNNand R 2 -TGNN ◦ F T achieve better performance than that of the baseline models and R-TGNN accoring to Table 2. However, we observed that although in theory, R 2 -TGNN ◦ F T has stronger expressive power than R 2 -TGNN, we did not see an improvement when using R 2 -TGNN ◦ F T ( 0 . 8% accuracy drop). To some extent, it may show that some commonly used benchmarks are inadequate for testing advanced GNN variants. Similar phenomena have also been observed in previous works Chen et al. [2019b], Barceló et al. [2020].

8 Conclusion

We analyze expressivity of R 2 -GNNs with and without graph transformation in multi-relational graphs under different situations. Furthermore, we extend our theoretical findings to the temporal graph setting. Our experimental results confirm our theoretical insights, particularly demonstrating the state-of-the-art performance achieved by our graph transformation technique.

References

Namyong Park, Andrey Kan, Xin Luna Dong, Tong Zhao, and Christos Faloutsos. Estimating node importance in knowledge graphs using graph neural networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 596-606, 2019.

David Jaime Tena Cucala, Bernardo Cuenca Grau, Egor V Kostylev, and Boris Motik. Explainable gnn-based models over knowledge graphs. In International Conference on Learning Representations , 2021.

Dingmin Wang, Yeyuan Chen, and Bernardo Cuenca Grau. Efficient embeddings of logical variables for query answering over incomplete knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 4652-4659, 2023.

Zhongkai Hao, Chengqiang Lu, Zhenya Huang, Hao Wang, Zheyuan Hu, Qi Liu, Enhong Chen, and Cheekong Lee. Asgn: An active semi-supervised graph neural network for molecular property prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 731-752, 2020.

Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems , pages 6790-6802, 2021.

Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh V Chawla. Few-shot graph learning for molecular property prediction. In Proceedings of the Web Conference 2021 , pages 2559-2567, 2021.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lio, and Petar Velivckovic. Principal neighbourhood aggregation for graph nets. In NeurIPS , 2020.

Floris Geerts and Juan L Reutter. Expressiveness and approximation properties of graph neural networks. arXiv preprint arXiv:2204.04661 , 2022.

Pablo Barceló, Egor V. Kostylev, Mikael Monet, Jorge Pérez, Juan Reutter, and Juan Pablo Silva. The logical expressiveness of graph neural networks. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=r1lZ7AEKvB .

Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018.

Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference , pages 593-607. Springer, 2018.

Namyong Park, Fuchen Liu, Purvanshi Mehta, Dana Cristofor, Christos Faloutsos, and Yuxiao Dong. Evokg: Jointly modeling event time and network structure for reasoning over temporal knowledge graphs. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining , pages 794-803, 2022.

Jianfei Gao and Bruno Ribeiro. On the equivalence between temporal and static equivariant graph representations. In International Conference on Machine Learning , pages 7052-7076. PMLR, 2022.

Floris Geerts and Juan L Reutter. Expressiveness and approximation properties of graph neural networks. In International Conference on Learning Representations , 2021.

Shuwen Liu, Bernardo Grau, Ian Horrocks, and Egor Kostylev. Indigo: Gnn-based inductive knowledge graph completion using pair-wise encoding. Advances in Neural Information Processing Systems , 34:2034-2045, 2021.

Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Y. Hammerla. Relational graph attention networks, 2019.

Xingyue Huang, Miguel Romero Orth, ˙ Ismail ˙ Ilkan Ceylan, and Pablo Barceló. A theory of link prediction via relational weisfeiler-leman, 2023.

Haiquan Qiu, Yongqi Zhang, Yong Li, and Quanming Yao. Logical expressiveness of graph neural network for knowledge graph reasoning, 2023.

J.-Y. Cai, M. Furer, and N. Immerman. An optimal lower bound on the number of variables for graph identification. In 30th Annual Symposium on Foundations of Computer Science , pages 612-617, 1989. doi: 10.1109/SFCS.1989.63543.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks, 2020.

Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. Recurrent event network: Autoregressive structure inference over temporal knowledge graphs. arXiv preprint arXiv:1904.05530 , 2019.

Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao Schardl, and Charles Leiserson. Evolvegcn: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5363-5370, 2020.

Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In International conference on neural information processing , pages 362-373. Springer, 2018.

Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations , 2018.

Da Xu, Chuanwei Ruan, Evren Körpeoglu, Sushant Kumar, and Kannan Achan. Inductive representation learning on temporal graphs. In International Conference on Learning Representations , 2020.

Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637 , 2020a.

Petar Ristoski and Heiko Paulheim. Rdf2vec: Rdf graph embeddings for data mining. In International Semantic Web Conference , pages 498-514. Springer, 2016.

Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. Advances in neural information processing systems , 32, 2019a.

Ting Chen, Song Bian, and Yizhou Sun. Are powerful graph neural nets necessary? a dissection on graph classification. arXiv preprint arXiv:1905.04579 , 2019b.

Brian A Davey and Hilary A Priestley. Introduction to lattices and order . Cambridge university press, 2002.

Jia Li, Zhichao Han, Hong Cheng, Jiao Su, Pengyun Wang, Jianfeng Zhang, and Lujia Pan. Predicting path failure in time-evolving graphs, 2019.

Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks, 2016. URL https://arxiv.org/abs/ 1612.07659 .

Jinyin Chen, Xueke Wang, and Xuanheng Xu. Gc-lstm: Graph convolution embedded lstm for dynamic link prediction, 2018. URL https://arxiv.org/abs/1812.04206 .

Franco Manessi, Alessandro Rozza, and Mario Manzo. Dynamic graph convolutional networks. Pattern Recognition , 97:107000, jan 2020. doi: 10.1016/j.patcog.2019.107000. URL https: //doi.org/10.1016%2Fj.patcog.2019.107000 .

Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, and Hao Yang. Dynamic graph representation learning via self-attention networks, 2018. URL https://arxiv.org/abs/1812.09430 .

Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs, 2020b. URL https: //arxiv.org/abs/2006.10637 .

Hava T. Siegelmann and Eduardo D. Sontag. On the computational power of neural nets. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory , COLT '92, page 440-449, New York, NY, USA, 1992. Association for Computing Machinery. ISBN 089791497X. doi: 10.1145/130385.130432. URL https://doi.org/10.1145/130385.130432 .

Appendix

Table of Contents

A Preliminaries for Proofs

In this section, we give some preliminaries which will be used to prove the theorems, propositions and lemmas shown in our main body. In what follows, we fix a unary predicate set P 1 and a binary predicate set P 2 .

Definition 18. For an R 2 -GNN, we say it is a 0/1-GNN if the recursive formula used to compute vectors x ( i ) v for each node v in a multi-relational graph G = { V, E , P 1 , P 2 } on each layer i is in the following form

where C ( i ) ,A ( i ) j ,R ( i ) are all integer matrices of size d i × d i -1 , b ( i ) is bias column vector with size d i × 1 , where d i -1 and d i are input/output dimensions, and f is defined as max (0 ,min ( x, 1)) .

Furthermore, we restrict the final output dimension be d L = 1 . Since all matrices have integer elements, initial vectors are integer vectors by initialisation function I ( · ) (Section 2.2), and max (0 ,min ( x, 1)) will map all integers to 0 / 1 , it's easy to see that the output of this kind of model is always 0 / 1 , which can be directly used as the classification result. We call such model 0/1-GNN. A model instance can be represented by { C ( i ) , ( A ( i ) j ) K j =1 ,R ( i ) ,b ( i ) } L i =1 , where K = | P 2 |

Lemma 19. Regard 0/1-GNN as node classifier, then the set of node classifiers represented by 0/1-GNN is closed under ∧ , ∨ , ¬ .

Proof. Given two 0/1-GNN A 1 , A 2 , it suffices to show that we can construct ¬A 1 and A 1 ∧ A 2 in 0/1-GNN framework. That's because construction of A 1 ∨ A 2 can be reduced to constructions of ∧ , ¬ by De Morgan's law, e.g., a ∨ b = ¬ ( ¬ a ∧ ¬ b ) .

1. Construct ¬A 1 . Append a new layer to A 1 with dimension d L +1 = 1 . For matrices and bias C ( L +1) , ( A ( L +1) j ) K j =1 ,R ( L +1) ,b ( L +1) in layer L + 1 , set C L +1 1 , 1 = -1 and b L +1 1 = 1 and other parameters 0 . Then it follows x ( L +1) v = max (0 ,min ( -x ( L ) v +1 , 1)) . Since x ( L ) v is the 0/1 classification result outputted by A 1 . It's easy to see that the above equation is exactly x ( L +1) v = ¬ x ( L ) v

2. Construct A 1 ∧ A 2 . Without loss of generality, we can assume two models have same layer number L and same feature dimension d l in each layer l ∈ { 1 ......L } . Then, we can construct a new 0/1-GNN A . A has L +1 layers. For each of the first L layers, say l -th layer, it has feature dimension 2 d l . Let { C ( l ) 1 , ( A ( l ) j, 1 ) K j =1 ,R ( l ) 1 ,b ( l ) 1 } , { C ( l ) 2 , ( A ( l ) j, 2 ) K j =1 ,R ( l ) 2 ,b ( l ) 2 } be parameters in layer l of A 1 , A 2 respectively. Parameters for layer l of A are defined below

Initialization function of A is concatenation of initial feature of A 1 , A 2 . Then it's easy to see that the feature x L v after running first L layers of A is a two dimension vector, and the two dimensions contains two values representing the classification results outputted by A 1 , A 2 respectively.

For the last layer L +1 , it has only one output dimension. We just set C L +1 1 , 1 = C L +1 1 , 2 = 1 , b L +1 1 = -1 and all other parameters 0 . Then it's equivalent to x ( L +1) v = max (0 ,min ( x ( L ) v, 1 + x ( L ) v, 2 -1 , 1)) where x ( L ) v, 1 , x ( L ) v, 2 are output of A 1 , A 2 respectively. It's easy to see that the above equation is equivalent to x ( L +1) v = x ( L ) v, 1 ∧ x ( L ) v, 2 so the A constructed in this way is exactly A 1 ∧ A 2

Definition 20. A FOC 2 formula is defined inductively according to the following grammar:

Definition 21. For any subset S ⊆ P 2 , let φ S ( x,y ) denote the FOC 2 formula ( ∧ r ∈ S r ( x,y )) ∧ ( ∧ r ∈ P 2 \ S ¬ r ( x,y )) . Note that φ S ( x,y ) means there is a relation r between x and y if and only if r ∈ S , so φ S ( x,y ) can be seen as a formula to restrict specific relation distribution between two nodes. RSFOC 2 is inductively defined according to the following grammar:

Next, we prove that FOC 2 and RSFOC 2 have the same expressiveness, namely, each FOC 2 node classifier can be rewritten in the form RSFOC 2 .

Proof. Comparing the definitions of RSFOC 2 and FOC 2 , it is obvious that RSFOC 2 ⊆ FOC 2 trivially holds, so we only need to prove the other direction, namely, FOC 2 ⊆ RSFOC 2 . In particular, a Boolean logical classifier only contains one free variable, we only need to prove that for any one-free-variable FOC 2 formula φ ( x ) , we can construct an equivalent RSFOC 2 formula ψ ( x ) .

We prove Lemma 22 by induction over k , where k is the quantifier depth of φ ( x ) .

In the base case where k = 0 , φ ( x ) is just the result of applying conjunction, disjunction or negation to a bunch of unary predicates A ( x ) , where A ∈ P 1 . Given that the grammar of generating φ ( x ) is the same in RSFOC 2 and FOC 2 when k = 0 , so the lemma holds for k = 0 .

For the indutive step, we assume that Lemma 22 holds for all RSFOC 2 formula with quantifier depth no more than m , we next need to consider the case when k = m +1 .

We can decompose φ ( x ) to be boolean combination of a bunch of FOC 2 formulas φ 1 ( x ) , . . . ,φ N ( x ) , each of which is in the form φ i ( x ) := A ( x ) where A ∈ P 1 or φ i ( x ) := ∃ ≥ n y ( φ ′ ( x,y )) . See the following example for reference.

Example 23. Assume φ ( x ) := ( A 1 ( x ) ∧ ∃ y ( r 1 ( x,y )) ) ∨ ( ∃ y ( A 2 ( y ) ∧ r 2 ( x,y ) ) ∧∃ y ( r 3 ( x,y )) ) . It can be decomposed into boolean combination of four subformulas shown as follows:

φ 1 ( x ) = A 1 ( x )

φ 2 ( x ) = ∃ y ( r 1 ( x,y ))

φ 3 ( x ) = ∃ y ( A 2 ( y ) ∧ r 2 ( x,y ) )

φ 4 ( x ) = ∃ y ( r 3 ( x,y ))

Wecan see that grammars of FOC 2 and RSFOC 2 have a common part: A ( x ) , φ 1 ∧ φ 2 , φ 1 ∨ φ 2 , ¬ φ 1 , so we can only focus on those subformulas φ i ( x ) in the form of ∃ ≥ n yφ ′ ( x,y ) . In other words, if we can rewrite these FOC 2 subformulas into another form satisfying the grammar of RSFOC 2 , we can naturally construct the desired RSFOC 2 formula ψ ( x ) equivalent to FOC 2 formula φ ( x ) .

Without loss of generality, in what follows, we consider the construction for φ ( x ) = ∃ ≥ n y ( φ ′ ( x,y )) . Note that φ ( x ) has quantifier depth no more than m +1 , and φ ′ ( x,y ) has quantifier depth no more than m .

We can decompose φ ′ ( x,y ) into three sets of subformulas { φ x i ( x ) } N x i =1 , { φ y i ( y ) } N y i =1 , { r i ( x,y ) } | P 2 | i =1 , where N x and N y are two natural numbers, φ x i ,φ y i are its maximal subformulas whose free variable is assigned to x and y , respectively. φ ′ ( x ) is the combination of these sets of subformulas using ∧ , ∨ , ¬ .

Example 24. Assume that we have a FOC 2 formula in the form of φ ′ ( x,y ) = ( r 1 ( x,y ) ∧ ∃ x ( r 2 ( x,y )) ) ∨ ( ∃ y ( ∃ x ( r 3 ( x,y )) ∨ ∃ y ( r 1 ( x,y )) ) ∧∃ y ( A 2 ( y ) ∧ r 2 ( x,y ) ) )

It can be decomposed into the following subformulas:

φ x 1 ( x ) := ∃ y ( ∃ x ( r 3 ( x,y )) ∨ ∃ y ( r 1 ( x,y )) ) ;

φ x 2 ( x ) := ∃ y ( A 2 ( y ) ∧ r 2 ( x,y ) ) ;

φ y 1 ( y ) := ∃ x ( r 2 ( x,y )) ;

r 1 ( x,y )

Assume that N := { 1 , . . . , N x } , we construct a RSFOC 2 formula φ x T ( x ) := ( ∧ i ∈ T φ x i ( x )) ∧ ( ∧ i ∈ N \ T ¬ φ x i ( x )) , where T ⊆ N . It is called the x-specification formula, which means φ x T ( x ) is true iff the following condition holds: for all i ∈ T , φ x i ( x ) is true and for all i ∈ N \ T , φ x i ( x ) is false .

By decomposing φ ′ ( x,y ) into three subformula sets, we know Boolean value of φ ′ ( x,y ) can be decided by Boolean values of these formulas { φ x i ( x ) } N x i =1 , { φ y i ( y ) } N y i =1 , { r i ( x,y ) } | P 2 | i =1 . Now for any two specific subsets S ⊆ P 2 , T ⊂ N , we assume φ S ( x,y ) and φ x T ( x ) are all true (Recall the definition of φ S ( x,y ) in Definition 21). Then Boolean values for formulas in { φ x i ( x ) } N x i =1 , { r i ( x,y ) } | P 2 | i =1 are determined and Boolean value of φ ′ ( x,y ) depends only on Boolean values of { φ y i ( y ) } N y i =1 . Therefore, we can write a new FOC 2 formula φ y S,T ( y ) which is a boolean combination of { φ y i ( y ) } N y i =1 . This formula should satisfy the following condition: For any graph G and two nodes a,b on it, the following holds,

By our inductive assumption, φ ′ ( x,y ) has a quantifier depth which is no more than m , so { φ x i ( y ) } N x i =1 , { φ y i ( y ) } N y i =1 also have quantifier depths no more than m . Therefore, each of them has RSFOC 2 correspondence. Furthermore, since ∧ , ∨ , ¬ are allowed operation in RSFOC 2 , φ x T ( x ) and φ y S,T ( y ) can also be rewritten as RSFOC 2 formulas.

Given that φ S ( x,y ) and φ x T ( y ) specify the boolean values for all { φ x i ( y ) } N x i =1 , { φ r i ( x,y ) } | P 2 | i =1 formulas, so we can enumerate all possibilities over S ⊆ P 2 and T ⊆ N . Obviously for any graph G and a node pair ( a,b ) , there exists an unique ( S,T ) pair such that φ S ( a,b ) ∧ φ x T ( a ) holds.

Hence, combining Equation (9), φ ′ ( x,y ) is true only when there exists a ( S,T ) pair such that φ S ( x,y ) ∧ φ x T ( x ) ∧ φ y S,T ( y ) is true . Formally, we can rewrite φ ′ ( x,y ) as following form:

In order to simplify the formula above, let ϕ T ( x ) denote the following formula:

Then we can simplify Equation (10) to the following form:

Recall that φ ( x ) = ∃ ≥ n y ( φ ′ ( x,y )) , so it can be rewritten as:

Since for any graph G and its node a , there exists exactly one T such that φ x T ( a ) is true . Therefore, Equation (13) can be rewritten as the following formula:

Let ̂ φ T ( x ) := ∃ ≥ n y ( ϕ T ( x,y )) . Since ∧ , ∨ are both allowed in RSFOC 2 . If we want to rewrite φ ( x ) in the RSFOC 2 form, it suffices to rewrite ̂ φ T ( x ) as a RSFOC 2 formula, which is shown as follows,

Similar to the previous argument, since for any graph G and of of its node pairs ( a,b ) , the relationspecification formula φ S ( x,y ) restricts exactly which types of relations exists between ( a,b ) , there is exactly one subset S ⊆ P 2 such that φ S ( a,b ) holds.

Therefore, for all S ⊆ P 2 , we can define n S as the number of nodes y such that φ S ( x,y ) ∧ φ y S,T ( y ) holds. Since for two different subsets S 1 ,S 2 ⊆ P 2 and a fixed y , φ S 1 ( x,y ) and φ S 2 ( x,y ) can't hold simultaneously, the number of nodes y that satisfies φ S ( x,y ) ∧ φ y S,T ( y ) is exactly the sum ∑ S ⊆ P 2 n S . Therefore, in order to express Equation (15), which means there exists at least n nodes y such that ∨ S ⊆ P 2 ( φ S ( x,y ) ∧ φ y S,T ( y ) ) holds, it suffices to enumerate all possible values for { n S | S ⊆ P 2 } that satisfies ( ∑ S ⊆ P 2 n S ) = n, n S ∈ N . Formally, we can rewrite ̂ φ T ( x ) as follows:

Note that ∃ ≥ n S y ( φ S ( x,y ) ∧ φ y S,T ( y )) satisfies the grammar of RSFOC 2 , so ̂ φ T ( x ) can be rewritten as RSFOC 2 . Then, since φ x T ( x ) can also be rewritten as RSFOC 2 by induction, combining Equation (14) and Equation (15), φ ( x ) is in RSFOC 2 . We finish the proof.

B Proof of Proposition 3

Proposition 3. FOC 2 ̸⊆ R 2 -GNN and R 2 -GNN ̸⊆ FOC 2 on some universal graph class G u .

Proof. First, we prove FOC 2 ⊈ R 2 -GNN.

Consider the two graphs G 1 ,G 2 in Figure 1. ( G 1 ,a ) , ( G 2 ,a ) can be distinguished by the FOC 2 formula φ ( x ) := ∃ ≥ 1 y ( p 1 ( x,y ) ∧ p 2 ( x,y )) . However, we will prove that any R 2 -GNN can't distinguish any node in G 1 from any node in G 2 .

Let's prove it by induction over the layer number L of R 2 -GNN. That's to say, we want to show that for any L ≥ 0 , R 2 -GNN with no more than L layers can't distinguish any node of G 1 from that of G 2 .

For the base case where L = 0 , since each node feature vector is initialized by the unary predicate information, so the result trivially holds.

Assume any R 2 -GNN with no more than L = m layers can't distinguish nodes of G 1 from nodes of G 2 . Then we want to prove the result for L = m +1 .

For any R 2 -GNN model A with m +1 layers, let A ′ denote its first m layers, we know outputs of A ′ on any node from G 1 or G 2 are the same, suppose the common output feature is x ( m ) .

Recall the updating rule of R 2 -GNN in Equation (2).We know the output of A on any node v in G 1 or G 2 is defined as follows,

Here C ( m +1) ,A ( m +1) 1 ,A ( m +1) 2 ,R ( m +1) are parameters in the layer m +1 of A , u 1 ( v ) ,u 2 ( v ) is the only r 1 ,r 2 -type neighbor of v , and a,b,c,d are nodes from the corresponding graph G 1 or G 2 . From Figure 1 we can see they are well defined.

By induction, since any node pairs from G 1 and G 2 can't be distinguished by A ′ , we have x ( m ) v , x ( m ) u 1 ( v ) , x ( m ) u 2 ( v ) , x ( m ) a , x ( m ) b , x ( m ) c , x ( m ) d are all the same feature x ( m ) . Therefore, Equation (17) have the same expression for all nodes v from G 1 and G 2 , which implies any A with m +1 layers can't distinguish nodes from G 1 and G 2 .

Next, we then prove R 2 -GNNs ⊈ FOC 2 .

Assume we want to construct a classifier c which classifies a node into true iff the node has a larger number of r 1 -type neighbors than that of r 2 -type neighbors.

First, we prove that we can construct an 0/1-GNN A to capture c . It only has one layer with parameters C (1) ,A (1) 1 ,A (1) 2 ,R (1) , and feature dimension d 0 = d 1 = 1 . We assume that each node has the same initial feature vector, i.e., 1 . We set A (1) 1 , (1 , 1) = 1 ,A (1) 2 , (1 , 1) = -1 , where A (1) 1 , (1 , 1) denotes the only element in A (1) 1 placed in the first row and first column (similar for A (1) 2 , (1 , 1) ) and all other

2

3

2

n

+2

1

2

n

+3

2

n

+4

4

n

+2

2

n r

2

-type neighbors

2

n

+1

r

1

-type neighbors

G

(

n

) :

2

3

2

n

+1

1

2

n

+2

2

n

+4

4

n

+2

2

n

+1

r

2

-type neighbors

2

n r

1

-type neighbors

H

(

n

) :

parameters 0 . It's easy to see that A is equivalent to our desired classifier c on any graph since we have x (1) v = max (0 ,min (1 , ∑ u ∈N G, 1 ( v ) 1 -∑ u ∈N G, 2 ( v ) 1)) .

Next, we show FOC 2 can't capture c on G s . In order to show that, for any natural number n , we can construct two single-edge graphs G ( n ) ,H ( n ) as follows:

We prove the result by contradiction. Assume there is a FOC 2 classifier φ that captures the classifier c , then it has to classify ( G ( n ) , 1) as true and ( H ( n ) , 1) as false for all natural number n . However, in the following we will show that it's impossible, which proves the non-existence of such φ .

Suppose threshold numbers used on counting quantifiers of φ don't exceed m , then we only need to prove that φ can't distinguish ( G ( m ) , 1) , ( H ( m ) , 1) , which contradicts our assumption.

For simplicity, we use G,H to denote G ( m ) ,H ( m ) . In order to prove the above argument. First, we define a node-classification function CLS ( · ) as follows. It has G or H as subscript and a node of G or H as input.

CLS G (1) = CLS H (1) = 1 . It means the function returns 1 when the input is the center of G or H .

CLS G ( v 1 ) = CLS H ( v 2 ) = 2 , ∀ v 1 ∈ [2 , 2 m + 2] , ∀ v 2 ∈ [2 , 2 m + 1] , which means the function returns 2 when the input is a r 1 -neighbor of center .

CLS G ( v 1 ) = CLS H ( v 2 ) = 3 , ∀ v 1 ∈ [2 m +3 , 4 m +2] , ∀ v 2 ∈ [2 m +2 , 4 m +2] , which means the function returns 3 when the input is a r 2 -neighbor of center .

Claim 1 : Given any u 1 ,v 1 ∈ V ( G ) , u 2 ,v 2 ∈ V ( H ) , if ( CLS G ( u 1 ) ,CLS G ( v 1 )) = ( CLS H ( u 2 ) ,CLS H ( v 2 )) , then any FOC 2 formula with threshold numbers no larger than m can't distinguish ( u 1 ,v 1 ) and ( u 2 ,v 2 ) .

This claim is enough for our result. We will prove that for any constant d and any FOC 2 formula ϕ with threshold numbers no larger than m and quantifier depth d , ϕ can't distinguish ( u 1 ,v 1 ) and ( u 2 ,v 2 ) given that ( CLS G ( u 1 ) ,CLS G ( v 1 )) = ( CLS H ( u 2 ) ,CLS H ( v 2 ))

The result trivially holds for the base case where d = 0 . Now let's assume the result holds for d ≤ k , we can now prove the inductive case when d = k +1 .

Since ∧ , ∨ , ¬ , r ( x,y ) trivially follows, we can only consider the case when ϕ ( x,y ) is in the form ∃ ≥ N yϕ ′ ( x,y ) ,N ≤ m or ∃ ≥ N xϕ ′ ( x,y ) ,N ≤ m , where ϕ ′ ( x,y ) is a FOC 2 formula with threshold

numbers no more than m and quantifier depth no more than k . Since these two forms are symmetrical, without loss of generality, we only consider the case ∃ ≥ N yϕ ′ ( x,y ) ,N ≤ m .

Let N 1 denote the number of nodes v ′ 1 ∈ V ( G ) such that ( G,u 1 ,v ′ 1 ) | = ϕ ′ and N 2 denote the number of nodes v ′ 2 ∈ V ( H ) such that ( H,u 2 .v ′ 2 ) | = ϕ ′ . Let's compare values of N 1 and N 2 . First, By induction, since we have CLS G ( u 1 ) = CLS H ( u 2 ) from precondition, so for any v ′ 1 ∈ V ( G ) , v ′ 2 ∈ V ( H ) , which satisfies CLS G ( v ′ 1 ) = CLS H ( v ′ 2 ) , ϕ ′ ( x,y ) can't distinguish ( u 1 ,v ′ 1 ) and ( u 2 ,v ′ 2 ) . Second, isomorphism tells us ϕ ′ can't distinguish node pairs from the same graph if they share the same CLS values. Combining these two facts, there has to be a subset S ⊆ { 1 , 2 , 3 } , such that N 1 = ∑ a ∈ S N G ( a ) and N 2 = ∑ a ∈ S N H ( a ) , where N G ( a ) denotes the number of nodes u on G such that CLS G ( u ) = a , ( N H ( a ) is defined similarly).

It's easy to see that N G (1) = N H (1) = 1 , and N G ( a ) ,N H ( a ) > m for a ∈ { 2 , 3 } . Therefore, at least one of N 1 = N 2 and m < min { N 1 ,N 2 } holds. In neither case ∃ ≥ N yϕ ′ ( x,y ) ,N ≤ m can distuigush ( u 1 ,v 1 ) and ( u 2 ,v 2 ) .

Note that in the above proof our graph class { G ( n ) ,H ( n ) | n ∈ N } is actually a simple graph class, so we can actually get the following stronger argument.

Corollary 24.1. R 2 -GNN ⊈ FOC 2 on some simple graph class.

C Proof of Theorem 4

Theorem 4. FOC 2 ⊆ R 2 -GNN on any simple graph class, and FOC 2 ⊊ R 2 -GNN on some simple graph class.

Proof. We just need to show FOC 2 ⊆ R 2 -GNN on any simple graph class, and the second part can be just concluded from Corollary 24.1. By Lemma 22, FOC 2 = RSFOC 2 , so it suffices to show RSFOC 2 ⊆ 0/1-GNN. By Lemma 19, 0/1-GNN is closed under ∧ , ∨ , ¬ , so we can only focus on formulas in RSFOC 2 of form φ ( x ) = ∃ ≥ n y ( φ S ( x,y ) ∧ φ ′ ( y )) ,S ⊆ P 2 . If we can construct an equivalent 0/1-GNN A for all formulas of above form, then we can capture all formulas in RSFOC 2 since other generating rules ∧ , ∨ , ¬ is closed under 0/1-GNN. In particular, for the setting of singleedge graph class, φ is meaningful only when | S | ≤ 1 . That's because | S | > 2 implies that φ is just the trivial ⊥ in any single-edge graph class G s .

Do induction over quantifier depth k of φ ( x ) . In the base case where k = 0 , the result trivially holds since in this situation, the only possible formulas that needs to consider are unary predicates A ( x ) , where A ∈ P 1 , which can be captured by the initial one-hot feature. Next, assume our result holds for all formulas with quantifier depth k no more than m , it suffices to prove the result when quantifier depth of φ ( x ) = ∃ ≥ n y ( φ S ( x,y ) ∧ φ ′ ( y )) is m +1 . It follows that quantifier depth of φ ′ ( y ) is no more than m .

By induction, there is a 0/1-GNN model A ′ such that A ′ = φ ′ on single-edge graph class. To construct A , we only need to append another layer on A ′ . This layer L +1 has dimension 1 , whose parameters C ( L +1) , ( A ( L +1) j ) K j =1 ,R ( L +1) ,b ( L +1) are set as follows:

When | S | = 1 : Suppose S = { j } , set A L +1 j, (1 , 1) = 1 ,b L +1 = 1 -n , where A L +1 j, (1 , 1) denotes the element on the first row and first column of matrix A ( L +1) j . Other parameters in this layer are 0 . This construction represents x ( L +1) v = max (0 ,min (( ∑ u ∈N G,j ( v ) x ( L ) u ) -( n -1) , 1)) . Since x ( L ) u is classification result outputted by A ′ which is equivalent to φ ′ , ∑ u ∈N G,j ( v ) x ( L ) u counts the number of j -type neighbor u of v that satisfies φ ′ ( u ) . Therefore x ( L +1) v = 1 if and only if there exists at least n j -type neighbors satisfying the condition φ ′ , which is exactly what φ ( x ) means.

When | S | = 0 : Let K = | P 2 | , for all j ∈ [ K ] , set A L +1 j, (1 , 1) = -1 , R ( L +1) 1 , 1 = 1 ,b L +1 = 1 -n and all other parameters 0 . This construction represents x ( L +1) v = max (0 ,min (( ∑ u ∈ V ( G ) x ( L ) u ) -( ∑ K j =1 ∑ u ∈N G,j ( v ) x ( L ) u ) -( n -1) , 1)) . Since we only

consider single-edge graph, ( ∑ u ∈ V ( G ) x ( L ) u ) -( ∑ K j =1 ∑ u ∈N G,j ( v ) x ( L ) u ) exactly counts the number of nodes u that satisfies φ ′ ( y ) and doesn't have any relation with v . It's easy to see that x ( L +1) v = 1 iff there exists at least n such nodes u , which is exactly what φ ( x ) means.

Hence, we finish the proof for Theorem 4 - for each FOC 2 formula over the single-edge graph class, we can construct an R 2 -GNN to capture it.

D Proof of Theorem 5 and Theorem 6

Theorem 5. R 2 -GNN ⊆ FOC 2 on any bounded graph class, and R 2 -GNN ⊊ FOC 2 on some bounded graph class.

Theorem 6. For any bounded graph class G b . Suppose any G ∈ G b has no more than N nodes, and G b has unary predicate set P 1 and relation (binary predicate) set P 2 . Let m 1 := | P 1 | ,m 2 := | P 2 | , then for any node classifier c , suppose c can be represented as an R 2 -GNN with depth (layer number) L , then by Theorem 5 there is a FOC 2 classifier φ equivalent to c over G b . Moreover, the followings hold:

1. The quantifier depth of φ is no more than L .

2. The size of φ (quantified by the number of nodes of φ 's parse tree) is no more than 2 2 f ( L ) , where f ( L ) := 2 2 2( N +1) f ( L -1) ,f (0) = O (2 2 2( m 1 + m 2 )) ) .

For Theorem 5, we just need to show R 2 -GNN ⊆FOC 2 on any bounded graph class. The second part can then be shown by the fact that the graph class { G 1 ,G 2 } in Figure 1 is a bounded graph class but FOC 2 ⊈ R 2 -GNN still holds. In the following proof, we also show how to get the complexity upper bound claimed in Theorem 6. If we want to prove R 2 -GNN ⊆ FOC 2 , it suffices to show that for any R 2 -GNN A , there exists an equivalent FOC 2 formula φ on any bounded graph class G b . It implies that for two graphs G 1 ,G 2 and their nodes a,b , if they are classified differently by A , there exists some FOC 2 formula φ that can distinguish them. Conversly, if a,b can't be distinguished by any FOC 2 formula, then they can't be distinguished by any R 2 -GNN as well.

Definition 25. For a set of classifiers Ψ = { ψ 1 ......ψ m } , a Ψ -truth-table T is a 0/1 string of length m . T can be seen as a classifier, which classifies a node v to be true if and only if for any 1 ≤ i ≤ m , the classification result of ψ i on v equals to T i , where T i denotes the i -th bit of string T . We define T (Ψ) := { 0 , 1 } m as the set of all Ψ -truth-tables. We have that for any graph G and its node v , v satisfies exactly one truth-table T .

Proposition 26. Let FOC 2 ( n ) denote the set of formulas of FOC 2 with quantifier depth no more than n . For any bounded graph class G b and n , only finitely many intrinsically different node classifiers on G b can be represented by FOC 2 ( n ) . Furthermore, define N,m 1 ,m 2 as in Theorem 6, the number of intrinsically different FOC 2 ( n ) node classifiers on G b and their parse tree sizes are all upper bounded by f ( n ) as defined in Theorem 6.

Proof. Suppose all graphs in G b have no more than N constants, then for any natural number m>N , formulas of form ∃ ≥ m y ( φ ( x,y )) are always false. Therefore, it's sufficient only to consider FOC 2 logical classifiers with threshold numbers no more than N on G b .

There are only m 1 + m 2 predicates, and each boolean combination of unary predicates using ∧ , ∨ , ¬ can be rewritten in the form of Disjunctive Normal Form (DNF) (Davey and Priestley [2002]). So there are only at most f (0) = 2 2 2( m 1 + m 2 ) intrinsically different formulas in FOC 2 with quantifier depth 0 . Note that 2( m 1 + m 2 ) is the number of terms, 2 2( m 1 + m 2 ) is the number of different truth-table conjunctions on these terms, and 2 2 2( m 1 + m 2 ) is the number of different DNFs on these conjunctions. Each DNF has parse tree of size at most 1 + 2 2( m 1 + m 2 ) (1 + 2 m 1 + 2 m 2 ) ≤ 1000 · 2 2 2( m 1 + m 2 ) . Therefore, define f (0) = 1000 · 2 2 2( m 1 + m 2 ) = O (2 2 2( m 1 + m 2 ) ) , we know the number of different FOC 2 formulas with quantifier depth 0 and parse tree size of these formulas can both be upper bounded by f (0) .

By induction, suppose there are only f ( k ) intrinsically different FOC 2 ( k ) formulas on G b . and each meaningful FOC 2 ( k +1) formula is generated by the following grammar

where φ 1 ,φ 2 are FOC 2 ( k +1) formulas and φ ′ is FOC 2 ( k ) formulas.

Given that only the rule ∃ ≥ m y ( φ ′ ( x,y )) can increase the quantifier depth from k to k + 1 , m ≤ N , and there are only f ( k ) intrinsically different φ ′ ( x,y ) ∈ FOC 2 ( k ) on G b by induction. Therefore, there are only (2 N + 2) f ( k ) intrinsically different FOC 2 ( k + 1) formulas of form ∃ ≥ m y ( φ ′ ( x,y )) , ∃ ≥ m x ( φ ′ ( x,y )) or in FOC 2 ( k ) on G b . Moreover, their boolean combination using ∧ , ∨ , ¬ can be always rewritten in the DNF form, So there are also finitely many intrinsically different FOC 2 ( k + 1) logical classifiers on G b . Similarly, we can bound the number of different DNF by f ( k + 1) = 2 2 2( N +1) f ( k ) , where 2( N + 1) f ( k ) is the number of "building blocks" which are sub-formulas with smaller quantifier depth or outermost symbol ∃ , 2 2( N +1) f ( k ) is the number of different conjunctions on these building blocks, and f ( k + 1) = 2 2 2( N +1) f ( k ) is the number of different DNFs on these conjunctions. Parse tree size of each of these DNFs is at most 1 + 2 2( N +1) f ( k ) (1 + 2( N +1) f ( k )(1 + f ( k ))) ≤ 2 2 2( N +1) f ( k ) = f ( k +1) . The LHS is from the inductive assumption that each FOC 2 ( k ) formula has a equivalent representation within f ( k ) parse tree size. The inequality is because we know f ( k ) ≥ 1000 . Thus, we can upper bound the number of intrinsically different FOC 2 ( k +1) formulas on G b and their parse tree size both by f ( k +1) .

Lemma 27. For any two pairs ( G 1 , v 1 ) and ( G 2 , v 2 ) , where G 1 and G 2 are two bounded graphs from G b and v 1 and v 2 are two nodes in G 1 and G 2 , respectively. If all logical classifiers in FOC 2 ( L ) can't distinguish v 1 ,v 2 , then any R 2 -GNN with layer no more than L can't distinguish them as well.

Proof. By one-hot feature initialization function of R 2 -GNN, FOC 2 (0) can distinguish all different one-hot intial features, so the lemma trivially holds for the base case ( L = 0 ).

For the inductive step, we suppose Lemma 27 holds for all L ≤ k , then we can assume v 1 ,v 2 can't be distinguished by FOC 2 ( k +1) . Let N = k +1

G 1 and G 2 are bounded graphs from G b , so FOC 2 ( N ) has finitely many intrinsically different classifiers according to Proposition 26. Let T T N ( v ) denote the FOC 2 ( N ) -truth-table satisfied by v . According to Definition 25, we know that for any T ∈ T ( FOC 2 ( N )) , there exists a FOC 2 ( N ) classifier φ T such that for any node v on G i , where i ∈ 1 , 2 , T T N ( v ) = T ⇔ ( G i ,v ) | = φ T .

Assume there is an R 2 -GNN A that distinguish v 1 ,v 2 with layer L = k +1 . Let ̂ A denote its first k layers. By update rule of R 2 -GNN illustrated in Equation 2, output of A on node v of graph G , x ( k +1) v only dependent on the following three things:

output of ̂ A on v , x ( k ) v

multiset of outputs of ̂ A on r -type neighbors of v for each r ∈ P 2 , { x ( k ) u | u ∈ N G,r ( v ) }

multiset of outputs of ̂ A on all nodes in the graph, { x ( k ) u | u ∈ N G,r ( v ) }

By induction, since v 1 ,v 2 can't be distinguished by FOC 2 ( k ) , they has same feature outputted by ̂ A . Then there are two remaining possibilities.

̸

{ {T T k ( u ) | u ∈ N G 1 ,r ( v 1 ) } } = { {T T k ( u ) | u ∈ N G 2 ,r ( v 2 ) } } for some binary predicate r . Therefore, there exists a FOC 2 ( k ) -truth-table T , such that v 1 ,v 2 have differently many r -type neighbors that satisfies φ T . Without loss of generality, suppose v 1 ,v 2 have n 1 ,n 2 ( n 1 < n 2 ) such neighbors respectively. we can write a FOC 2 ( k +1) formula ∃ ≥ n 2 y ( r ( x,y ) ∧ φ T ( y )) that distinguishes v 1 and v 2 , which contradicts the precondition that they can't be distinguished by FOC 2 ( k +1) classifiers.

̸

{ {T T k ( u ) | u ∈ V ( G 1 ) } } = { {T T k ( u ) | u ∈ V ( G 2 ) } } . Therefore, there exists a FOC 2 ( k ) -truth-table T , such that G 1 ,G 2 have differently many nodes that satisfies φ T . Without loss of generality, suppose G 1 ,G 2 have n 1 ,n 2 ( n 1 < n 2 ) such nodes respectively. we can write

a FOC 2 ( k +1) formula ∃ ≥ n 2 yφ T ( y ) that distinguishes v 1 and v 2 , which contradicts the precondition that they can't be distinguished by FOC 2 ( k +1) classifiers.

Since all possibilities contradicts the precondition that v 1 ,v 2 can't be distinguished by FOC 2 ( k +1) , such an A that distinguishes v 1 ,v 2 doesn't exist.

We can now gather all of these to prove Theorem 5 and Theorem 6.

Proof. For any R 2 -GNN A , suppose it has L layers. For any graph G ∈ G b and its node v , let T T L ( v ) denote the FOC 2 ( L ) -truth-table satisfied by v . For any T ∈ T ( FOC 2 ( L )) , since G b is a bounded graph class, using Proposition 26, there exists a FOC 2 ( L ) classifier φ T such that for any node v in graph G ∈ G b , T T L ( v ) = T ⇔ ( G,v ) | = φ T . Moreover, by Proposition 26, since T is a truth table on at most f ( L ) formulas, φ T can be written as a conjunction over f ( L ) literals, which means φ T has parse tree size at most 1 + f ( L ) 2 since by Proposition 26, every formula in FOC 2 ( L ) is equivalent to some FOC 2 formula with parsee tree size at most f ( L ) .

By Lemma 27, If two nodes v 1 ,v 2 have same FOC 2 ( L ) -truth-table ( T T L ( v 1 ) = T T L ( v 2 ) ), they can't be distinguished by A . Let S denote the subset of T ( FOC 2 ( L )) that satisfies A . By Proposition 26 and Definition 25, Φ := { φ T | T ∈ S } is a finite set with | Φ | ≤ 2 f ( L ) , then disjunction of formulas in Φ , ( ∨ T ∈ S φ T ) is a FOC 2 classifier that equals to A under bounded graph class G b . Furthermore, by the above upper bound of parse tree size of any φ T , ( ∨ T ∈ S φ T ) has parse tree size no more than 1 + 2 f ( L ) (1 + f ( L ) 2 ) ≤ 2 2 f ( L ) , where the inequality is from f ( L ) ≥ 1000 . We complete the proof.

E Proof of Theorem 9

Theorem 9. R 2 -GNN ⊆ R 2 -GNN ◦ F on any universal graph class G u .

Proof. Assume that we have a predicate set P = P 1 ∪ P 2 , K = | P 2 | and let P ′ = P ∪ { primal,aux 1 ,aux 2 } denote the predicate set after transformation F . For any R 2 -GNN A under P , we want to construct another R 2 -GNN A ′ under P ′ , such that for any graph G under P and its node v , v has the same feature outputted by A ( G,v ) and A ′ ( F ( G ) ,v ) . Let L denote the layer number of A .

We prove this theorem by induction over the number of layers L . In the base ( L = 0 ), our result trivially holds since the one-hot initialization over P ′ contains all unary predicate information in P . Now suppose the result holds for L ≤ k , so it suffices to prove it when L = k +1 .

For the transformed graph F ( G ) , primal (v) is true if and only if v is the node in the original graph G . Without loss of generality, if we use one-hot feature initialization on P ′ , we can always keep an additional dimension in the node feature vector x v to show whether primal (v) is true , its value is always 0 / 1 , in the proof below when we use x to denote the feature vectors, we omit this special dimension for simplicity. But keep in mind that this dimension always keeps so we can distinguish original nodes and added nodes.

Recall that an R 2 -GNN is defined by { C ( i ) , ( A ( i ) j ) K i =1 ,R ( i ) } L i =1 . By induction, let ̂ A denote the first k layers of A , and let ̂ A ′ denote the R 2 -GNN equivalent with ̂ A on F transformation such that ̂ A = ̂ A ′ ◦ F . We will append three layers to ̂ A ′ to construct A ′ that is equivalent to A . Without loss of generality, we can assume all layers in A have same dimension length d . Suppose L ′ is the layer number of ̂ A ′ , so we will append layer L ′ +1 ,L ′ +2 ,L ′ +3 . for all l ∈ { L ′ +1 ,L ′ +2 ,L ′ +3 } , let { C a, ( l ) ,C p, ( l ) , ( A ∗ , ( l ) j ) K j =1 ,A ∗ , ( l ) aux 1 ,A ∗ , ( l ) aux 2 ,R ∗ , ( l ) } denote the parameters in l -th layer of A . Here,

A ∗ , ( l ) aux 1 ,A ∗ , ( l ) aux 2 denotes the aggregation function corresponding to two new predicates aux1 , aux2 , added in transformation F , and C p, ( l ) ,C a, ( l ) are different combination function that used for primal nodes and non-primal nodes. Note that with the help of the special dimension mentioned above, we can distinguish primal nodes and non-primal nodes. Therefore, It's safe to use different combination functions for these two kinds of nodes. Note that here since we add two predicates aux1 , aux2 , the input for combination function should be in the form C p ( x 0 , ( x j ) K j =1 , x aux 1 , x aux 2 , x g ) where x 0 is the feature vector of the former layer, and x j , 1 ≤ j ≤ K denote the output of aggregation function

A ∗ , ( l ) j , x aux 1 , x aux 2 denote the output of aggregation function A ∗ , ( l ) aux 1 ,A ∗ , ( l ) aux 2 , and x g denotes the feature outputted by global readout function R ∗ , ( l ) . For aggregation function and global readout function, their inputs are denoted by X , meaning a multiset of feature vector. Note that all aggregation functions and readout functions won't change the feature dimension, only combination functions C p, ( l ) ,C a, ( l ) will transform d l -1 dimension features to d l dimension features.

1). layer L ′ +1 : input dimension is d , output dimension is d ′ = Kd . For feature vector x with length d ′ , let x ( i ) , i ∈ { 1 , . . . , K } denote its i -th slice in dimension [( i -1) d + 1 ,id ] . Let [ x 1 , . . . , x m ] denote concatenation of x 1 , . . . , x m , and let [ x ] n denote concatenation of n copies of x , 0 n denote zero vectors of length n . parameters for this layer are defined below:

Other parameters in this layer are set to functions that always output zero-vector.

We can see here that the layer L ′ +1 do the following thing:

For all primal nodes a and its non-primal neighbor e ab , pass concatenation of K copies of x a to x e ab , and remains the feature of primal nodes unchanged.

2). layer L ′ +2 , also has dimension d ′ = Kd , has following parameters.

All other parameters in this layer are set to function that always outputs zero vectors. This layer do the following thing:

For all primal nodes, keep the feature unchanged, for all added node pair e ab ,e ba . Switch their feature, but for all r i ∈ P 2 , if there is no r i relation between a,b , the i -th slice of x e ab and x e ba will be set to 0 .

3). layer L ′ +3 , has dimension d , and following parameters.

Note that C ( L ) ,A ( L ) j ,R ( L ) are all parameters in the last layer of A mentioned previously. All other parameters in this layer are set to functions that always output zero vectors. We can see that this layer simulates the work of last layer of A as follows:

For all 1 ≤ j ≤ K , use the j -th slice of feature vector x ( j ) to simulate A ( L ) j and store results of aggregation function A ( L ) j on this slice.

Global readout trivially emulates what R ( L ) does, but only reads features for primal nodes. It can be done since we always have a special dimension in feature to say whether it's a primal node.

We just simulate what C ( L ) does on primal nodes. For 1 ≤ j ≤ K The type r j aggregation result (output of A ( L ) j ) used for input of C ( L ) is exactly j -th slice of return value of A ∗ , ( L ′ +3) aux 1 .

By construction above, A ′ is a desired model that have the same output as A .

F Proof of Theorem 10

Theorem 10. FOC 2 ⊆ R 2 -GNN ◦ F on any universal graph class G u .

Proof. For any FOC 2 classifier φ under predicate set P , we want to construct a 0/1-GNN A on P ′ = P ∪ { primal,aux 1 ,aux 2 } equivalent to φ with graph transformation F .

Recall that FOC 2 = RSFOC 2 shown in Lemma 22 and 0/1-GNNs ⊆ R 2 -GNNs, it suffices to prove that 0/1-GNN ◦ F capture RSFOC 2 . By Lemma 19, since ∧ , ∨ , ¬ are closed under 0/1-GNN it suffices to show that when φ is in the form ∃ ≥ n ( φ S ( x,y ) ∧ φ ′ ( y ) ) ,S ⊆ P 2 , we can capture it.

We prove by induction over quantifier depth m of φ . Since 0 -depth formulas are only about unary predicate that can be extracted from one-hot initial feature, our theorem trivially holds for m = 0 . Now, we assume it also holds for m ≤ k , it suffices to prove the case when m = k +1 . Then there are two possibilities:

̸

1. When S = ∅ :

Consider the following logical classifier under P ′ :

̂ φ S ( x ) restricts that for any r ∈ P ′ , x has r -type neighbor if and only if r ∈ S . Review the definition of transformation F , we know that for any added node e ab , ( F ( G ) ,e ab ) | = ̂ φ S if and only if ( G,a,b ) | = φ S ( a,b ) , where φ S ( x,y ) is the relation-specification formula defined in Definition 21 That is to say for any r i , 1 ≤ i ≤ K , there is relation r i between a,b if and only if i ∈ S .

Now consider the following formula:

For any graph G and its node v , it's easy to see that ( G,v ) | = φ ⇔ ( F ( G ) ,v ) | = ̂ φ . Therefore we only need to capture ̂ φ by 0/1-GNN on every primal node of transformed graphs. By induction, since quantifier depth of φ ′ ( y ) is no more than k , we know φ ′ ( y ) is in 0/1-GNN. ̂ φ is generated from φ ′ ( y ) using rules ∧ and ∃ y ( r ( x,y ) ∧ φ ′ ( y ) ) . By Lemma 19, ∧ is closed under 0/1-GNN. For ∃ y ( r ( x,y ) ∧ φ ′ ( y ) ) , we find that the construction needed is the same as construction for singleelement S on single-edge graph class G s used in Theorem 4. Therefore, since we can manage these two rules, we can also finish the construction for ̂ φ , which is equivalent to φ on primal nodes of transformed graph.

2. When S = ∅

First, consider the following two logical classifiers:

¯ φ says a node is primal, and satisfies φ ′ ( x ) . Since φ ′ ( x ) has quantifier depth no more than k , and ∧ is closed under 0/1-GNN. There is a 0/1-GNN A 1 equivalent to ¯ φ on transformed graph. Then, consider the following formula.

˜ φ ( x ) evaluates on added nodes e ab on transformed graph, e ab satisfies it iff b satisfies φ ′

Now for a graph G and its node v , define n 1 as the number of nodes on F ( G ) that satisfies ¯ φ , and define n 2 as the number of aux1 -type neighbors of v on F ( G ) that satisfies ˜ φ . Since φ ( x ) = ∃ ≥ n y ( φ ∅ ( x,y ) ∧ φ ′ ( y )) It's easy to see that ( G,v ) | = φ if and only if n 1 -n 2 ≥ n .

Formally speaking, for a node set S , let | S | denote number of nodes in S , we define the following classifier c such that for any graph G and its node a , c ( F ( G ) ,a ) = 1 ⇔ ( G,a ) | = φ

So how to construct a model A to capture classifier c ? First, by induction ¯ φ, ˜ φ are all formulas with quantifier depth no more than k so by previous argument there are 0/1-GNN models ¯ A , ˜ A that capture them respectively. Then we can use feature concatenation technic introduced in Equation (6) to construct a model ̂ A based on ¯ A , ˜ A , such that ̂ A has two-dimensional output, whose first and second dimensions have the same output as ¯ A , ˜ A respectively.

Then, suppose ̂ A has L layers, The only thing we need to do is to append a new layer L +1 to ̂ A , it has output dimension 1 . parameters of it are { C ( L +1) , ( A ( L +1) j ) K j =1 ,A ( L +1) aux 1 ,A ( L +1) aux 2 ,R ( L +1) } as defined in Equation (5). The parameter settings are as follows:

denotes the value in the first row and second column of A

In this construction, we have x ( L +1) v = max (0 ,min (1 , ∑ u ∈ V ( F ( G )) x ( L ) u, 1 -∑ u ∈N F ( G ) ,aux 1 ( v ) x ( L ) u, 2 -( n -1))) , which has exactly the same output as classifier c defined above in Equation (32). Therefore, A is a desired model.

G Proof of Theorem 11

Theorem 11. R 2 -GNN ◦ F ⊆ FOC 2 on any bounded graph class G b .

Before we go into theorem itself, we first introduce Lemma 28 that will be used in following proof.

Lemma 28. Let φ ( x,y ) denote a FOC 2 formula with two free variables, for any natural number n , the following sentence can be captured by FOC 2 :

There exists no less than n ordered node pairs ( a,b ) such that ( G,a,b ) | = φ .

Let c denote the graph classifier such that c ( G ) = 1 iff G satisfies the sentence above.

Proof. The basic intuition is to define m i , 1 ≤ i < n as the number of nodes a , such that there are exactly i nodes b that φ ( a,b ) is true . Specially, we define m n as the number of nodes a , such that there are at least n nodes b that φ ( a,b ) is true . Since ∑ n i =1 im i exactly counts the number of valid ordered pairs when m n = 0 , and it guarantees the existence of at least n valid ordered pairs when m n > 0 . It's not hard to see that for any graph G , c ( G ) = 1 ⇔ ∑ n i =1 im i ≥ n . Futhermore, fix a valid sequence ( m 1 ......m n ) such that ∑ n i =1 im i ≥ n , there has to be another sequence ( k 1 ......k n ) such that n ≤ ∑ n i =1 ik i ≤ 2 n and k i ≤ m i for all 1 ≤ i ≤ n . Therefore, We can enumerate all possibilities of valid ( k 1 ......k n ) , and for each valid ( k 1 ......k n ) sequence, we judge whether there are at least k i such nodes a for every 1 ≤ i ≤ n .

Formally, φ i ( x ) := ∃ [ i ] yφ ( x,y ) can judge whether a node a has exactly i partners b such that φ ( a,b ) = 1 , where ∃ [ i ] yφ ( x,y ) denotes "there are exactly i such nodes y " which is the abbreviation of formula ( ∃ ≥ i yφ ( x,y )) ∧ ( ¬∃ ≥ i +1 yφ ( x,y )) . The FOC 2 formula equivalent to our desired sentence c is as follows:

This FOC 2 formula is equivalent to our desired classifier c .

With the Lemma 28, we now start to prove Theorem 11.

Proof. By Theorem 5, it follows that R 2 -GNNs ◦ F ⊆ FOC 2 ◦ F . Therefore it suffices to show FOC 2 ◦ F ⊆ FOC 2 .

2) ( L +1) aux 1 .

By Lemma 22, it suffices to show RSFOC 2 ◦ F ⊆ FOC 2 . Since ∧ , ∨ , ¬ are common rules. We only need to show for any RSFOC 2 formula of form φ ( x ) := ∃ ≥ n y ( φ S ( x,y ) ∧ φ ′ ( y )) under transformed predicate set P ′ = P ∪ { aux 1 ,aux 2 ,primal } , there exists an FOC 2 formula φ 1 such that for any graph G under P and its node v , ( G,v ) | = φ 1 ⇔ ( F ( G ) ,v ) | = φ .

In order to show this, we consider a stronger result:

For any such formula φ , including the existence of valid φ 1 , we claim there also exists an FOC 2 formula φ 2 with two free variables such that the following holds: for any graph G under P and its added node e ab on F ( G ) , ( G,a,b ) | = φ 2 ⇔ ( F ( G ) ,e ab ) | = φ . Call φ 1 ,φ 2 as first/second discriminant of φ .

Now we need to prove the existence of φ 1 and φ 2 .

Weprove by induction over quantifier depth m of φ , Since we only add a single unary predicate primal in P ′ , any φ ( x ) with quantifier depth 0 can be rewritten as ( primal ( x ) ∧ φ 1 ( x )) ∨ ( ¬ primal ( x ) ∧ φ 2 ( x )) , where φ 1 ( x ) ,φ 2 ( x ) are two formulas that only contain predicates in P . Therefore, φ 1 can be naturally seen as the first discriminant of φ . Moreover, since φ 2 ( x ) always evaluates on non-primal nodes, it is equivalent to ⊥ or ⊤ under ¬ primal ( x ) constraint. Therefore, the corresponding ⊥ or ⊤ can be seen as the second discriminant, so our theorem trivially holds for m = 0 . Now assume it holds for m ≤ k , we can assume quantifier depth of φ = ∃ ≥ n y ( φ S ( x,y ) ∧ φ ′ ( y )) is m = k +1 .

Consider the construction rules of transformation F , for any two primal nodes in F ( G ) , there is no relation between them, for a primal node a and an added node e ab , there is exactly a single relation of type aux1 between them. For a pair of added nodes e ab ,e ba , there are a bunch of relations from the original graph G and an additional aux2 relation between them. Therefore, it suffices to only consider three possible kinds of S ⊆ P 2 ∪ { aux 1 ,aux 2 } according to three cases mentiond above. Then, we will construct first/second determinants for each of these three cases. Since φ ′ ( y ) has quantifier depth no more than k , by induction let ̂ φ 1 , ̂ φ 2 be first/second discriminants of φ ′ by induction.

1. S = { aux1 } :

for primal node a , φ ( a ) means the following: there exists at least n nodes b , such that there is some relation between a,b on G and the added node e ab on F ( G ) satisfies φ ′ . Therefore, the first determinant of φ can be defined as following:

for added nodes e ab on F ( G ) , φ ( e ab ) means a satisfies φ ′ , so the second determinant of φ is the following:

̸

2. S = { aux2 } ∪ T,T ⊆ P 2 ,T = ∅

primal nodes don't have aux2 neighbors, so first determinant is trivially false .

For added node e ab , e ab satisfies φ iff there are exactly relations between a,b of types in T , and e ba satisfies φ ′ . Therefore the second determinant is as follows, where φ T ( x,y ) is the relationspecification formula under P introduced in Definition 21

3. S = ∅

For a subset S ⊆ P 2 ∪ { aux 1 ,aux 2 } , let φ S ( x,y ) denote the relation-specification formula under P 2 ∪ { aux 1 ,aux 2 } defined in Definition 21.

Since we consider on bounded graph class G b , node number is bounded by a natural number N . For any node a on F ( G ) , let m denote the number of nodes b on F ( G ) such that φ ′ ( b ) = 1 , let m 0 denote the number of nodes b on F ( G ) such that φ ′ ( b ) = 1 and there is a single relation aux1 , between ( a,b ) on F ( G ) , (That is equivalent to φ { aux 1 } ( a,b ) = 1 ). For any T ⊆ P 2 , let m T denote the number of nodes b on F ( G ) such that φ ′ ( b ) = 1 and a,b has exactly relations of types in T ∪{ aux 2 } on F ( G ) , (That is equivalent to φ T ∪{ aux 2 } ( a,b ) = 1 ).

Note that the number of nodes b on F ( G ) such that a,b don't have any relation, (That is equivalent to φ ∅ ( a,b ) = 1 ) and φ ′ ( b ) = 1 equals to m -m 0 -∑ T ⊆ P 2 m T . Therefore, for any transformed graph F ( G ) and its node v , ( F ( G ) ,v ) | = φ ⇔ m -m 0 -∑ T ⊆ P 2 m T ≥ n . Since | V ( G ) | ≤ N for all G in bounded graph class G b , transformed graph F ( G ) has node number no more than N 2 . Therefore, we can enumerate all possibilities of m,m 0 ,m T ≤ N 2 ,T ⊂ P 2 such that the above inequality holds, and for each possibility, we judge whehter there exists exactly such number of nodes for each corresponding parameter. Formally speaking, φ can be rewritten as the following form:

where ∃ [ m ] y denotes there are exactly m nodes y .

Since first/second determinant can be constructed trivially under combination of ∧ , ∨ , ¬ , and we've shown how to construct determinants for formulas of form ∃ ≥ n y ( φ S ( x,y ) ∧ φ ′ ( y )) when S = { aux 1 } and S = { aux 2 } ∪ T,T ⊆ P 2 in the previous two cases. Therefore, in Equation (38) and Equation (39), the only left part is the formula of form ∃ [ m ] yφ ′ ( y ) . The only remaining work is to show how to construct first/second determinants for formula in form φ ( x ) := ∃ ≥ n yφ ′ ( y ) .

Let m 1 denote the number of primal nodes y that satisfies φ ′ ( y ) and let m 2 denote the number of non-primal nodes y that satisfies φ ′ ( y ) . It's not hard to see that for any node v on F ( G ) , ( F ( G ) ,v ) | = φ ⇔ m 1 + m 2 ≥ n . Therefore, φ ( x ) = ∃ ≥ n yφ ′ ( y ) that evaluates on F ( G ) is equivalent to the following sentence that evaluates on G : 'There exists two natural numbers m 1 ,m 2 such that the following conditions hold: 1. m 1 + m 2 = n . 2. There are at least m 1 nodes b on G that satisfies ̂ φ 1 , (equivalent to ( F ( G ) ,b ) | = φ ′ ). 3. There are at least m 2 ordered node pairs a,b on G such that a,b has some relation and ( G,a,b ) | = ̂ φ 2 , (equivalent to ( F ( G ) ,e ab ) | = φ ′ )."

Formally speaking, rewrite the sentence above as formula under P , we get the following construction for first/second determinants of φ .

where φ m 2 is the FOC 2 formula that expresses 'There exists at least m 2 ordered node pairs ( a,b ) such that ( G,a,b ) | = ̂ φ 2 ( x,y ) ∧ ( ∨ r ∈ P 2 r ( x,y )) " . We've shown the existence of φ m 2 in Lemma 28

H Proof of Theorem 15

Theorem 15. time-and-graph ⊊ R 2 -TGNN ◦ F T = time-then-graph.

For a graph G with n nodes, let H V ∈ R n × d v denote node feature matrix, and H E ∈ R n × n × d e denote edge feature matrix, where H E ij denote the edge feature vector from i to j .

First we need to define the GNN used in their frameworks. Note that for the comparison fairness, we add the the global readout to the node feature update as we do in R 2 -GNNs. It recursively calculates the feature vector H V, ( l ) i of the node i at each layer 1 ≤ l ≤ L as follows:

where N ( i ) denotes the set of all nodes that adjacent to i , and u ( l ) ,g ( l ) ,r ( l ) are learnable functions. Note that here the GNN framework is a little different from the general definition defined in Equation (2). However, this framework is hard to fully implement and many previous works implementing time-and-graph or time-then-graph Gao and Ribeiro [2022] (Li et al. [2019], Seo et al. [2016], Chen et al. [2018], Manessi et al. [2020], Sankar et al. [2018],Rossi et al. [2020b]) don't reach the expressiveness of Equation (41). This definition is more for the theoretical analysis. In contrast, our

definition for GNN in Equation (1) and Equation (2) is more practical since it is fully captured by a bunch of commonly used models such as Schlichtkrull et al. [2018]. For notation simplicity, for a GNN A , let H V, ( L ) = A ( H V , H E ) denote the node feature outputted by A using H V , H E as initial features.

Proposition 29. (Gao and Ribeiro [2022]):time-and-graph ⊊ time-then-grahp

The above proposition is from Theorem 1 of Gao and Ribeiro [2022]. Therefore, in order to complete the proof of Theorem 15, we only need to prove R 2 -TGNN ◦ F T = time-then-graph .

Let G = { G 1 , . . . , G T } denote a temporal knowledge graph, and A t ∈ R n ×| P 1 | , E t ∈ R n × n ×| P 2 | , 1 ≤ t ≤ T denonte one-hot encoding feature of unary facts and binary facts on timestamp t , where P 1 ,P 2 are unary and binary predicate sets.

The updating rule of a time-then-graph model can be generalized as follows:

where A is a GNN defined above, RNN is an arbitrary Recurrent Neural Network. X ∈ R n × d is the final node feature output of time-then-graph .

First we need to prove time-then-graph ⊆ R 2 -TGNN ◦ F T . That is, for any time-then-graph model, we want to construct an equivalent R 2 -TGNN A ′ to capture it on transformed graph. We can use nodes added after transformation to store the edge feature H E , and use primal nodes to store the node feature H V . By simulating RNN through choosing specific functions in R 2 -TGNN, we can easily construct a R 2 -TGNN A ′ such that for any node i , and any node pair i,j with at least one edge in history, x i = H V i and x e ij = H E i,j hold, where x i and x e ij are features of corresponding primal node i and added node e ij outputted by A ′ .

Note that A ′ is a R 2 -TGNN, it can be represented as A ′ 1 ...... A ′ T , where each A ′ t , 1 ≤ t ≤ T is a R 2 -GNN. A ′ has simulated work of RNN , so the remaining work is to simulate A ( H V , H E ) . We do the simulation over induction on layer number L of A .

When L = 0 , output of A is exactly H V , which has been simulated by A ′ above.

Suppose L = k +1 , let ˜ A denote R 2 -GNN extracted from A but without the last layer k +1 . By induction, we can construct a R 2 -TGNN ˜ A ′ that simulates ˜ A ( H V , H E ) . Then we need to append three layers to ˜ A ′ to simulate the last layer of A .

Let u ( L ) ,g ( L ) ,r ( L ) denote parameters of the last layer of A . Using notations in Equation (2), let { C ( l ) , ( A ( l ) j ) | P 2 | j =1 ,A ( l ) aux 1 ,A ( l ) aux2 ,R ( l ) } 3 l =1 denote parameters of the three layers appended to ˜ A ′ T . They are defined as follows:

First, we can choose specific function in the first two added layers, such that the following holds:

For any added node e ij , feature outputted by the new model is x (2) e ij = [ H E ij , x ′ i , x ′ j ] , where x (2) denotes the feature outputted by the second added layer, and x ′ i , x ′ j are node features of i,j outputted by ˜ A ′ . For a feature x of added node of this form, we define x 0 , x 1 , x 2 as corresponding feature slices where H E ij , x ′ i , x ′ j have been stored.

For any primal node, its feature x only stores x ′ i in x 1 , and x 0 , x 2 are all slices of dummy bits.

Let X be a multiset of features that represents function input. For the last added layer, we can choose specific functions as follows:

where x aux 1 , x g are outputs of R (3) and A (3) aux 1 , and all useless inputs of C (3) are omitted. Comparing this construction with Equation (41). It's east to see that after the last layer appended, we can construct an equivalent R 2 -TGNN A ′ that captures A on transformed graph. By inductive argument, we prove time-then-graph ⊆ R 2 -TGNN ◦ F T .

Then we need to show R 2 -TGNN ◦ F T ⊆ time-then-graph .

In Theorem 16, we will prove R 2 -TGNN ◦ F T = R 2 -GNN ◦ F ◦ H . Its proof doesn't dependent on Theorem 15, so let's assume it's true for now. Then, instead of proving R 2 -TGNN ◦ F T , it's sufficient to show R 2 -GNN ◦ F ◦ H ⊆ time-then-graph .

Let P T 1 , P T 2 denote the set of temporalized unary and binary predicate sets defined in Definition 12. Based on most expressive ability of Recurrent Neural Networks shown in Siegelmann and Sontag [1992], we can get a most expressive representation for unary and binary fact sequences through RNN . A most expressive RNN representation function is always injective, thus there exists a decoder function translating most-expressive representations back to raw sequences. Therefore, we are able to find an appropriate RNN such that its output features H V , H E in Equation (42), Equation (43) contain all information needed to reconstruct all temporalized unary and binary facts related to the corresponding nodes.

For any R 2 -GNN A on transformed collpsed temporal knowledge graph, we want to construct an equivalent time-then-graph model { RNN , A ′ } to capture A . In order to show the existence of the time-then-graph model, we will do an inductive construction over layer number L of A . Here in order to build inductive argument, we will consider a following stronger result and aim to prove it: In additional to the existence of A ′ , we claim there also exists a function f A with the following property: For any two nodes a,b with at least one edge, f A ( x ′ a , x ′ b , H E ab ) = x e ab , where x ′ a , x ′ b , H E ab are features of a , b and edge information between a,b outputted by A ′ , and x e ab is the feature of added node e ab outputted by A◦ F ◦ H . It suffices to show that there exists such function f A as well as a time-then-graph model { RNN , A ′ } such that the following conditions hold:

For any graph G and its node a,b ∈ V ( G ) ,

2.If there is at least one edge between a,b in history, f A ( H V, ( l ) a , H V, ( l ) b , H E ab ) = x e ab . Otherwise, f A ( H V, ( l ) a , H V, ( l ) b , H E ab ) = 0

where H V, ( l ) a , H V, ( l ) b are node features outputted by A ′ , while x a , x e ab are node features outputted by A on transformed collpased graph. Enc ( X ) is some injective encoding that stores all information of multiset X . For a node feature H V, ( l ) a of above form, let H V, ( l ) a, 0 := x a , H V, ( l ) a, 1 = Enc ( { { x e aj | j ∈ N ( a ) } } ) denote two slices that store independent information in different positions.

For the base case L = 0 . the node feature only depends on temporalized unary facts related to the corresponding node. Since by RNN we can use most expressiveness representation to capture all unary facts. A specific RNN already captures A when L = 0 . Moreover, there is no added node e ab that relates to any unary fact, so a constant function already satisfies the condition of f A when L = 0 . Therefore, our result holds for L = 0

Assume L = k +1 , let ̂ A denote the model generated by the first k layers of A . By induction, there is time-then-graph model ̂ A ′ and function f ̂ A ′ that captures output of ̂ A ′ on transformed collapsed graph. We can append a layer to ̂ A ′ to build A ′ that simulates A . Let { C ( L ) , ( A ( L ) j ) T | P 2 | j =1 ,A ( L ) aux 1 ,A ( L ) aux2 ,R ( L ) } denote the building blocks of layer L of A , and let u ∗ ,g ∗ ,r ∗ denote functions used in the layer that will be appended to ̂ A ′ . They are defined below:

where x g , x r are outputs of g ∗ and r ∗ . Dec ( X ) is a decoder function that do inverse mapping of Enc ( X ) mentioned above, so Dec ( H V, ( l -1) j, 1 ) is actually { { x e aj | j ∈ N ( a ) } } . Note that primal nodes

in transformed graph only has type aux1 - neighbors, so two inputs x g , x r , one for aux1 aggregation output and one for global readout are already enough for computing the value. Comparing the three rules above with Equation (2), we can see that our new model A ′ perfectly captures A .

We've captured A , and the remaining work is to construct f A defined above to complete inductive assumption. We can just choose a function that simulates message passing between pairs of added nodes e ab and e ba as well as message passing between e ab and a , and that function satisfies the condition for f A . Formally speaking, f A can be defined below:

Let's explain this equation, Sim A L ( a,g,s,b,e ) is a local simulation function which simulates singleiteration message passing in the following scenario:

Suppose there is a graph H with three constants V ( H ) = { a,e ab ,e ba } . There is an aux1 edge between a and e ab , an aux2 edge between e ab and e ba , and additional edges of different types between e ab and e ba . The description of additional edges can be founded in e . Initial node features of a,e ab ,e ba are set to a,s,b respectively. and the global readout output is g . Finally, run L -th layer of A on H , and Sim A L is node feature of e ab outputted by A L .

Note that if we use appropriate injective encoding or just use concatenation technic, H ( l -1) g , H V, ( l -1) i , H V, ( l -1) j can be accessed from H V, ( l ) i , H V, ( l ) i . Therefore the above definition for f A

is well-defined. Moreover, in the above explanation we can see that f A ( H V, ( l -1) i , H V, ( l -1) j , H E ij ) is exactly node feature of e ij outputted by A on the transformed collapsed graph, so our proof finishes.

I Proof of Theorem 16

Theorem 16. R 2 -TGNN ◦ F T = R 2 -TGNN ◦ F ◦ H .

First, we recall the definition for R 2 -TGNN as in Equation (53):

We say a R 2 -TGNN is homogeneous if A 1 , . . . , A T share the same parameters. In particular, we first prove Lemma 30, namely, homogeneous R 2 -TGNN and R 2 -TGNN (where paramters in A 1 , . . . , A T may differ) have the same expressiveness.

Lemma 30. homogenous R 2 -TGNN = R 2 -TGNN

Proof. The forward direction homogeneous R 2 -TGNN ⊆ R 2 -TGNN trivially holds. It suffices to prove the backward direction.

Let A : {A t } T t =1 denote a R 2 -TGNN. Without loss of generality, we can assume all models in each timestamps have the same layer number L . Then for each 1 ≤ t ≤ T , we can assume all A t can be represented by { C ( l ) t , ( A ( l ) t,j ) | P 2 | j =1 ,R ( l ) t } L l =1 . Futhormore, without loss of generality, we can assume all output dimensions for A ( l ) t,j ,R ( l ) t and C ( l ) t are d . As for input dimension, all of these functions also have input dimension d for 2 ≤ l ≤ L . Specially, by updating rules of R 2 -TGNN Equation (53), in the initialization stage of each timestamp we have to concat a feature with length | P 1 | to output of the former timestamp, so the input dimension for A (1) t,j ,R (1) t ,C (1) t is d + | P 1 | .

We can construct an equivalent homogeneous R 2 -TGNN with L layers represented by { C ∗ , ( l ) , ( A ∗ , ( l ) j ) | P 2 | j =1 ,R ∗ , ( l ) } L l =1 . For 2 ≤ l ≤ L , C ∗ , ( l ) A ∗ , ( l ) j ,R ∗ , ( l ) use output and input feature dimension d ′ = Td . Similar to the discussion about feature dimension above, since we need to concat the unary predicates information before each timestamp, for layer l = 1 , C ∗ , (1) ,A ∗ , (1) j ,R ∗ , (1) have input dimension d ′ + | P 1 | and output dimension d ′ . For dimension alignment, x 0 v used in Equation (53) is defined as zero-vector with length d ′ .

Next let's define some symbols for notation simplicity. For a feature vector x , let x [ i,j ] denotes the slice of x in dimension [ i,j ] . By the discussion above, in the following construction process we will only need feature x with dimension d ′ or d ′ + | P 1 | . When x has dimension d ′ , x ( i ) denotes x [( i -1) d +1 ,id ] , otherwise it denotes x [ | P 1 | +( i -1) d +1 , | P 1 | + id ] . Let [ x 1 ...... x T ] or [ x t ] T t =1 denotes the concatenation of a sequence of feature x 1 ...... x T , and [ x ] n denote concatenation of n copies of x , 0 n denotes zero vectors of length n . Furthermore. Let X denotes a multiset of x . Follows the updating rules defined in Equation (2), for all 1 ≤ j ≤ | P 2 | , 1 ≤ l ≤ L, A ∗ , ( l ) j ,R ∗ , ( l ) should get input of form X , and the combination function C ∗ , ( l ) should get input of form ( x 0 , ( x j ) | P 2 | j =1 , x g ) , where x 0 is from the node itself, ( x j ) | P 2 | j =1 are from aggregation functions ( A ∗ , ( l ) j ) | P 2 | j =1 and x g is from the global readout R ∗ , ( l ) . The dimension of x or X should match the input dimension of corresponding function. For all 1 ≤ l ≤ L , parameters in layer l for the new model are defined below

The core trick is to use T disjoint slices x (1) ...... x ( T ) to simulate T different models A 1 ...... A T at the same time, Since these slices are isolated from each other, a proper construction above can be found. The only speciality is that in layer l = 1 , we have to incorporate the unary predicate information x [1 , | P 1 | ] into each slice. By the construction above, we can see that for any node v , x ( T ) v is exactly the its feature outputted by A . Therefore, we finally construct an homogeneous R 2 -TGNN equivalent with A .

Now, we start to prove Theorem 16.

Theorem 16. R 2 -TGNNs ◦ F T = R 2 -GNNs ◦ F ◦ H on any universal graph class G u .

Proof. Since R 2 -TGNN ◦ F T only uses a part of predicates of P ′ = F ( H ( P )) in each timestamp, the forward direction R 2 -TGNN ◦ F T ⊆ R 2 -GNN ◦ F ◦ H trivially holds.

For any R 2 -GNN A under P ′ , we want to construct an R 2 -TGNN A ′ under F T ( P ) such that for any temporal knowledge graph G , A ′ outputs the same feature vectors as A on F T ( G ) . We can assume A is represented as ( C ( l ) , ( A ( l ) j ) K j =1 ,A ( l ) aux 1 ,A ( l ) aux 2 ,R ( l ) ) L l =1 , where K = T | P 2 | .

First, by setting feature dimension to be d ′ = T | P | +3 . We can construct an R 2 -TGNN A ′ whose output feature stores all facts in F ( H ( G )) for any graph G . Formally speaking, A ′ should satisfy the following condition:

For any primal node a , its feature outputted by A ′ ◦ F T should store all unary facts of form A i ( a ) ,A i ∈ T | P 1 | or primal ( a ) on F ( H ( G )) . For any non-primal node e ab , its feature outputted by A ′ ◦ F T should store all binary facts of form r i ( a,b ) ,r i ∈ T | P 2 | or r aux 1 ( a,b ) ,r aux 2 ( a,b ) where b is another node on F ( H ( G )) .

The A ′ is easy to construct since we have enough dimension size to store different predicates independently, and these facts are completely encoded into the initial features of corresponding timestamp. Let ( A ′ 1 ...... A ′ T ) denote A ′ .

Next, in order to simulate A , we need to append some layers to A ′ T . Let L denote the layer number of A , we need to append L layers represented as ( C ∗ , ( l ) , ( A ∗ , ( l ) j ) | P 2 | j =1 ,A ∗ , ( l ) aux 1 ,A ∗ , ( l ) aux 2 ,R ∗ , ( l ) ) L l =1

Since we have enough information encoded in features, we can start to simulate A . Since neighbor distribution of primal nodes don't change between F T ( G ) T and F ( H ( G )) , it's easy to simulate all messages passed to primal nodes as destinations by A ∗ , ( l ) aux 1 . For messages passed to non-primal node

R

2

-TGNN

time-and-graph

R

2

-GNN

◦

H

R

2

-GNN

◦

F

◦

H

R

2

-TGNN

◦

F

T

time-then-graph

⊊

⊊

⊊

⊈

e ab as destination, it can be divided into messages from a and messages from e ba . The first class of messages is easy to simulate since the aux 1 edge between e ab and a is the same on F T ( G ) T and F ( H ( G )) .

For the second class of messages, since edges of type r i , 1 ≤ i ≤ T | P 2 | may be lost in F T ( G ) T , we have to simulate these messages only by the unchanged edge of type aux2 . It can be realized by following construction:

where K = T | P 2 | , A ′ , ( l ) j ( X ) := A ( l ) j ( X ) if and only if e ba has neighbor r j on F ( H ( G )) , otherwise A ′ ( l ) j ( X ) := 0 . Note that X is exactly the feature of e ba , and we can access the information about its r j neighbors from feature since A ′ has stored information about these facts.

In conclusion, we've simulated all messages between neighbors. Furthermore, since node sets on F T ( G ) T and F ( H ( G )) are the same, global readout R ( l ) is also easy to simulate by R ∗ , ( l ) . Finally, using the original combination function C ( l ) , we can construct an R 2 -TGNN on F T equivalent to A on F ( H ( G )) for any temporal knowledge graph G .

J Proof of Theorem 17

Based on Theorem 15, Theorem 16 and Corollary 11.2, in order to prove Theorem 17, it suffices to show the following theorems.

Theorem 31. If time range T > 1 R 2 -TGNN ⊊ R 2 -GNN ◦ H .

Theorem 32. If time range T > 1 R 2 -TGNN ⊈ time-and-graph.

Proof. Since a formal proof Theorem 32 relates to too many details in definition of time-and-graph (Please refer to Gao and Ribeiro [2022]) which is not the focus here. We will just a brief proof sketch of Theorem 32: That's because time-and-graph can not capture a chain of information that is continuously scattered in time intervals. Specifically, φ ( x ) := ∃ ≥ 1 y, ( r 2 1 ( x,y ) ∧ ( ∃ ≥ 1 x,r 1 1 ( y,x )) ) can't be captured by time-and-graph but φ ( x ) is in R 2 -TGNN.

We mainly give a detaild proof of Theorem 31: Since in each timestamp t , R 2 -TGNN only uses a part of predicates in temporalized predicate set P ′ = H ( P ) , R 2 -TGNN ⊆ R 2 -GNN ◦ H trivially holds. To show R 2 -TGNN is strictly weaker than R 2 -GNN ◦ H . Consider the following classifier:

Let time range T = 2 , and let r be a binary predicate in P 2 . Note that there are two different predicates r 1 ,r 2 in P ′ = H ( P ) . Consider the following temporal graph G with 5 nodes { 1 , 2 , 3 , 4 , 5 } . its two snapshots G 1 ,G 2 are as follows:

It follows that after transformation H , the static version of G is:

Consider the logical classifier ∃ y ( r 1 ( x,y ) ∧ ( ∃ xr 2 ( x,y )) ) under P ′ .It can be captured by some R 2 -GNN under P ′ . Therefore, R 2 -GNN ◦ H can distinguish nodes 1 , 4 .

However, any R 2 -TGNN based on updating rules in Equation (53) can't distinguish these two nodes, so R 2 -TGNN is strictly weaker than R 2 -GNN ◦ H .

Based on Theorem 31, we can consider logical classifier φ 3 := ∃ ≥ 2 y ( p 1 1 ( x,y ) ∧ p 2 1 ( x,y )) . Note that this classifier is just renaming version of Figure 1. Therefore φ 3 can't be captured by R 2 -GNN ◦ H , not to say weaker framework R 2 -GNN by Theorem 31.

.

K Experiment Supplementary

K.1 Synthetic dataset generation

For each synthetic datasets, we generate 7000 graphs as tranining set and 500 graphs as test set. Each graph has 50 -1000 nodes. In graph generation, we fix the expected edge density δ . In order to generate a graph with n nodes, we pick δn pairs of distinct nodes uniformly randomly. For each selected node pair a,b , each timestamp t and each binary relation type r , we add r t ( a,b ) and r t ( a,b ) into the graph with independent probability 1 2 .

K.2 Statistical Information for Datasets

We list the information for synthetic dataset in Table 4 and real-world dataset in Table 5. Note that synthetic datasets contains many graphs, but real-world datasets only contains a single graph. Therefore, for real-world dataset, we have two disjoint node set as train split and test split for training and testing respectively. In training, the model can see the subgraph induced by train split and unlabelled nodes, in testing, the model can see the whole graph but only evaluate the performance on test split.

K.3 Hyper-parameters

For all experiments, we did grid search according to Table 6.

K.4 More Results

Apart from those presented in main part, we have some extra experimental results here:

1. Extra results on synthetic datasets but using different base model architecture, where R-GAT refers to Busbridge et al. [2019] and R 2 -GAT refers to its extension with global readout. Please Refer to Table 7. These results show the generality of our results on different base models within the framework.

2. Extra results for static real-world datasets. Add a base model R-GATBusbridge et al. [2019] and two larger real-world datasets DGS and AM from Schlichtkrull et al. [2018]. Please refer to Table 8. From the results for two bigger datasets DGM and AM, we can see our framework outperforms the other baselines, which confirms the scalability of our method and theoretical results. These results show our method is effective both on small and large graphs.

3. Extra results for temporal real-world dataset Brain-10. Please refer to Table 9. These results implies that our method is effective on different base models in temporal settings. Moreover, we can see separate improvements from global readout and graph transformation respectively. As we said in the main part, the drop in the last column may be due to the intrinsic drawbacks of current real-world datasets. Many real-world datasets can not be perfectly modeled as first-order-logic classifier. This non-logical property may lead to less convincing experimental results. As Barceló et al. [2020] commented, these commonly used benchmarks are inadequate for testing advanced GNN variants.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with classifying multi-relational graphs, and specifically, the R-GNN architecture, which deals with more than one type of edge by producing a different neighbor set for each of them, aggregating the messages of each neighbor, and then merging them together in a final update layer. The first results in the paper are that these architectures are limited, as Figure 1 shows, these architectures cannot even count the number of total neighbours of a node. To fix this, the proposal is to maintain the architecture, but apply a graph transformation task, some sort of graph reification. Intersectingly, R-GNNs and R^2-GNNs recover the expressive power of similar architectures in the single-relation scenario, when applied not under the original graphs but in the transformed graphs. The second part of the paper looks to implement these ideas specifically in Temporal Graphs, wherein one has a different relation for each timestamp measured in the temporal part. As multi-relational graphs, this is indeed a nice application of the framework. Results and experiments show that the proposed architecture competes with several other proposals for classification of temporal graphs.\n\nSTRENGTHS:\n* Sound, robust theoretical study relating multi-label GNNs with logic. The proofs seem correct and use a variety of techniques.\n *   Claims are backed up with examples, so that it is easy to quickly grasp the ideas of the paper.\n *   The claims in the paper are backed up with experiments: It really does seem that the power of R-GNNs (and similar architectures) benefits from applying the graph transformation.\n\nWEAKNESSES:\n*   The results are tailored specifically to a single architecture (R-GNNs). One could think of several other ways to incorporate edge information, I know for instance GATs (Velickovic et al. 2018), but the paper does not discuss any other approach, and we don't know if the weakness shown in the results of section 4 are due to the specific architecture, or if there is a bigger problem underlying multi-relation graphs that must be tackled with the transformation.\n\n*    The proposed solution involves a linear transformation in graphs, but probably demands much more layers in a GNN, as the length of every path is now multiplied by two. This involves adding extra costs to the learning process, and probably some difficulties in using the node embedding of the graph.\n\n*    Synthetics experiments do not compare against other GNN architectures capable of dealing with different edges, appart from R-GNN. Real life experiments do compare against other temporal variants, but it is difficult to see if the added power is due to the global readout, the transformation, or the mix of everything in the architecture proposed.\n\nQUESTIONS:\n*  Please justify why a study on R-GNN is important, and whether your insights could be valuable when dealing with any other architecture incorporating edges.\n\n  *  I'd tend to think that maybe every multi-labelled GNN would benefit from the graph transformation approach, and like wise for the global readout. What would happen if I run GRU-GCN or TGN over a temporal graph that has already been transformed? Or if instead of using R^2-GNN I use any other GNN with the graph transformation?\n\nLIMITATIONS:\n*    Please discuss potential inconveniences regarding using the transformation in graphs. You probably need more layers, with the added cost on training. Also, even if the transformation is linear, it is not free for huge graphs, so your approach is probably better tailored to datasets containing several small graphs.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper justifies the logical expressiveness of R^2-GNN from the perspective of \"universal\" graph classes, including multigraphs and infinite graphs. Specifically, the R^2-GNN seems to be identical to the previous ACR-GNN in [1]. The engagement of graph classes makes theoretical statements in finer settings than previous ones in [1] which only involve one graph. The thorough inspection of different graph classes captures the expressiveness power of R^2-GNN compared to logical formulas in FOC_2. Moreover, this paper adapts a commonly used graph transformation $F$ to enhance the expressiveness of R^2-GNN to be stronger than FOC_2 on universal graph classes.\n\nThe theoretical results collapse into existing ones in [1] when the graph classes contain finite graphs, which are naturally bounded. The theoretical results can be also extended to temporal graphs under static representations, which are studied in [2]. To achieve this, the authors investigate the expressiveness hierarchy between the collapse function $H$, two static representations, GNN and TGNN (also defined by the author), and graph transformation $F$.\n\nEmpirical results support the expressiveness results partially related to $F$, $H$, and TGNN.\n\n[1] Barceló, P., Kostylev, E. V., Monet, M., Pérez, J., Reutter, J., & Silva, J. P. (2020, April). The logical expressiveness of graph neural networks. In 8th International Conference on Learning Representations (ICLR 2020).\n[2] Gao, J., & Ribeiro, B. (2021). On the equivalence between temporal and static graph representations for observational predictions. arXiv preprint arXiv:2103.07016.\n\nSTRENGTHS:\nThis paper makes a large amount of work to (1) justify the logical expressiveness of R^2-GNN compared to FOC_2 in universal, bounded, and simple graph classes; (2) empirical technics to improve the expressiveness of GNN.\n\nThe theoretical results are original. Though some results are still missing to achieve a complete understanding of this problem. The presented ones are significant and inspiring.\n\nWEAKNESSES:\nThe presentation of this paper should be improved given the dense representation of theorems and proofs. \n\nFirstly, the content is not self-contained. The definition of static representation of temporal graphs is not included even in the appendix.\n\nMost importantly, the key definition to justify logical expressiveness is missing. In Lines 130-131, the author states that the R^2-GNN is the set of all R^2-GNN-based boolean node classifiers. However, it is far from a satisfactory definition, which might be risky for accepting this paper. Please check the question part for my questions about the definition.\n\nAnotherW ambiguous part of this paper is that the graph transformation $F$ for multi-graphs is isolated to FOC_2 classifiers. In fact, $F$ should be orthogonal to classifiers.\n\nQUESTIONS:\n1. What is the definition of one boolean classifier set A as the subset of another boolean classifier set B for a given graph class GC? There are two possible definitions\n\n1. For any a in A, there exists a b(a) in B so that a and b(a) achieve the same results on all G in GC.\n2. For any G in GC and any a in A, there exists a b(G, a) in B so that a and b(G, a) achieve the same result on G.\n\n2. $F$ enhanced the power of R^2-GNN classifiers but it never relates the R^2-GNN. Can we state that $F$ also enhances the expressiveness of FOC_2 classifiers? What will Figure 3 be if we include FOC_2 \\odot F (or with proper modification of FOC_2 with new predicates introduced)?\n\n3.  For the temporal graph, the author draws two lines of hierarchies of expressiveness. What are the relationships between R^2-TGNN and time-and-graph and R^2-GNN\\odot H and time-and-graph?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new connection between the power of the relational GNNs (working on multi-relational graphs) with a class of first-order logic functions. This class of functions named $\\mathcal{FOC}_2$ is a subset of first-order logic functions constrained to having only two variables, but includes counting quantifiers. The theoretical analysis considers three classes of multi-relational graphs: universal graph class, bounded size multi-relational graphs, and simple graphs that two nodes can have just one type of connection. Based on these classes paper proves in general none of $R^2-GNN$ and $\\mathcal{FOC}_2$ class of functions is a subset of each other, but adding a preprocess to the graph can make a model that is at least as powerful as the union of both classes. They also extend their results to the temporal graphs, where time series graphs can be collapsed into a static graph or considered as separate graphs. The paper also compares two different classes of models, time-and-graph versus time-then-graph, and proves second class is absolutely more powerful. Empirical results on synthetic and real-world graphs have been also conducted for supporting theoretical results.\n\nSTRENGTHS:\n1. Paper for almost all parts is very well-written.\n2. Paper finds the exact class of functions that can be learned by most common relational GNNs. \n3. Theorems show the theoretical superiority of time-then-graph functions on temporal graphs over the time-and-graph models.\n4. Paper finds a hierarchal classification of the power of different models on temporal graphs.\n5. Having supporting experiments for showing that theoretical implications have real effects on real-world datasets.\n\nWEAKNESSES:\n1. Intuition behind the functions used for synthetic graphs has not quite been explained. A little more explanation on why these functions and if they are just a random selection or one of the most simple functions that could convey a point will be helpful.\n2. The paper relies a lot on definitions of time-then-graph and time-and-graph models and refers to previous work for their definitions. These definitions are better to be briefly explained at least as a section in the Appendix.\n3. Section 6 starts with the Experiments and it is hard at first to understand that all synthetic are temporal. Particularly, superscripts on the relations were confusing at first without knowing they are temporal indices. It would have been better to clearly indicate that before starting and also include synthetic experiments for the general multi-relational graphs part.\n4. Real-world experiment on the Brain-10 dataset is a little inconsistent with theoretical results. \n\nMinor errors:\n1. Inconsistent use of $FOC$ and $\\mathcal{FOC}$ in the notations.\n2. Line 110, a space missing between \"vectors\" and \"$\\mathbf{x}_v^{(i)}$\".\n3. Line 339, \"somorphism\" -> \"isomorphism\"\n\nQUESTIONS:\n1. Node-level binary classification tasks are limited in the real world. Theoretical results are insightful, however, I was wondering if these results can imply anything about any other type of tasks on the graphs, e.g. multi-class node classifications or graph-level tasks.\n2. In Corollaries 4.1 and 9.1, it is interesting to see that both classes have precisely same power. However, this power seems to appear because of $\\mathcal{FOC}_2$ being able to enumerate all possible combinations of relation counts. Is there any complexity measurement, e.g. time-complexity of evaluating a function on a graph, on how complex can the function under $\\mathcal{FOC}_2$ be as the bound on the number of nodes in the graph or number of possible relations in the class increases?\n\nLIMITATIONS:\nThe paper clearly defines the class of functions analysis. There are limitations on what types of graphs and tasks theoretical results work, but these limitations are clearly indicated in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the logical characterization of GNN --- node classification, by investigating the relational case i.e. multi-relational graphs. The paper generalizes the work of Barcelo et al. [ICLR 2020], who provided the logical characterization of GNNs for node classification in terms of First Order Logic with two variables and counting quantifiers (FOC2). The paper discuss the R2-GNN model, that extends R-GNN --- a GNN architecture for multi-relational graphs --- with a global read out function. The global readout adds aggregation over node features to the combination function in each layer. Some of the key results of the paper are the following:\n\n- R2-GNN are not captured in FOC2 and FOC2 is not captured by R2-GNN (for unbounded graphs)\n- R2-GNN are  captured in FOC2 for bounded graphs (given a known upper-bound on the number of nodes) but not vice versa\n- R2-GNN = FOC2 for simple graphs \n\nThe paper then extends the expressivity of R2-GNNs to full FOC2 , by introducing a transformation F that runs in linear time w.r.t the multi-graph size (O(|V|+|E|)). \n\nFinally, the authors extend there framework to temporal graphs. This is achieved by simply adding additional predicates indexed by time. The authors then discuss the expressivity of this framework with the previously introduced transformation  F applied to each time stamp  and another transformation comprising union of each time stamped graph.\n\nPost-rebuttal: I have read author's rebuttal, and I am more confident about my rating and will keep it, i.e. acceptance.\n\nSTRENGTHS:\nThe paper extends logical characterization of GNNs in an interesting direction i.e. over relational structures. The theory presented is rather intuitive and the authors try to convey the key proof ideas. They also exploit their theoretical investigation to expand the expressivity of GNNs over relational structures and temporal graphs. They  provide convincing experiments (although slightly redundant, see weaknesses) over synthetic and real-world data.\n\nWEAKNESSES:\n- Quite terse for non-experts\n- Comparison of all the aggregation functions in tables are not much discussed in the paper and are not really important to the main message of the paper. Maybe this could be removed/shortened to give space for more proof intuitions?\n\nQUESTIONS:\nWhen using many predicates, it seems that one could encounter some form of curse of dimensionality --- Do you think R2-GNNs suffer such issues? Specially in the temporal/transformation case, when number of predicates are further increased.\n\nIn general, some discussion on how the complexity of learning changes with number of predicates could be interesting\n\nLIMITATIONS:\nThe authors discuss the fact that R2-TGNN*F^{T} do not improve performance over R2-TGNN on real-world dataset, whereas on their synthetic dataset they do observe improvement. This ambivalence between theory and practice is quite interesting and further discussion of this--- as to what exactly is different in real-world data that makes more expressive models inferior --- could be very interesting", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors provide a theoretical analysis of graph neural network (GNN) expressivity on multi-relational and temporal graphs. Specifically, they extend the multi-relational graph convolution network R-GCN to a class of architectures they call R-GNN, which allows for arbitrary aggregation (pooling neighbors with respect to a particular relation) and combination functions (pooling these aggregations with the node representation itself). They further extend this to a class of architectures they call R²-GNN, which adds a global \"readout\" function which aggregates the feature vectors of all nodes in the graph. They analyze the capability of R²-GNN to classify nodes and compare it to that of $\\mathcal{FOC}_2$, a restricted subset of first-order logic which only allows formulas with at most 2 variables but also allows counting quantifiers. They find that neither is a subset of the other, in particular R²-GNN is not able to distinguish whether two neighbors of a given node with different relations are, in fact, the same node. However by first augmenting the graph with a particular transformation $F$ they are able to overcome this restriction, and ultimately show that $R^2-GNNs \\circ F = \\mathcal{FOC}_2$ on any graph class with a bounded number of nodes.\n\nAll of this is then extended to temporal graphs, which can also be converted to a multi-relational graph (with a unique set of relations per time-step). A hierarchy of expressivity with respect to current temporal graph frameworks is presented. Finally, experimental results validating the theoretical expressivity on synthetic and real-world datasets are presented.\n\nSTRENGTHS:\nThe paper is, to the best of my knowledge, original, and theoretical results analyzing the expressivity of graph neural networks in the multi-relational setting are certainly of significant interest to the graph neural network community.\n\nThe work is of a high quality, and presented clearly. Despite the highly technical nature of the results, the authors do a good job in motivating their work and providing intuition for the proofs.\n\nWEAKNESSES:\nIn general, a point of caution with any strongly theoretical work is whether it applies in practice. In this setting the largest concern comes from the graph transformations. It is not unusual for a graph transformation to extend capabilities in the way the authors have done. A classic example is to extend a method which works on undirected graphs to the setting of directed graphs by creating a new graph with twice as many nodes, one representing \"head\" and \"tail\" for each node, however this can make some relationships (eg. transitivity) very difficult to observe. In the author's setting, while the graph augmentation may afford the ability to prove theoretical expressivity results, one might be concerned that the resulting graph is augmented to the point that a GNN architecture may require greater depth to perform the same sort of tasks it did previously. Of course the authors have provided empirical evidence that this augmentation is not problematic in at least three graphs, which is quite reasonable to support their approach, however it is something to bear in mind.\n\nQUESTIONS:\n1. What is the initialization used for the nodes in $V' \\setminus V$? To the best I can tell, they do not have any unary predicates, so they would seem to be initialized with a vector of all zeros.\n2. Not a question as much as a comment - the R²-GNN architecture is equivalent to the R-GNN architecture if we first augment the graph to have a new binary relation which forms a complete graph on all nodes. Given that the work already includes a graph augmentation step, an alternative presentation could potentially avoid the R²-GNN architecture altogether, and simply express the entire thing as R-GNN on an augmented graph. I'm not sure it is clearer to take this approach, but it might be worth noting.\n\nMost of the paper was very well written, but I did find the following typos (mostly in the intro):  \nL37: authors -> the authors  \nL45: authors -> the authors  \nL46: in -> in the  \nL47: edges in graphs -> edge in the graph  \nL48: of -> (delete)  \nL48: graph needs -> graphs need  \nL51: as -> as a  \nL57: be -> can be  \nL60: to -> to the  \nL60: Besides -> Moreover  \nL61: extensively -> extensively in  \nL69: in -> in the  \nL70: power -> the power  \nL71: such -> such a  \nL72: mutli -> multi  \nL91: to -> to denote  \nL96: node Boolean -> Boolean node  \nL111: vectorsx -> vectors x  \nL338: do somorphism -> perform an isomorphism\n\nA minor suggestion: move the sentence in lines 170-171 to around line 85, to motivate the definition of bounded and simple graph classes.\n\nLIMITATIONS:\nI did not see much discussion of the limitations of the proposed model. Some of the concerns outlined in the weaknesses section above are good candidates for this.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work calibrate the logic expressiveness of R2-GNNs as node classifiers on multi-relational graphs. Motivated by some negative results, authors boost R2-GNNs with a graph transformation, which enables R2-GNN to capture FOC2 formulas. They further extend expressiveness results and graph transformation to temporal settings and derive an expressiveness hierarchy of temporal GNNs.\n\nSTRENGTHS:\n1. Clear theoretical results on static and temporal settings.\n\n2. Proposed transformation is straightforward and scalable.\n\nWEAKNESSES:\n1. Nearly no related work. There is no related work section. Only the closest related work is mentioned in introduction and some related works are missing. For example, besides [1], some works also analyze GNN's expressivity as boolean classifier. Weifeiler-Leman test have been connected to boolean graph classifier for a long time [2]. [3] also analyze boolean link classifier.\n\n2. The experiments are only conducted on small datasets. You can use larger DGS and AM in RGCN paper [4].\n\n\n[1] Pablo Barceló, Egor V. Kostylev, Mikael Monet, Jorge Pérez, Juan Reutter, and Juan Pablo Silva. The logical expressiveness of graph neural networks. ICLR, 2020\n\n[2] Jin-yi Cai, Martin Fürer, Neil Immerman, An Optimal Lower Bound on the Number of Variables for Graph Identification. FOCS 1989: 612-617.\n\n[3] Xingyue Huang, Miguel A. Romero Orth, Ismail Ilkan Ceylan, Pablo Barceló, A Theory of Link Prediction via Relational Weisfeiler-Leman. CoRR abs/2302.02209 (2023).\n\n[4] Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, Max Welling: Modeling Relational Data with Graph Convolutional Networks. ESWC 2018.\n\nQUESTIONS:\n1. There is no scalability comparison between the proposed method and baselines. Please add it.\n\nLIMITATIONS:\nLimitation is not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the R^2-GNN that captures the logic features in the graph. The proposed methodology largely resembles the previous work of ACR-GNN, where a GNN variant is proposed to capture the logic feature for graphs with no relation types. The proposed R^2-GNN extends the previous method by considering the graphs that involve a set of unary and binary predicates and potentially temporal information. To do so, the authors apply the same readout function to the base R-GNN model and show that it can capture the full logic classifiers with an additional graph transformation F.  Also an temporal variant R^2-TGNN is proposed to handle the temporal graphs.\n\nSTRENGTHS:\nThe paper is overall easy to read, however, I didn't check all the proof in detail.\n\nWEAKNESSES:\n## Novelty\n\nAs mentioned in summary, the authors follow the same method in ACR-GNN in modifying R-GNN into R^2-GNN to enable the model to capture the FOC_2 logic family. It seems to me the main methodological differences are the additional graph transformation F and the discussion on temporal graphs. \n- The former helps the model to distinguish nodes with different relations, while it seems to work empirically, it is more or less an incremental modification to the framework. \n- For the temporal graphs, the author propose to \"temporalize\" the graph by mapping the graph snapshots into one single static graph and process it with a slightly modified model, namely R^2-TGNN.\n\nThat said, this paper is not very novel methodology-wise.\n\n\n## Quality\n\nThe definition of a multi-relational graph is more or less redundant and confusing: this is effectively the graph with both unary and binary predicates, and many existing KGs already contain facts of both these types. One can refer to it as a real-world graph or just an ordinary graph instead of as \"multi-relational\".\n\nSome model and design choices lack justification:\n- It is unclear why the authors pick the R-GNN as the base model, provided that: (1) most existing GNNs can handle graphs with different relations and (2) the proposed modifications such as readout function and transformation F and H are orthogonal to the choice of the model. The authors may want to consider evaluating their method on different base models empirically to better demonstrate the difference\n- It is also unclear why the task is limited to node classification only: this leads to a narrow choice of the benchmarks in the experiments (L329-L330) and leaves some popular datasets untested such as GDELT and ICEWS.\n\nI'm also concerned about the real-world datasets experiments. The authors show the R^2-(T)RGNN achieves better performance than other GNN variants. While empirically this does have some merits, it does not provide any insights into the proposed model. Since the model focuses on learning the latent FOC_2 classifiers. One should first inspect the datasets and see what are the learnable FOC_2 rules in there. And if the model performs better, the authors should take the opportunity and analyze if this indeed resulted from the extra capabilities introduced to the model.\n\n\n## Clarity\n\nThe paper is overall easy to read, however, I didn't check all the proof in detail.\n\n\n## Significance\n\n\nApart from the lack of novelty, this paper also seems to lack significance. The FOC_2 logic is a small subset of first-order logic. Being able to capture these classifiers leads to minor benefits to the GNN literature: it does not provide sufficient explanation of the GNN model, nor does it tackle the difficult logical reasoning problems on graphs such as multi-hop reasoning, induction, and so on. In the experiments, the model is tested on only synthetic and three real-world datasets, and with no comparison to the SOTA models, so the empirical significance is also unclear.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}