SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dingmin Wang*
dingmin.wang@cs.ox.ac.uk
Yeyuan Chen*
yeyuanch@umich.edu
As a powerful framework for graph representation learning, Graph Neural Networks (GNNs) have garnered significant attention in recent years. However, to the best of our knowledge, there has been no formal analysis of the logical expressiveness of GNNs as Boolean node classifiers over multi-relational graphs, where each edge carries a specific relation type. In this paper, we investigate  $\mathcal{FOC}_2$ , a fragment of first-order logic with two variables and counting quantifiers. On the negative side, we demonstrate that the  $\mathbb{R}^2$ -GNN architecture, which extends the local message passing GNN by incorporating global readout, fails to capture  $\mathcal{FOC}_2$  classifiers in the general case. Nevertheless, on the positive side, we establish that  $\mathbb{R}^2$ -GNN models are equivalent to  $\mathcal{FOC}_2$  classifiers under certain restricted yet reasonable scenarios. To address the limitations of  $\mathbb{R}^2$ -GNN regarding expressiveness, we propose a simple graph transformation technique, akin to a preprocessing step, which can be executed in linear time. This transformation enables  $\mathbb{R}^2$ -GNN to effectively capture any  $\mathcal{FOC}_2$  classifiers when applied to the "transformed" input graph. Moreover, we extend our analysis of expressiveness and graph transformation to temporal graphs, exploring several temporal GNN architectures and providing an expressiveness hierarchy for them. To validate our findings, we implement  $\mathbb{R}^2$ -GNN and the graph transformation technique and conduct empirical tests in node classification tasks against various well-known GNN architectures that support multi-relational or temporal graphs. Our experimental results consistently demonstrate that  $\mathbb{R}^2$ -GNN with the graph transformation outperform the baseline methods on both synthetic and real-world datasets. The code is available at https://github.com/hdmmlz/multi-graph.
Graph Neural Networks (GNNs) have become a standard paradigm for learning with graph structured data, such as knowledge graphs Park et al. [2019], Tena Cucala et al. [2021], Wang et al. [2023] and molecules Hao et al. [2020], Gasteiger et al. [2021], Guo et al. [2021]. GNNs take as input a graph where each node is labelled by a feature vector, and then they recursively update the feature vector of each node by processing a subset of the feature vectors from the previous layer. For example, many GNNs update a node's feature vector by combining its value in the previous layer with the output of some aggregation function applied to its neighbours' feature vectors in the previous layer; in this case, after  $k$  iterations, a node's feature vector can capture structural information about the node's  $k$ -hop neighborhood. GNNs have proved to be very efficient in many applications like knowledge graph completion and recommender systems. Most previous work on GNNs mainly revolves around finding
*Equal contribution, listed in alphabetical order.
GNN architectures (e.g. using different aggregation functions or graph-level pooling schemes) which offer good empirical performance Kipf and Welling [2016], Xu et al. [2018], Corso et al. [2020]. The theoretical properties of different architectures, however, are not yet well understood.
In Xu et al. [2018], the authors first proposed a theoretical framework to analyze the expressive power of GNNs by establishing a close connection between GNNs and the Weisfeiler-Lehman (1-WL) test for checking graph isomorphism. Similarly, Geerts and Reutter [2022] provides an elegant way to easily obtain bounds on the separation power of GNNs in terms of the Weisfeiler-Leman (k-WL) tests. However, the characterization in terms of the Weisfeiler-Lehman test only calibrates distinguishing ability. It cannot answer which Boolean node classifier can be expressed by GNNs. To this end, Barceló et al. [2020] consider a class of GNNs named ACR-GNNs proposed in Battaglia et al. [2018], where the update function uses a "global" aggregation of the features of all nodes in the graph in addition to the typical aggregation of feature vectors of neighbour nodes. Then, the authors of the paper prove that in the single-relational  $^2$  scenario, ACR-GNNs can capture every Boolean node classifier expressible in the logic  $\mathcal{FOC}_2$ .
However, most knowledge graphs need multiple relation types. For example, in a family tree, there are multiple different relation types such as "father" and "spouse". In this paper, we consider the abstraction of a widely used GNN architecture called R-GCN Schlichtkrull et al. [2018], which is applicable to multi-relational graphs. Following Barceló et al. [2020], we define  $\mathbb{R}^2$ -GNN as a generalization of R-GCN by adding readout functions to the neighborhood aggregation scheme. We show that although adding readout functions enables GNNs to aggregate information of isolated nodes that can not be collected by the neighborhood-based aggregation mechanism,  $\mathbb{R}^2$ -GNN are still unable to capture all Boolean node classifiers expressible as formulas in logic  $\mathcal{FOC}_2$  in multi-relational scenarios if applied "directly" to the input. This leaves us with the following questions: (1) Are there reasonable and practical sub-classes of multi-relational graphs for which  $\mathcal{FOC}_2$  can be captured by  $\mathbb{R}^2$ -GNN? (2) Is there some simple way to encode input graphs, so that all  $\mathcal{FOC}_2$  node classifiers can be captured by  $\mathbb{R}^2$ -GNN for all multi-relational graphs?
In this paper, we provide answers to the above questions. Moreover, we show that our theoretical findings also transfer to temporal knowledge graphs, which are studied extensively in Park et al. [2022] and Gao and Ribeiro [2022]. In particular, we leverage the findings from Gao and Ribeiro [2022] which shows that a temporal graph can be transformed into an "equivalent" static multi-relational graph. Consequently, our results, originally formulated for static multi-relational graphs, naturally extend to the domain of temporal knowledge graphs. Our contributions are as follows:
- We calibrate the logic expressiveness of  $\mathbf{R}^2$ -GNN as node classifiers over different subclasses of multi-relational graphs.
- In light of some negative results about the expressiveness of  $\mathbf{R}^2$ -GNN found in the multi-relational scenario, there is a compelling need to boost the power of  $\mathbf{R}^2$ -GNN. To address this challenge, we propose a graph transformation and show that such a transformation enables  $\mathbf{R}^2$ -GNN to capture each classifier expressible as a  $\mathcal{FOC}_2$  formula in all multi-relational graphs.
- We expand the scope of expressiveness results and graph transformation from static multi-relational graphs to temporal settings. Within this context, we propose several temporal GNN architectures and subject them to a comparative analysis with frameworks outlined in Gao and Ribeiro [2022]. Ultimately, we derive an expressiveness hierarchy.
2.1 Multi-relational Graphs
A multi-relational graph is a 4-tuple  $G = (V, \mathcal{E}, P_1, P_2)$ , where  $V, P_1, P_2$  are finite sets of nodes, types and relations (a.k.a., unary/binary predicates)<sup>3</sup>, respectively, and  $\mathcal{E}$  is a set of triples of the form  $(v_1, p_2, v_2)$  or  $(v, type, p_1)$ , where  $p_1 \in P_1, p_2 \in P_2, v_1, v_2, v \in V$ , and type is a special symbol.
Next, given arbitrary (but fixed) finite sets  $P_{1}$  and  $P_{2}$  of unary and binary predicates, respectively, we define the following three kinds of graph classes:
2The "single-relational" means there is only one type of edges in the graph.
3For directed graphs, we assume  $P_{2}$  contains relations both in two directions (with inverse-predicates). Moreover, we assume there exists an "equality relation"  $\mathsf{EQ}\in P_2$  such that  $\forall x,y\in V,x = y\Leftrightarrow \mathsf{EQ}(x,y) = 1$
- a universal graph class can be any set of graphs of the form  $(V, \mathcal{E}, P_1, P_2)$ .
- a bounded graph class is a universal graph class for which there exists  $n \in \mathbb{N}$  such that each graph in the class has no more than  $n$  nodes;
- a simple graph class is a universal graph class where for each graph  $(V, \mathcal{E}, P_1, P_2)$  in the class, and for each pair of nodes  $v_1, v_2 \in V$ , there exists at most one triple in  $\mathcal{E}$  of the form  $(v_1, p_2, v_2)$ , where  $p_2 \in P_2$ .
We typically use symbols  $\mathcal{G}_u$ ,  $\mathcal{G}_b$ , and  $\mathcal{G}_s$  to denote universal, bounded, and simple graph classes, respectively.
Definition 1. For a given graph class over predicates  $P_{1}$  and  $P_{2}$ , a Boolean node classifier is a function  $\mathcal{C}$  such that for each graph  $G = (V, \mathcal{E}, P_{1}, P_{2})$  in that graph class, and each  $v \in V$ ,  $\mathcal{C}$  classifies  $v$  as true or false.
2.2 Graph Neural Networks
Node Encoding We leverage a GNN as a Boolean node classifier for multi-relational graphs, which cannot be directly processed by GNN architectures, requiring graphs where each node is labelled by an initial feature vector. Therefore, we require some form of encoding to map a multi-relational graph to a suitable input for a GNN. Such an encoding should keep graph permutation invariance Geerts and Reutter [2021] since we don't want a GNN to have different outputs for isomorphic graphs. Inspired by Liu et al. [2021] for a multi-relational graph  $G = (V,\mathcal{E},P_1,P_2)$  and an ordering  $p_1,p_2,\dots ,p_k$  of the predicates in  $P_{1}$ , we define an initialization function  $I(\cdot)$  which maps each node  $v\in V$  to a Boolean feature vector  $I(v) = \mathbf{x}_v$  with a fixed dimension  $|P_1|$ , where the  $i$ th component of the vector is set to 1 if and only if the node  $v$  is of the type  $p_i$ , that is,  $(\mathbf{x}_v)_i = 1$  if and only if  $(v,type,p_i)\in \mathcal{E}$ . If  $P_{1}$  is an empty set, we specify that each node has a 1-dimension feature vector whose value is 1. Clearly, this encoding is permutation invariant.
R-GNN R-GCN Schlichtkrull et al. [2018] is a widely-used GNN architecture that can be applied to multi-relational graphs. By allowing different aggregation and combination functions, we extend R-GCN to a more general form which we call R-GNN. Formally, let  $\left\{\{A_j^{(i)}\}_{j=1}^{|P_2|}\right\}_{i=1}^L$  and  $\{C^{(i)}\}_{i=1}^L$  be two sets of aggregation and combination functions. An R-GNN computes vectors  $\mathbf{x}_v^{(i)}$  for every node  $v$  of the multi-relational graph  $G = (V, \mathcal{E}, P_1, P_2)$  on each layer  $i$ , via the recursive formula
$$
\mathbf {x} _ {v} ^ {(i)} = C ^ {(i)} \left(\mathbf {x} _ {v} ^ {(i - 1)}, \left(A _ {j} ^ {(i)} \left(\left\{\left\{\mathbf {x} _ {u} ^ {(i - 1)} \mid u \in \mathcal {N} _ {G, j} (v) \right\} \right\}\right)\right) _ {j = 1} ^ {| P _ {2} |}\right) \tag {1}
$$
where  $x_{v}^{(0)}$  is the initial feature vector as encoded by  $I(\cdot), \{\cdot\}$  denotes a multiset,  $(\cdot)_{j=1}^{|P_2|}$  denotes a tuple of size  $|P_2|$ ,  $\mathcal{N}_{G,j}(v)$  denotes the neighbours of  $v$  via a binary relation  $p_j \in P_2$ , that is, nodes  $w \in V$  such that  $(v, p_j, w) \in \mathcal{E}$ .
$\mathbf{R}^2$ -GNN  $\mathbf{R}^2$ -GNN extends R-GNN by specifying readout functions  $\{R^{(i)}\}_{i = 1}^{L}$ , which aggregates the feature vectors of all the nodes in a graph. The vector  $\mathbf{x}_v^{(i)}$  of each node  $v$  in  $G$  on each layer  $i$ , is computed by the following formula
$$
\mathbf {x} _ {v} ^ {(i)} = C ^ {(i)} \left(\mathbf {x} _ {v} ^ {(i - 1)}, \left(A _ {j} ^ {(i)} \left(\{\{\mathbf {x} _ {u} ^ {(i - 1)} | u \in \mathcal {N} _ {G, j} (v) \}\}\right)\right) _ {j = 1} ^ {| P _ {2} |}, R ^ {(i)} \left(\{\{\mathbf {x} _ {u} ^ {(i - 1)} | u \in V \}\}\right)\right) \tag {2}
$$
Every layer in an  $\mathbf{R}^2$ -GNN first computes the aggregation over all the nodes in  $G$ ; then, for every node  $v$ , it computes the aggregation over the neighbors of  $v$ ; and finally, it combines the features of  $v$  with the two aggregation vectors; the result of this operation is the new feature vector for  $v$ . Please note that an R-GNN can be seen as a special type of  $\mathbf{R}^2$ -GNN where the combination function simply ignores the output of the readout function.
It is worth noting that R-GNN as well as  $\mathbf{R}^2$ -GNN is not a specific model architecture; it is a framework that contains a bunch of different GNN architectures. In the paper, we mentioned it's generalized from R-GCN (Schlichtkrull et al. [2018]), but our primary objective is to establish a comprehensive framework that serves as an abstraction of most Message-Passing GNNs (MPGNN).
In the definitions (Equations (1) and (2)), the functions can be set as any functions, such as matrix multiplications or QKV-attentions. Most commonly used GNN such as R-GCN (Schlichtkrull et al. [2018]) and R-GAT (Busbridge et al. [2019]) are captured (upper-bounded) within our R-GNN frameworks. Other related works, such as (Barceló et al. [2020], Huang et al. [2023], Qiu et al. [2023]) also use intrinsically the same framework as our  $\mathrm{R - GNN / R^2}$ -GNN, which has been widely adopted and studied within the GNN community. We believe that analyzing these frameworks can yield common insights applicable to numerous existing GNNs
GNN-based Boolean node classifier In order to translate the output of a GNN to a Boolean value, we apply a Boolean classification function  $CLS: \mathbb{R}^d \to \{\text{true,false}\}$ , where  $d$  is the dimension of the feature vectors  $\mathbf{x}_v^L$ . Hence, a Boolean node classifier based on an  $\mathbf{R}^2$ -GNN  $\mathcal{M}$  proceeds in three steps: (1) encode the input multi-relational graph  $G$  as described above, (2) apply the  $\mathbf{R}^2$ -GNN, and (3) apply  $CLS$  to the output of the  $\mathbf{R}^2$ -GNN. This produces a true or false value for each node of  $G$ . In what follows, we abuse the language and represent a family of GNN-based Boolean node classifiers by the name of the corresponding GNN architecture; for example,  $\mathbf{R}^2$ -GNN is the set of all  $\mathbf{R}^2$ -GNN-based Boolean node classifiers.
2.3 Logic  $\mathcal{FOC}_2$  Formulas
In this paper, we focus on the logic  $\mathcal{FOC}_2$ , a fragment of first-order logic that only allows formulas with at most two variables, but in turn permits to use counting quantifiers. Formally, given two finite sets  $P_1$  and  $P_2$  of unary and binary predicates, respectively, a  $\mathcal{FOC}_2$  formula  $\varphi$  is inductively defined according to the following grammar:
$$
\varphi := A (x) \mid r (x, y) \mid \varphi \wedge \varphi \mid \varphi \vee \varphi \mid \neg \varphi \mid \exists^ {\geq n} y (\varphi) \text {w h e r e} A \in P _ {1} \text {a n d} r \in P _ {2} \tag {3}
$$
where  $x / y$  in the above rules can be replaced by one another. But please note that  $x$  and  $y$  are the only variable names we are allowed to use (Though we can reuse these two names). In particular, a  $\mathcal{FOC}_2$  formula  $\varphi$  with exactly one free variable  $x$  represents a Boolean node classifier for multi-relational graphs as follows: a node  $v$  is assigned to true iff the formula  $\varphi_v$  obtained by substituting  $x$  by  $v$  is satisfied by the (logical) model represented by the multi-relational graph. Similarly as the GNN-based Boolean node classifiers, in what follows, we abuse the language and represent the family of  $\mathcal{FOC}_2$  Boolean node classifiers by its name  $\mathcal{FOC}_2$ .
2.4 Inclusion and Equality Relationships
In this paper, we will mainly talk about inclusion/non-inclusion/equality/strict-inclusion relationships between different node classifier families on certain graph classes. To avoid ambiguity, we give formal definitions of these relationships here. These definitions are all quite natural.
Definition 2. For any two sets of node classifier  $A, B$ , and graph class  $\mathcal{G}$ , We say:
-  $A \subseteq B$  on  $\mathcal{G}$ , iff for any node classifier  $a \in A$ , there exists some node classifier  $b \in B$  such that for all graph  $G \in \mathcal{G}$  and  $v \in V(G)$ , it satisfies  $a(G, v) = b(G, v)$  (Namely,  $a$  and  $b$  evaluate the same for all instances in  $\mathcal{G}$ ). It implies  $B$  is more expressive than  $A$  on  $\mathcal{G}$ .
-  $A \nsubseteq B$  on  $\mathcal{G}$ , iff the above condition in item 1 doesn't hold.
-  $A \subsetneq B$  on  $\mathcal{G}$ , iff  $A \subseteq B$  but  $B \nsubseteq A$ . It implies  $B$  is strictly more expressive than  $A$  on  $\mathcal{G}$ .
-  $A = B$  on  $\mathcal{G}$ , iff  $A \subseteq B$  and  $B \subseteq A$ . It implies  $A$  and  $B$  has the same expressivity on  $\mathcal{G}$ .
The relationship between first-order logic and the Weisfeiler-Lehman test was initially established by Cai et al. [1989]. Subsequently, more recent works such as Xu et al. [2018], have connected the Weisfeiler-Lehman test with expressivity of GNN. This line of research has been followed by numerous studies, including Maron et al. [2020], which explore the distinguishability of GNNs using the Weisfeiler-Lehman test technique. In particular, Barceló et al. [2020] introduced the calibration of logical expressivity in GNN-based classifiers and proposed a connection between  $\mathcal{FOC}_2$  and  $\mathbf{R}^2$ -GNN in single-relational scenario. This led to the emergence of related works, such as Huang et al. [2023], Geerts and Reutter [2021], and Qiu et al. [2023], all of which delve into the logical expressivity of GNNs. Moreover, the theoretical analysis provided in Gao and Ribeiro [2022] has inspired us to extend our results to temporal graph scenarios.
Figure 1: Multi-edge graphs  $G_{1}$  and  $G_{2}$ , and a  $\mathcal{FOC}_2$  formula  $\varphi(x)$  that distinguishes them;  $\varphi(x)$  evaluates node  $a$  in  $G_{1}$  to true and node  $a$  in  $G_{2}$  to false.
4 Logic expressiveness of  $\mathbf{R}^2$ -GNN in multi-relational graphs
Our analysis begins with the observation that certain Boolean classifiers can be represented as  $\mathcal{FOC}_2$  formulas, but remain beyond the expressiveness of any  $\mathbb{R}^2$ -GNN (and consequently, any R-GNN or R-GCN). An illustrative example of this distinction is provided in Figure 1. In this example, we make the assumption that  $P_{1}$  is empty, thereby ensuring that all nodes in both  $G_{1}$  and  $G_{2}$  possess identical initial feature vectors. Additionally,  $P_{2}$  is defined to comprise precisely two relations, namely,  $p_1$  and  $p_2$ . It is evident that no  $\mathbb{R}^2$ -GNN can distinguish the node  $a$  in  $G_{1}$  from node  $a$  in  $G_{2}$  - that is, when an  $\mathbb{R}^2$ -GNN performs the neighbour-based aggregation, it cannot distinguish whether the  $p_1$ -neighbour of  $a$  and the  $p_2$ -neighbour of  $a$  are the same. Moreover, the global readout aggregation cannot help in distinguishing those nodes because all nodes have the same feature vector.
We proceed to formalize this intuition and, in the reverse direction, offer a corresponding result. We demonstrate that there exist Boolean classifiers that fall within the scope of  $\mathbf{R}^2$ -GNN but elude capture by any  $\mathcal{FOC}_2$  formula.
Proposition 3.  $\mathcal{FOC}_2\not\subseteq R^2\text{-GNN}$  and  $R^2\text{-GNN}\not\subseteq \mathcal{FOC}_2$  on some universal graph class.
We prove Proposition 3 in the Appendix. Here, we give some intuition about the proof. The first result is proved using the example shown in Figure 1, which we have already discussed. To show  $\mathbf{R}^2$ -GNN  $\nsubseteq \mathcal{FOC}_2$ , we construct a classifier  $c$  which classifies a node into true iff the node has a larger number of  $r_1$ -type neighbors than that of  $r_2$ -type neighbors. We can prove that we can easily construct an  $\mathbf{R}^2$ -GNN to capture  $c$ . However, for  $\mathcal{FOC}_2$ , this cannot be done, since we can only use counting quantifiers expressing that there exist at most or at least a specific number of neighbors connected via a particular relation, but our target classifier requires comparing indefinite numbers of neighbors via two relations. Thus, we proceed by contradiction, assume that there exists a  $\mathcal{FOC}_2$  classifier equivalent to  $c$ , and then find two large enough graphs with nodes that cannot be distinguished by the classifier (but can be distinguished by  $c$ ).
In some real-world applications, it is often possible to find an upper bound on the size of any possible input graph or to ensure that any input graph will contain at most one relation between every two nodes. For this reason, we next present restricted but positive&practical expressiveness results on bounded and simple graph classes.
Theorem 4.  $\mathcal{FOC}_2\subseteq R^2$  -GNN on any simple graph class, and  $\mathcal{FOC}_2\subsetneq R^2$  -GNN on some simple graph class.
The key idea of the construction is that we will first transform the  $\mathcal{FOC}_2$  formula into a new form which we call relation-specified  $\mathcal{FOC}_2$  (an equivalent form to  $\mathcal{FOC}_2$ , see more details in our Appendix), and then we are able to construct an equivalent  $\mathbb{R}^2$ -GNN inductively over the parser tree of the transformed formula.
Having Theorem 4, one may wonder about the inclusion relationship of  $\mathbf{R}^2$ -GNN and  $\mathcal{FOC}_2$  in the backward direction. In Proposition 3, we showed that for arbitrary universal graph classes, this inclusion relationship fails. However, given a bounded graph class, we can show that for each  $\mathbf{R}^2$ -GNN Boolean node classifier, one can write an equivalent  $\mathcal{FOC}_2$  classifier. An intuition about why this is the case is that all graphs in a bounded graph class will have at most  $n$  constants, for some known  $n \in \mathbb{N}$ , so for each  $\mathbf{R}^2$ -GNN classifier, we can construct an equivalent  $\mathcal{FOC}_2$  classifier with a finite number of sub-formulas to recover the features obtained at different layers of  $\mathbf{R}^2$ -GNN.
Theorem 5.  $R^2$ -GNN  $\subseteq$ $\mathcal{FOC}_2$  on any bounded graph class, and  $R^2$ -GNN  $\subsetneq$ $\mathcal{FOC}_2$  on some bounded graph class.
Figure 2: Graph Transformation.
Combining Theorem 4 and Theorem 5, we have the following corollary.
Corollary 5.1.  $R^2$ - $GNN = \mathcal{FOC}_2$  on any bounded simple graph class.
At Last, one may be curious about the complexity of logical classifier in Theorem 5. Here we can give a rather loose bound as follows:
Theorem 6. For any bounded graph class  $\mathcal{G}_b$ . Suppose any  $G \in \mathcal{G}_b$  has no more than  $N$  nodes, and  $\mathcal{G}_b$  has unary predicate set  $P_1$  and relation (binary predicate) set  $P_2$ . Let  $m_1 := |P_1|, m_2 := |P_2|$ , then for any node classifier  $c$ , suppose  $c$  can be represented as an  $R^2$ -GNN with depth (layer number)  $L$ , then by Theorem 5 there is a  $\mathcal{FOC}_2$  classifier  $\varphi$  equivalent to  $c$  over  $\mathcal{G}_b$ , and the following hold:
- The quantifier depth of  $\varphi$  is no more than  $L$ .
- The size of  $\varphi$  (quantified by the number of nodes of  $\varphi$ 's parse tree) is no more than  $2^{2f(L)}$ , where  $f(L) := 2^{2(N+1)f(L-1)}$ ,  $f(0) = O(2^{2(m_1 + m_2)})$ .