SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fabian Jogl
Machine Learning Research Unit
Center for Artificial Intelligence and Machine Learning
TU Wien, Vienna, Austria
fabian.jogl@tuwien.ac.at
Maximilian Thiessen
maximilian.thiessen@tuwien.ac.at
Thomas Gartner
thomas.gaertner@tuwien.ac.at
We systematically investigate graph transformations that enable standard message passing to simulate state-of-the-art graph neural networks (GNNs) without loss of expressivity. Using these, many state-of-the-art GNNs can be implemented with message passing operations from standard libraries, eliminating many sources of implementation issues and allowing for better code optimization. We distinguish between weak and strong simulation: weak simulation achieves the same expressivity only after several message passing steps while strong simulation achieves this after every message passing step. Our contribution leads to a direct way to translate common operations of non-standard GNNs to graph transformations that allow for strong or weak simulation. Our empirical evaluation shows competitive predictive performance of message passing on transformed graphs for various molecular benchmark datasets, in several cases surpassing the original GNNs.
We systematically investigate which variants of non-standard message passing graph neural networks can be implemented with standard message passing graph neural networks (MPNNs) on a transformed graph without losing expressivity. While MPNNs are a very popular type of graph neural networks (GNNs), they are not able to represent every function on the space of graphs [Xu et al., 2019, Morris et al., 2019] due to their limited expressivity: for all MPNNs, there are pairs of non-isomorphic graphs which always have the same embedding. A common approach to create more expressive GNNs is to change the message passing function of MPNNs. If a GNN is more expressive than MPNNs by adapting the message passing function, we call this non-standard message passing. Examples of this are message passing variants that operate on subgraphs [Frasca et al., 2022, Bevilacqua et al., 2021] or tuples of nodes [Morris et al., 2019, 2020, 2022]. It is known that some variants of non-standard message passing can be implemented with MPNNs on a transformed graph without losing expressivity as demonstrated by Morris et al. [2019, 2020, 2022] and Qian et al. [2022]. We formalize this as an MPNN simulating a GNN: a GNN can be simulated if there exists a graph transformation such that combining an MPNN with that graph transformation is at least as expressive as the GNN. As simulation only requires graph transformations as pre-processing, it allows the use of off-the-shelf MPNN implementations which simplifies and speeds up the implementation of GNNs. Another advantage is that simulation is programming framework agnostic: only the graphs needs to be transformed—independently of how the MPNN is implemented. Finally, simulation allows to easily exchange GNNs in existing workflows without requiring changes to the model. Despite
the benefits of simulation, this approach has not been thoroughly investigated: there is currently no formal definition of simulation and it is not known which GNNs can be simulated. In this paper, we define simulation and provide sufficient criteria for non-standard message passing to be simulated.
Related Work. Xu et al. [2019] and Morris et al. [2019] proved that MPNNs have limited expressivity. This lead to the development of new GNNs that have higher expressivity than MPNNs. Such GNNs often operate on structures that differ from graphs, for example (1) Morris et al. [2019, 2020, 2022] proposed GNNs operating on  $k$ -tuples of nodes, (2) Bodnar et al. [2021a,b] proposed GNNs on topological structures, and (3) Bevilacqua et al. [2021] proposed GNNs operating on subgraphs. The idea of implementing non-standard message passing by standard message passing dates back to at least Otto [1997] who showed that instead of performing the higher-order message passing of  $k$ -WL it is possible to use 1-WL (classical message passing) on a transformed structure [Grohe et al., 2021]. To the best of our knowledge, the first GNN that has been implemented through a graph transformation is  $k$ -GNN [Morris et al., 2019], which is a generalization from  $k$ -WL to GNNs. Already Morris et al. [2019] refer to this concept as simulation. In follow up work, Morris et al. [2020] and Morris et al. [2022] implemented similar GNNs with non-standard message passing as MPNNs together with graph transformations. Similarly,  $k$ -OSWL [Qian et al., 2022] was designed to utilize WL. However, none of these papers have formalized the idea of simulation for GNNs (or WL) explicitly. To this end, we [Jogl et al., 2022a,b] showed that CW Networks [Bodnar et al., 2021a], DS [Bevilacqua et al., 2021], DSS [Bevilacqua et al., 2021], and  $\delta$ - $k$ -GNNs/WL [Morris et al., 2020] can be implemented as an MPNN on a transformed graph. A similar high-level idea was proposed in a positional paper by Velicković [2022]. In parallel, Hajij et al. [2023] introduced combinatorial complexes that generalize structures such as graphs or cell complexes and showed that computations on these complexes can be realized as message passing on graphs. While simulation has been used and advocated in the past there is no unified solution to obtaining graph transformations, theoretical justification, or thorough empirical evaluation. We fill this gap and find many GNNs can be simulated by MPNNs on transformed graphs. More details can be found in Appendix A and Appendix B.
Our Approach. We introduce simulation of non-standard message passing, which formalizes the idea of implementing a GNN, which uses non-standard message passing, through an MPNN on a transformed graph. A message passing algorithm can be strongly simulated if an MPNN together with a graph transformation can achieve the same expressivity in every iteration of message passing. To prove that many GNNs can be strongly simulated, we define the class of augmented message passing (AMP) algorithms which contains many common GNNs. We prove that all AMP algorithms can be strongly simulated and present a meta algorithm to generate the necessary graph transformations. This allows us to show that eleven recently proposed GNNs can be strongly simulated. Specifically, the GNNs by Morris et al. [2019, 2020, 2022] and Qian et al. [2022] perform AMP. This gives an additional mathematical justification to their approach by proving that their GNNs can be implemented with MPNNs without losing expressivity over an implementation with non-standard message passing.
Furthermore, we investigate three constructions that demonstrate the limits of strong simulation: time dependent neighborhoods, nested aggregations, and non-pairwise message passing. We prove that these constructions either cannot be strongly simulated efficiently or cannot be strongly simulated at all. However, if we are only interested in the MPNN achieving the same expressivity as the GNN with non-standard message passing after a sufficient number of layers, it is possible to implement all three constructions with an MPNN. We call this weak simulation: a message passing algorithm can be weakly simulated if there exists a non-negative integer  $\zeta$  such that an MPNN together with a graph transformation can achieve the same expressivity as the message passing algorithm in one iteration after every  $\zeta$  iterations of message passing. We show that Message Passing Simplicial Networks [Bodnar et al., 2021b], CW Networks [Bodnar et al., 2021a], DSS [Bevilacqua et al., 2021], and  $K$ -hop message passing [Feng et al., 2022] can be weakly simulated. Finally, we evaluate a representative set of graph transformation empirically. Our graph transformations lead to competitive performance compared to the simulated algorithms and often lead to more accurate predictions.
Main Contributions. We introduce the concept of strong and weak simulation (Section 3) of message passing. This generalizes existing ideas behind the implementation of several GNNs [Morris et al., 2019, 2020, 2022, Qian et al., 2022]. We provide an automated way of proving that a GNN can be simulated and deriving the necessary graph transformations. We prove that there exist architectures that cannot be strongly simulated (Section 4) but only weakly simulated (Section 5). Our empirical evaluation (Section 6) demonstrates that simulation achieves competitive performance.
A graph  $G$  is a triple  $G = (V, E, F)$  consisting of a set of vertices  $V$ , a set of directed edges  $E \subseteq \{(x, y) \mid x, y \in V, x \neq y\}$ , and a function  $F$  that assigns a feature to every vertex and edge. For a set of objects  $U$  and integer  $l > 0$ , we denote by  $2^U = \{X \mid X \subseteq U\}$  the powerset of  $U$ , i.e., the set of all subsets of  $U$ , and by  $U^l = \{(u_1, \ldots, u_l) \mid u_1, \ldots, u_l \in U\}$  the set of all tuples of length  $l$  built from  $U$ . In this paper we work on a generalization of graphs we call relational structures that use a generalized notion of neighborhood. For a set of objects  $U$ , we call any function  $\mathcal{N}: U \to 2^{(U^\ell)}$  a neighborhood function. A neighborhood function assigns to every object a set of tuples of length  $\ell$  which we call its neighbors. We use  $\ell(\mathcal{N})$  to denote the length of these tuples. We call any tuple  $(w, u)$  a directed edge where  $u \in U$  and  $w \in \mathcal{N}(u)$  (the in-neighbours) with  $\ell(\mathcal{N}) = 1$ . For an integer  $k > 0$ , we use  $[[k]]$  to denote the set  $\{1, \ldots, k\}$  and  $\{\{\cdot\}\}$  to denote a multiset.
Definition 2.1. (Relational Structure) Let  $k \geq 0$  be an integer,  $U$  be a set of objects, and let  $\mathcal{N}_1, \ldots, \mathcal{N}_k$  be neighborhood functions over  $U$ . Furthermore, let  $F$  be a function such that  $F(u)$  assigns a feature to every  $u \in U$  and  $F^i((x, u))$  assigns a feature to every directed edge with  $i \in [[k]]$  and  $x \in \mathcal{N}_i(u)$  with  $\ell(\mathcal{N}_i) = 1$ . Then, the tuple  $X = (U, \mathcal{N}_1, \ldots, \mathcal{N}_k, F)$  is a relational structure.
If it is clear from context from which neighborhood  $\mathcal{N}_i$  an edge  $(x,u)$  is, we will simply write its features as  $F((x,u))$  instead of  $F^i ((x,u))$ . If the number of neighborhoods is important we call such a structure a  $k$ -relational structure. Note, that a graph  $G = (V,E,F)$  is a special case of a 1-relational structure with only one neighborhood  $\mathcal{N}_G(v) = \{w\mid (w,v)\in E_G\}$ , i.e.,  $G = (V,\mathcal{N}_G,F)$ . For graphs  $G,H$  we refer to their vertices, edges, and features by adding the graph as a subscript. We say that  $G,H$  are isomorphic if there exists an edge and feature preserving bijective function  $\alpha :V_{G}\to V_{H}$  between their vertices: for every vertex  $v\in V_{G}$  it holds that  $F_{G}(v) = F_{H}(\alpha (v))$  and for every  $x,y\in V_{G}$ , it holds that  $(x,y)\in E_G$  if and only if  $(\alpha (x),\alpha (y))\in E_H$  with  $F_{G}((x,y)) = F_{H}((\alpha (x),\alpha (y)))$ . No polynomial time algorithm is known which
Figure 1: Left: graph. Center: regular cell complex built from the graph through a graph-to-structure encoding [Bodnar et al., 2021a]. Vertices correspond to 0-dimensional cells, edges to 1-dimensional cells (yellow) and induced cycles to 2-dimensional cells (blue). Right: a graph created by structure-to-graph encoding the regular cell complex to a graph. Vertices corresponds to cells as indicated by color.
decides whether two graphs are isomorphic [Babai, 2016]. However, heuristics such as those based on colorings and color update functions allow us to distinguish many pairs of non-isomorphic graphs. Furthermore, they allow us to model the ability of GNNs to distinguish graphs [Xu et al., 2019, Morris et al., 2019]. For a relational structure over a set of objects  $U$ , a coloring is a function  $c: U \to \chi$  where  $\chi$  is a known set of colors. Suppose we have two disjoint sets  $U$  and  $U'$  with two colorings  $c$  over  $U$  and  $c'$  over  $U'$ . Then, we define the joint coloring  $c \cup c'$  for every  $x \in (U \cup U')$  as  $c(x)$  if  $x \in U$  and  $c'(x)$  otherwise. For two colorings  $c$  and  $d$  defined over the same set of objects  $U$ , we say that  $c$  refines  $d$  if for every pair of objects  $u, r \in U$  it holds that  $c_u = c_r \Rightarrow d_u = d_r$ . Let  $U, U'$  be sets of objects with  $U' \subseteq U$ . Let  $c$  be a coloring of  $U$  and  $d$  a coloring of  $U'$ . Then,  $c$  refines  $d$  if for every pair of objects  $u, r \in U'$  it holds that  $c_u = c_r \Rightarrow d_u = d_r$ .
Definition 2.2. (Color Update Function) Let  $k \geq 1$  be an integer and  $\mathcal{X}$  a set of  $k$ -relational structures. A color update function takes any  $X = (U, \mathcal{N}_1, \ldots, \mathcal{N}_k, F) \in \mathcal{X}$ , a coloring  $c$  of  $U$ , and a  $u \in U$  and outputs a new color  $\phi(u, X, c)$  of  $u$ . We denote the new coloring of the whole set  $U$  as  $\phi(X, c)$ .
We assume color update functions  $\phi$  to be computable and denote the time to compute  $\phi$  with respect to an input  $U$  as  $\tau_{\phi}(|U|)$ . Color update functions are used to iteratively update an initial coloring  $c^0$  such that  $c^{t + 1} = \phi (X,c^t)$  for every  $t\geq 0$ . We denote  $t$  applications of  $\phi$  to compute  $c^t$  as  $\phi^t (X,c^0)$ . We also say that  $\phi^0 (X,c^0) = c^0$ . The initial coloring is often given by the features  $c^0 = F|_U$ , meaning the color  $c^0$  is given by the domain restriction of  $F$  to the set of objects  $U$ . Then, the color of each object  $u\in U$  is  $c_u^0 = F_u$ . If we are not given features, then we use a constant coloring, i.e., a coloring that assigns the same (arbitrary) color to each object. Color update functions can sometimes
be used to determine whether two graphs are isomorphic. For this, a coloring is computed for both graphs. If at some iteration the colorings of the two graphs are different this implies that the graphs are not isomorphic. If the colorings are never different, then the result of this method is inconclusive. A common color update function is the Weisfeiler-Leman algorithm (WL). In this work we define WL to use edge features.
Definition 2.3. (WL) The Weisfeiler-Leman algorithm (WL) is a color update function operating on a graph  $G = (V, E, F)$  defined as  $\mathrm{WL}(v, G, c) = \mathrm{HASH}(c_v, \{(c_u, F((u, v)) \mid u \in \mathcal{N}_G(v)\})$  for  $v \in V$  where  $\mathrm{HASH}$  is an injective mapping to colors.
Message passing graph neural networks (MPNNs) can be seen as a generalization of WL where the colors correspond to learned node embeddings and each layer of the MPNN corresponds to a color update function. The color update function in layer  $t \geq 1$  of an MPNN is defined as  $\phi(v, G, c) = \mathrm{COMB}^t(c_v, \mathrm{AGG}^t(\{(c_x, F((x, v)) \mid x \in \mathcal{N}_G(v)\}))$  where  $\mathrm{COMB}^t$  and  $\mathrm{AGG}^t$  are learnable functions, e.g., multi-layer perceptrons (MLPs). We refer to WL and MPNNs as standard message passing. It has been shown that MPNNs are at most as expressive as WL, i.e., it can only distinguish non-isomorphic graphs that WL can distinguish. As there exist pairs of graphs that WL cannot distinguish this means that there exists graphs that MPNNs cannot distinguish either [Xu et al., 2019, Morris et al., 2019]. A common approach to improve the expressivity of standard message passing is to apply a function  $T$  that transforms graphs to other relational structures. We call this mapping  $T$  from a graph to a relational structure a graph-to-structure encoding. An example of an graph-to-structure encoding can be seen in Figure 1. This mapping  $T$  is combined with a color update function  $\phi$  tailored to this new structure. We use non-standard message passing to refer to color update functions that operate on relational structures that are not graphs. Some examples of non-standard message passing are  $k$ -WL [Immerman and Lander, 1990] which operates on  $k$ -tuples of nodes and CW Networks [Bodnar et al., 2021a] which operate on regular cell complexes.
3 Strong Simulation
We show that standard message passing together with graph transformations can achieve the same expressivity as many algorithms with non-standard message passing. As standard message passing operates on graphs we need to map relational structures to graphs. Note that merely guaranteeing at least the same expressivity as a color update function  $\phi$  in each iteration can easily be achieved by, e.g., using the final coloring of  $\phi$  as node features. To avoid such a trivial solution, we enforce straightforward restrictions on the generated graphs.
Definition 3.1. (Structure-to-graph encoding) Let  $R$  be a mapping  $R: X \mapsto G$  that maps relational structures  $X = (U, \mathcal{N}_1, \ldots, \mathcal{N}_k, F)$  to graphs  $G = (V, E, F')$ . We call  $R$  a structure-to-graph encoding if it can be written as  $R(X) = R_{\mathrm{feat}}(R_{\mathrm{graph}}(U, \mathcal{N}_1, \ldots, \mathcal{N}_k), F)$  such that
1.  $R_{\mathrm{graph}}$  maps a relational structure without features to a graph  $G = (V, E, F_{\mathrm{graph}})$ .
2.  $R_{\mathrm{feat}}((V, E, F_{\mathrm{graph}}), F) = (V, E, F')$  creates the features  $F'$  by concatenating each (node or edge) feature from  $F_{\mathrm{graph}}$  with the corresponding feature from  $F$  if it exists.
As a graph-to-structure encoding  $T$  maps graphs to structures and a structure-to-graph encoding  $R$  maps structures to graphs, this means that  $R \circ T$  maps graphs to graphs. We call such functions  $R \circ T$  graph transformations, an example can be seen in Figure 1. As we define relational structures over sets of objects, this implies that  $R$  is permutation equivariant with respect to a permutation of the objects. Furthermore, since  $R$  is a function it is also deterministic. Next, we define strong simulation of color update functions.
Definition 3.2. (Strong Simulation) Let  $\phi$  be a color update function. Let  $R$  be a structure-to-graph encoding that runs in  $\mathcal{O}(\tau_{\phi}(|U|))$  for every relational structure with object set  $U$  and creates a graph with vertex set  $V \supseteq U$ . We consider two arbitrary relational structures from the domain of  $\phi$ , say  $X = (U, \mathcal{N}_1, \ldots, \mathcal{N}_k, F)$  and  $X' = (U', \mathcal{N}_1', \ldots, \mathcal{N}_k', F')$ . Let  $(V_1, E_1, F_1) = R(X)$  and  $(V_2, E_2, F_2) = R(X')$ . We say  $\phi$  can be strongly simulated under  $R$  if for every  $t \geq 0$  it holds that  $\mathrm{WL}^t(R(X), F_1|_{V_1}) \cup \mathrm{WL}^t(R(X'), F_2|_{V_2})$  refines  $\phi^t(X, F|_U) \cup \phi^t(X', F'|_{U'})$ .
<sup>1</sup>Here, we define MPNNs to use edge features which practically is almost always the case.
Intuitively, a color update function  $\phi$  can be strongly simulated if instead of running  $t$  iterations of  $\phi$  on relational structure  $X$  we can instead run WL on the graph  $R(X)$  for  $t$  iterations and achieve at least the same expressivity in every iteration. Strong simulation is a stronger property than expressivity: strong simulation implies being at least as expressive in every iteration whereas expressivity usually refers only to the final coloring. This guarantees that WL on  $R(X)$  is always at least as expressive as  $\phi$  and not just in some iteration. There exist MPNNs that are as expressive as WL in every iteration [Xu et al., 2019]. Thus, every layer of an MPNN can be as expressive as the corresponding iteration of WL. Note that as long as we restrict the size of the graphs (the number of vertices), strong simulation allows to learn a color update function  $\phi$ , e.g., a GNN, exactly. Indeed, if we combine each layer of an MPNN with a large enough MLP (with appropriate activation), we can learn to map the MPNN's embeddings to the output of  $\phi$ . This problem of fitting a finite number of points exactly is well studied and known as memorization, see e.g. Yun et al. [2019] for an overview.
To strongly simulate a large number of GNNs we introduce a class of color update functions we call augmented message passing (AMP). AMP is designed to capture many GNNs and we prove that AMP can be strongly simulated. AMP makes use of a building block we call atoms. Atoms model operations of color update functions such as colors, constants, or features that are focused on single vertices or pairs of vertices. AMP extends atoms with more general operations that are not necessarily focused on a single vertex such as function applications or aggregations over neighborhoods.
Definition 3.3. (Atom). Let  $X = (U, \mathcal{N}_1, \ldots, \mathcal{N}_k, F)$  be a relational structure. Let  $v, w \in U$  and  $c_v, c_w$  be the colors of  $v$  and  $w$ . Then, an atom  $\rho(v, c_v, w, c_w)$  is a function that has exactly one of the following four forms:
(A1)  $\rho(v, c_v, w, c_w) = k(v, w)$  for a constant  $k(v, w)$  that only depends on  $v$  and  $w$  but not on the colors,
(A2)  $\rho(v, c_v, w, c_w) \in \{F(v), F(w), F((w, v))\}$  where  $F((w, v))$  is only part of this set if the edge  $(w, v)$  is part of at least one neighborhood of  $X$ ,
(A3)  $\rho (v,c_v,w,c_w)\in \{c_v,c_w\}$  , or
(A4)  $\rho (v,c_v,w,c_w) = (\rho_1(v,c_v,w,c_v),\ldots ,\rho_m(v,c_v,w,c_w))$  with atom  $\rho_{i}$  for all  $i\in [[m]]$
Definition 3.4. (Augmented Message Passing). Let  $X = (U, \mathcal{N}_1, \dots, \mathcal{N}_k, F)$  be a relational structure with  $\ell(\mathcal{N}_i) = 1$  for all  $i \in [[k]]$ . We call a color update function  $\phi$  augmented message passing (AMP) if for all  $u \in U$  and coloring  $c$  of  $X$ , it can recursively be defined as exactly one of the following four forms:
(S1)  $\phi (u,X,c) = \rho (u,c_u,u,c_u)$  where  $\rho$  is an atom,
(S2)  $\phi (u,X,c) = \{\{\rho (u,c_u,w,c_w)\mid w\in \mathcal{N}_i(u)\} \}$  where  $\rho$  is any atom and any  $i\in [[k]]$
(S3)  $\phi (u,X,c) = (\phi_1(u,X,c),\ldots ,\phi_m(u,X,c))$  where all  $\phi_1,\dots ,\phi_m$  are AMP, or
(S4)  $\phi (u,X,c) = f\left(\phi '(u,X,c)\right)$  where  $f$  maps colors to colors and  $\phi^{\prime}$  is AMP.
Having defined AMP, we next prove that AMP can be strongly simulated by providing a structure-to-graph encoding for every color update function from AMP. Full proofs are in Appendix C.
Theorem 3.5. Augmented message passing can be strongly simulated.
Proof sketch. We define augmented message encoding (AME) which returns a structure-to-graph encoding for every given AMP algorithm. Let  $\phi$  be AMP that operates on a relational structure  $X = (U, \mathcal{N}_1, \ldots, \mathcal{N}_k, F)$ .  $\mathrm{AME}_{\phi}(X)$  returns a graph  $G$  with vertex set  $V_G = U$ . Any constant value atom (A1) in  $\phi$  and any feature in  $F$  is stored in the vertex or edge features of  $G$ . If messages are passed between two objects then the graph will have an edge between the corresponding vertices with an edge feature encoding the neighborhood. We prove that every AMP  $\phi$  can be strongly simulated under the structure-to-graph encoding  $\mathrm{AME}_{\phi}$ .
AMP does not only contain variants of WL, but also GNN layers such as the layer of GSN [Bouritsas et al., 2022]. We say a GNN can be strongly simulated, if all of its layers correspond to color update functions that can be strongly simulated by the same structure-to-graph encoding  $R$ . This means
Figure 2: Left: nested aggregations. Right: non-pairwise message passing of 2-tuples. In both examples the state of two vertices gets combined and sent to the blue vertex. Note that for non-pairwise aggregations we have an order on the nodes in a tuple whereas for nested aggregations the aggregated vertices are unordered.
Figure 3: WL on the graph on the left (right) side weakly simulates the nested aggregations (non-pairwise message passing) from Figure 2 by adding additional vertices (yellow). This shows how weak simulation of both constructions is done in a similar way with the difference being that for non-pairwise aggregation we encode the order in the tuples on the edges.