SlowGuess/NeurIPS_2023_OCR huggingface dataset:
that for every relational structure  $X$  and every  $t \geq 0$ , WL achieves at least the same expressivity in iteration  $t$  on  $R(X)$  as layer  $t$  of the GNN on  $X$ . We prove that this is always the case for color update functions that only differ in their use of function applications (S4) (see Appendix C.3). Different layers of a GNN commonly only differ by function applications (S4) as these correspond to learned functions. As many GNN layers use AMP, we can state one of the main results of this paper.
Corollary 3.6. The following GNNs and variants of WL can be strongly simulated:
1.  $VV_{C}$ -GNN [Sato et al., 2019]
2.  $k$ -WL / GNN [Morris et al., 2019]
3.  $\delta -k$  (L)WL / (L)GNN [Morris et al., 2020]
5. GSN-e and GSN-v [Bouritsas et al., 2022]
4.  $(k,s)$ -LWL / SpeqNet [Morris et al., 2022]
6. DS-WL / GNN [Bevilacqua et al., 2021]
7.  $k$ -OSWL / OSAN [Qian et al., 2022]
8.  $M_{k}$  GNN [Papp and Wattenhofer, 2022]
9. GMP [Wijesinghe and Wang, 2022]
10. Shortest Path Networks [Abboud et al., 2022]
11. Generalized Distance WL [Zhang et al., 2023]
Proof sketch. The proof consists of showing that the color update function underlying the non-standard message passing is AMP. By Theorem 3.5 this implies that the color update function can be simulated.  $\square$
Runtime Complexity. AME creates a vertex for every object and edges for every pair of objects that exchange messages. This implies that, creating the encoding requires the same amount of time as one iteration of AMP. Furthermore, classical message passing on the transformed graphs passes the same number of messages as AMP in every iteration. Thus, the overall runtime complexity of strongly simulating AMP is equivalent to that of running AMP itself.
Example. We illustrate AMP on the example of WL with triangle counts and without edge features. For every  $v \in V_G$ , we denote by  $\triangle_v$  the number of triangles in the graph containing  $v$ . We define the color update function of WL with triangle counts:  $\phi(v, G, c) = (c_v, \{(c_u, \triangle_u) \mid u \in \mathcal{N}_G(v)\})$ . We show that WL with triangle counts is AMP. Consider the tuple  $(c_u, \triangle_u)$ , this consists of a color  $c_u$  atom (A3) with a constant  $\triangle_u$  atom (A1) combined into a tuple atom (A4). Thus,  $(c_u, \triangle_u)$  is an atom which means that  $\{(c_u, \triangle_u) \mid u \in \mathcal{N}_G(v)\}$  is AMP that aggregates this atom over the neighborhood  $\mathcal{N}_G$  (S2). Finally, this AMP is combined with the color atom  $c_v$  (A1) and combined into a tuple AMP (S3). Thus, the color update function of WL with triangle counts performs AMP which implies that it can be strongly simulated. For this it suffices to attach  $\triangle_v$  to the features of every vertex  $v \in V$ .
4 Limits of Strong Simulation
To show the limits of strong simulation we identify three constructions that either cannot be strongly simulated or cannot be strongly simulated efficiently: time dependent neighborhoods, nested ag-
gregations, and non-pairwise message passing. These constructions have previously been used by Deac et al. [2022], Bevilacqua et al. [2021], Feng et al. [2022], Bodnar et al. [2021b], and Bodnar et al. [2021a]. We begin with time dependent neighborhoods which are used in the alternating message passing of Deac et al. [2022]. We illustrate them via the following example. Consider a color update function that updates the color of a vertex  $a$  based on the color of  $b$  in iteration  $t$  but not in iteration  $t'$ . As it depends on the iteration whether  $a$  and  $b$  are neighbors, we call this time dependent neighborhoods. To model this in a graph, we require an edge between  $a$  and  $b$  in iteration  $t$  or we would lose expressivity whenever the color of  $b$  is important for the coloring of  $a$ . However, this means that the color of  $a$  will also be updated with the color of  $b$  in iteration  $t'$ . Thus, we can strongly simulate time dependent neighborhoods but we cannot do so efficiently (i.e., with the same number of sent messages), as we might need to send many more messages.
Nested aggregations are a form of color update functions used by Bevilacqua et al. [2021] and Feng et al. [2022]. Consider the following color update function from DSS-WL [Bevilacqua et al., 2021]  $\phi(v, X, c) = (c_v, \{\{c_y \mid y \in \mathcal{N}(x)\} \mid x \in \mathcal{N}'(v)\})$  (see Figure 2 left). We call this nesting of two aggregations ( $\{c_{\cdot} \mid \cdot \in \mathcal{N}(\cdot)\}$ ) a nested aggregation. To strongly simulate this we need an edge from every vertex of  $\bigcup_{x \in \mathcal{N}'(v)} \mathcal{N}(x)$  to  $v$ . Furthermore, we need a way to group vertices  $\mathcal{N}(x)$  together for every  $x \in \mathcal{N}'(v)$ . We prove that this is impossible with a structure-to-graph encoding and thus that nested-aggregations cannot be strongly simulated.
The third construction that we cannot strongly simulate is non-pairwise message passing which has been used by Bodnar et al. [2021b,a]. Non-pairwise message passing refers to aggregating colors as ordered tuples over a neighborhood  $\mathcal{N}$  with  $\ell(\mathcal{N}) > 1$ . As an example, suppose we want to update the color of vertex  $v$  based on the ordered tuple of colors  $(c_a, c_b)$  of vertices  $a$  and  $b$  (see Figure 2 right). This is different from the aggregation by WL or AMP as those aggregations are unordered, i.e.,  $\{\{c_a, c_b\}\}$ . An issue arises when multiple such ordered tuples have to be used to update a color, as there is no way this can be done within the concept of strong simulation. Similar to nested aggregations, non-pairwise message passing cannot be strongly simulated. Note that non-pairwise message passing is a special case of nested-aggregations where an additional order on the aggregated objects is given as is demonstrated in Figure 2.
Theorem 4.1. Nested aggregations and non-pairwise message passing cannot be strongly simulated.
Proof sketch. To prove this (in Appendix D), we construct two relational structures that can be distinguished with nested aggregations in one iteration. These two structures have the same adjacencies but different features. We prove that no structure-to-graph encoding can create two graphs that one iteration of WL can distinguish. By the definition of graph-to-structure encoding, it is not possible for the edges of the resulting graph to depend on the original features. Thus, the two relational structure will lead to two graphs that have the same edges. We prove that WL cannot distinguish these two graphs in one iteration. The proof for non-pairwise message passing works similarly.  $\square$
5 Weak Simulation
We have shown in the previous section that not all color update functions can be strongly simulated. In this section, we introduce the more general concept of weak simulation and show that color update functions can be weakly simulated that are impossible to strongly simulate. Weak simulation differs from strong simulation by requiring only the same expressivity after certain number of iterations instead of every iteration.
Definition 5.1. (Weak Simulation) Let  $\phi$  be a color update function. Let  $R$  be a structure-to-graph encoding that runs in  $\mathcal{O}(\tau_{\phi}(|U|))$  for every relational structure with object set  $U$  and creates a graph with vertex set  $V \supseteq U$ . We consider two arbitrary relational structures from the domain of  $\phi$ , say  $X = (U, \mathcal{N}_1, \ldots, \mathcal{N}_k, F)$  and  $X' = (U', \mathcal{N}_1', \ldots, \mathcal{N}_k', F')$ . Let  $(V_1, E_1, F_1) = R(X)$  and  $(V_2, E_2, F_2) = R(X')$ . We say  $\phi$  can be weakly simulated under  $R$  with simulation factor  $\zeta \geq 1$  if for every  $t \geq 0$  it holds that  $\mathrm{WL}^{\zeta \cdot t}(R(X), F_1|_{V_1}) \cup \mathrm{WL}^{\zeta \cdot t}(R(X'), F_2|_{V_2})$  refines  $\phi^t(X, F|_U) \cup \phi^t(X', F'|_{U'})$ .
2We have altered the notation of DSS-WL from the original paper for the sake of simplicity.
The simulation factor  $\zeta$  measures the relative slowdown of weak simulation with respect to the original GNN. It follows that strong simulation is a special case of weak simulation as every strongly simulatable algorithm is weakly simulatable with a simulation factor of 1. Next, we prove that it is possible to weakly simulate non-pairwise aggregations and nested aggregations. This shows that weak simulation contains a larger set of color update functions than strong simulation. The full proof can be found in Appendix E.
Theorem 5.2. Nested aggregations and non-pairwise message passing can be weakly simulated with simulation factor 2.
Proof sketch. For weak simulation of nested aggregations consider the following construction. Suppose a nested aggregation  $\{\{\{c_y^t\mid y\in \mathcal{N}(x)\} \} \mid x\in \mathcal{N}(v)\}\}$ . For every  $x\in \mathcal{N}(v)$  we create incoming edges from  $\mathcal{N}(x)$  and an outgoing edge  $(x,v)$  (see Figure 3 left). This allows for weak simulation with a simulation factor of 2. Non-pairwise message passing can be weakly simulated in a similar way. Suppose we intend to update the color of vertex  $v$  based on the ordered tuple of colors  $(c_a,c_b)$  of vertices  $a$  and  $b$ . Then, we create a vertex  $x$  and an edge  $(x,v)$  for this message. Furthermore, we create an edge  $(a,x)$  labeled with 1 to indicate that it is the first element in the tuple and an edge  $(b,x)$  labeled with 2 (see Figure 3 right). This construction weakly simulates the non-pairwise message passing with simulation factor 2.
As a consequence, we can weakly simulate the following algorithms.
Corollary 5.3. The following GNNs and variants of WL can be weakly simulated:
1. Message Passing Simplicial Networks [Bodnar et al., 2021b]
2. CW Networks [Bodnar et al., 2021a]
3. DSS [Bevilacqua et al., 2021]
4. K-hop message passing and KP-GNNs [Feng et al., 2022]
Discussion. It is possible to design graph transformations that result in smaller graphs, enabling more efficient weak simulation than the transformations automatically following from Theorem 5.2. We demonstrate that this is possible for all algorithms from Corollary 5.3. For Message Passing Simplicial Networks and CW Networks it is possible to construct graphs that require no additional vertices for weak simulation (see Appendix F.1). Similarly, it is possible to weakly simulate DSS without creating a large number of additional edges by adding a second copy of the original graph if we accept increasing the simulation factor to 3 as a trade-off (see Appendix F.2). Interestingly,  $K$ -hop message passing and KP-GNNs induce an additional order on the message passing which even allows for efficient strong simulation (see Appendix F.3). We did not find any architectures where weak simulation led to large increases in time and space complexity. However, in general, we cannot rule out the possibility of architectures that exhibit such an increase. This indicates the possibility for more powerful non-standard message passing, opening a promising direction for future work.
We have shown many cases in which graph transformation based methods achieve the same expressivity as non-standard message passing. In this section, we empirically investigate whether MPNNs together with graph transformations can also achieve similar predictive performance as the simulated GNNs with non-standard message passing. We strongly simulate DS [Bevilacqua et al., 2021], weakly simulate DSS [Bevilacqua et al., 2021] and weakly simulate CW Networks (CWN) [Bodnar et al., 2021a], as representative approaches. For a non-standard message passing algorithm, we denote the corresponding graph transformation with a bar, e.g.,  $\overline{\mathrm{DSS}}$ . The code for our experiments can be found at https://github.com/ocatias/GNN-Simulation.
Models. With GIN [Xu et al., 2019] and GCN [Kipf and Welling, 2017] we use two of the most common MPNNs as baselines. We combine these MPNNs with the graph transformations  $\overline{\mathrm{DS}}$ ,  $\overline{\mathrm{DSS}}$  and  $\overline{\mathrm{CWN}}$ . DS,  $\overline{\mathrm{DS}}$ , DSS, and  $\overline{\mathrm{DSS}}$  require a policy that maps a graph to subgraphs, for this we chose the common 3-egonets policy that extracts the induced 3-hop neighborhood for each node [Bevilacqua et al., 2021]. We chose this policy as it creates only small subgraphs and Bevilacqua
Table 1: Predictive Performance of different GNNs on 10 different datasets. BEATS MPNN is the percentage of datasets on which the model beats the corresponding MPNN (either GIN or GCN, as noted in the model name). BEATS NSMP is the percentage of datasets where the graph transformation based model beats the corresponding non-standard message passing model. BEST MODEL is the percentage of datasets where this model achieves the best results among all models.
BEATS MPNN	BEATS NSMP	BEST MODEL
GIN	-	-	0%
GCN	-	-	0%
DS (GIN)	60%	-	10%
DS + GIN	60%	60%	0%
DS + GCN	70%	30%	0%
DSS (GIN)	70%	-	20%
DSS + GIN	70%	60%	30%
DSS + GCN	70%	30%	10%
CWN	60%	-	20%
CWN + GIN	60%	50%	0%
CWN + GCN	50%	30%	10%
et al. [2021] reported good results for it. More details on  $\overline{\mathrm{DSS}}$  can be found in Appendix F.2. For CWN and  $\overline{\mathrm{CWN}}$  we construct cells as described by Bodnar et al. [2021a]. More details on  $\overline{\mathrm{CWN}}$  can be found in Appendix G and F.1. We apply the same training and evaluation procedure to all GNNs.
Datasets. Due to the large number of combination of models and graph transformations, we focus on medium size datasets. To experimentally investigate how the graph transformations increase expressivity, we perform an initial investigation with GIN on the synthetic CSL dataset [Murphy et al., 2019, Dwivedi et al., 2023]. For real world prediction tasks, we use all real world datasets with less than  $10^{5}$  graphs that provide a train, validation, and test split used in Bodnar et al. [2021a], Bevilacqua et al. [2021]: ZINC [Gómez-Bombarelli et al., 2018, Sterling and Irwin, 2015], ogbg-molhiv and ogbg-moltox21 [Hu et al., 2020]. Additionally, we add seven small molecule datasets from OGB [Hu et al., 2020]. In total, we evaluate on 10 real-life datasets. $^{3}$
Setup. For real-life datasets we combine all baseline models (GCN, GIN) with all graph transformations (DS, DSS, CWN) and tune hyperparameters individually (including CWN, DS and DSS; see Appendix H). We also measure the preprocessing and training speeds for different models (details and speed results are in Appendix H.3).
Results. Table 2 shows the results on CSL. GIN achieves  $10\%$  accuracy whereas all other methods achieve a perfect accuracy. This confirms that just like non-standard message passing, MPNNs with graph transformations have superior expressivity compared to GIN. Table 1 summarizes the results on all real life datasets. Indeed, MPNNs with graph transformations achieve the best result on half of them. Just like non-standard message passing, MPNNs with graph transformations outperform MPNNs in the majority of experiments. Notably, graph transformations combined with GIN outperform non-
standard message passing in more than half of the experiments. This shows that simulated networks not only have provably at least the same expressivity but also perform surprisingly well in practice.
Table 2: Accuracy on CSL. Bold results outperform GIN.
MODEL	CSL
ACCURACY
GIN	0.1 ± 0.0
CWN	1.0 ± 0.0
CWN + GIN	1.0 ± 0.0
DSS	1.0 ± 0.0
DSS + GIN	1.0 ± 0.0
DS	1.0 ± 0.0
DS + GIN	1.0 ± 0.0
<sup>3</sup>We use the following datasets: ZINC with 12k nodes, ogbg-molhiv, ogbg-moltox21, ogbg-molesol, ogbg-molbace, ogbg-molclintosh, ogbg-molbbbp, ogbg-molsider, ogbg-moltoxcast, and ogbg-mollipo.
Discussion. The purpose of simulating a GNN is to allow an MPNN to learn similar embeddings of graphs as the GNN. We expect such embeddings to model the similarity of graphs in a more fine-grained way than just distinguishing graphs. Thus, we have to avoid trivial solutions: any MPNN can be made as expressive as any other GNN when the output of the GNN is attached to the input of the MPNN. In fact, it is even possible to make an MPNN maximally expressive by precomputing the isomorphism class of the graph, e.g., a number uniquely encoding whether two graphs are isomorphic, and inputting it into the MPNN. However, such techniques defeat the purpose of learning a graph representation in the first place and potentially require a super-polynomial runtime [Babai, 2016]. To avoid such scenarios, we introduce two requirements for weak and strong simulation: (1) the features of the resulting graph depend on the original features solely by concatenation and (2) the transformation must operate in linear time relative to the simulated color update function. These two constraints render it impossible to precompute the colors generated by the simulated method for a non-constant number of iterations and prohibit the computation of graph isomorphism classes.
Implications. Our work has implications on (1) the theory of GNNs, (2) the design of new GNNs, and (3) the implementation GNNs with graph transformations. For (1): simulation allows to investigate the expressivity of different GNNs through a unified lens by analyzing the corresponding graph transformation. It should be possible to obtain VC bounds for any weakly simulatable GNN by using the results from Morris et al. [2023]. Similarly, we believe that it is possible to use Geerts and Reutter [2022] to get expressivity upper-bounds in terms of the  $k$ -WL test for any weakly simulatable GNN. For (2): our theorems indicate that nested aggregations and non-pairwise message passing cannot be strongly simulated and are thus fundamentally different from the message passing paradigm. Thus, to build GNNs that go beyond MPNNs in expressivity it seems promising to investigate such constructions. For (3): instead of implementing a GNN with non-standard message passing, our theorems imply it can be implemented as a graph transformation together with an MPNN. This makes it easier to implement GNNs as of-the-shelf MPNNs can be used and makes the resulting method agnostic from the deep learning framework as the graph transformation does most of the heavy lifting.
Conclusion. We introduced weak and strong simulation of non-standard message passing. Simulation allows an MPNN together with a graph transformation to obtain the same expressivity every  $\zeta \geq 1$  layers (weak simulation) or in every layer (strong simulation) as GNNs with non-standard message passing. We have proposed a straightforward way of proving that an algorithm can be simulated and to generate the corresponding graph transformations. This generalizes previous ideas [Otto, 1997, Morris et al., 2019, 2020, 2022, Qian et al., 2022] and makes simulating GNNs accessible to anyone who intends to build novel GNNs. In total, we have shown that 15 GNNs can be simulated. We chose these 15 models to showcase the variety of GNNs that fall under augmented message passing and expect many more GNNs to be simulatable as well. In future research, we will investigate the use of simulation to analyze aspects of GNNs such as generalization, oversmoothing, and oversquashing.
Acknowledgements. Part of this work has been supported by the Vienna Science and Technology Fund (WWTF) project ICT22-059. The authors would like to thank the reviewers for the feedback. We are especially thankful to our colleagues Pascal Welke and Tamara Drucks for giving extensive feedback on a draft of this paper.
R. Abboud, I. I. Ceylan, M. Grohe, and T. Lukasiewicz. The surprising power of graph neural networks with random node initialization. In *IJCAI*, 2021.
R. Abboud, R. Dimitrov, and I. I. Ceylan. Shortest path networks for graph property prediction. In Learning on Graphs Conference, 2022.
L. Babai. Graph isomorphism in quasipolynomial time. In STOC, 2016.
P. Barceló, F. Geerts, J. L. Reutter, and M. Ryschkov. Graph neural networks with local graph parameters. In NeurIPS, 2021.
A. D. Keros, V. Nanda, and K. Subr. Dist2Cycle: A simplicial neural network for homology localization. In AAAI, 2022.
T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
C. Morris, M. Ritzert, M. Fey, W. Hamilton, J. Lenssen, G. Rattan, and M. Grohe. Weisfeiler and Leman go neural: Higher-order graph neural networks. In AAAI, 2019.
C. Morris, G. Rattan, and P. Mutzel. Weisfeiler and Leman go sparse: Towards scalable higher-order graph embeddings. In NeurIPS, 2020.
C. Morris, G. Rattan, S. Kiefer, and S. Ravanbakhsh. SpeqNets: Sparsity-aware permutation-equivariant graph networks. In ICML, 2022.
C. Morris, F. Geerts, J. Tonshoff, and M. Grohe. WL meet VC. In ICML, 2023.
R. L. Murphy, B. Srinivasan, V. Rao, and B. Ribeiro. Relational Pooling for Graph Representations. In ICML, 2019.
M. Otto. Bounded variable logics and counting - a study in finite models. In Volume 9 of Lecture Notes in Logic, 1997.
P. A. Papp and R. Wattenhofer. A theoretical comparison of graph neural network extensions. In ICML, 2022.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
C. Qian, G. Rattan, F. Geerts, C. Morris, and M. Niepert. Ordered subgraph aggregation networks. In NeurIPS, 2022.
R. Sato, M. Yamada, and H. Kashima. Approximation ratios of graph neural networks for combinatorial problems. In NeurIPS, 2019.
R. Sato, M. Yamada, and H. Kashima. Random features strengthen graph neural networks. In SDM, 2021.
T. Sterling and J. J. Irwin. ZINC 15 – ligand discovery for everyone. Journal of Chemical Information and Modeling, 2015.
E. Thiede, W. Zhou, and R. Kondor. Autobahn: Automorphism-based graph neural nets. In NeurIPS, 2021.
P. Velicković. Message passing all the way up. In ICLR Workshop on Geometrical and Topological Representation Learning, 2022.
A. Wijesinghe and Q. Wang. A new perspective on "how graph neural networks go beyond Weisfeiler-Lehman?" In ICLR, 2022.
K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In ICLR, 2019.
C. Yun, S. Sra, and A. Jabbabaie. Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity. NeurIPS, 2019.
B. Zhang, S. Luo, L. Wang, and D. He. Rethinking the expressive power of GNNs via graph biconnectivity. In ICLR, 2023.
In this section, we talk about research on the expressivity of GNNs and how it relates to our paper. Xu et al. [2019] and Morris et al. [2019] proved that MPNNs have limited expressivity. This leads to the development of new GNNs that have higher expressivity than MPNNs. We distinguish between GNNs that perform non-standard message passing and GNNs that use an MPNN on a transformed graph. We begin with approaches that correspond to MPNNs on a transformed graph. A common approach for this adds additional features to the graph. It has been proven that adding random features to vertices improves the expressivity of GNNs [Abboud et al., 2021, Dasoulas et al., 2020, Sato et al., 2021] which allows them to learn every function defined on a graph and thus gives them maximum expressivity. However, to do this they sacrifice determinism, and permutation invariance / equivariance. Some permutation equivariant approaches extend the graph features by counting patterns such as Barceló et al. [2021] which extend the MPNN with rooted homomorphism counts of a set of patterns. All approaches that consist of deterministically adding additional features to vertices and edges in a permutation equivariant way can be trivially strongly simulated and belong to augmented message passing. Graph Structural Networks (GSN-e and GSN-v; Bouritsas et al. [2022]) introduce a new graph convolution layer which extends messages with subgraph isomorphism counts. This cannot be directly seen as transforming graphs. However, it is very similar to adding these subgraph isomorphism counts to the vertex features and thus to an MPNN operating on a transformed graph.
A Related Work: Expressivity of GNNs