SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our main results are stated in Theorems 4.3-4.8. Before we present the theorems, we need to give revised definitions of  $C(S,u,G)$  for some certain substructures  $S$ . This is because in those substructures not all nodes are structurally equal.
Definition 4.1. If  $S$  is an  $\ell$ -path with  $\ell \geqslant 2$ ,  $C(S, u, G)$  is defined to be the number of  $\ell$ -paths in  $G$  starting from node  $u$ .
Figure 2a illustrates Definition 4.1 for the  $\ell = 4$  case.
Definition 4.2. The substructures in Figures 2b, 2c and 2d are called tailed triangles, chordal cycles and triangle-rectangles, respectively. If  $S$  is a tailed triangle (or chordal cycle or triangle-rectangle),  $C(S, u, G)$  is defined to be the number of tailed triangles (or chordal cycles or triangle-rectangles) that occur as subgraphs of  $G$  and include node  $u$  at a position shown in the figures.
(a) A 4-path that starts at node  $u$
(b) A tailed triangle that passes node  $u$
(c) A chordal cycle that passes node  $u$
Figure 2: Illustrations of node-level counts of certain substructures.
(d) A triangle-rectangle that passes node  $u$
Now we state our main theorems. In the following, the definition for "node-level counting" is the same as Definition 2.2, but one should treat  $C(S,u,G)$  differently (following Definitions 4.1 and 4.2) when  $S$  is a path, a tailed triangle, a chordal cycle, or a triangle-rectangle. To define the output of a  $d$ -DRFWL(2) GNN  $f$  on domain  $\mathcal{G} \times \mathcal{V}$ , we denote
$$
f (G, u) = h _ {u u} ^ {(T)}, \quad u \in \mathcal {V} _ {G}, \tag {12}
$$
i.e., we treat the embedding of  $(u, u)$  as the representation of node  $u$ . For 1-DRFWL(2) GNNs, we have
Theorem 4.3. 1-DRFWL(2) GNNs can node-level count 3-cycles, but cannot graph-level count any longer cycles.
For 2-DRFWL(2) GNNs, we investigate not only their cycle counting power, but also their ability to count many other graph substructures. Our results include
Theorem 4.4. 2-DRFWL(2) GNNs can node-level count 2, 3, 4-paths.
Theorem 4.5. 2-DRFWL(2) GNNs can node-level count 3, 4, 5, 6-cycles.
Theorem 4.6. 2-DRFWL(2) GNNs can node-level count tailed triangles, chordal cycles and triangle-rectangles.
Theorem 4.7. 2-DRFWL(2) GNNs cannot graph-level count  $k$ -cycles with  $k \geqslant 7$  or  $k$ -cliques with  $k \geqslant 4$ .
For  $d$ -DRFWL(2) GNNs with  $d \geqslant 3$ , we have
Theorem 4.8. For any  $d \geqslant 3$ ,  $d$ -DRFWL(2) GNNs can node-level count 3, 4, 5, 6, 7-cycles, but cannot graph-level count any longer cycles.
The proofs of all theorems within this section are included in Appendix C. To give an intuitive explanation for the cycle counting power of  $d$ -DRFWL(2) GNNs, let us consider, e.g., why 2-DRFWL(2) GNNs can count up to 6-cycles. The key reason is that they allow a distance-2 tuple  $(u,v)$  to receive messages from other two distance-2 tuples  $(u,w)$  and  $(w,v)$ , and are thus aware of closed 6-walks (since  $6 = 2 + 2 + 2$ ). Indeed, if we forbid such kind of message passing, the modified 2-DRFWL(2) GNNs can no longer count 6-cycles, as experimentally verified in Appendix F.2.
The cycle counting power of GNNs. It is proposed in [33] to use GNNs' ability to count given-length cycles as a metric for their expressiveness. Prior to this work, Arvind et al. [5] and Fürer [25] have discussed the cycle counting power of the FWL(2) test: FWL(2) can and only can graph-level count up to 7-cycles. Huang et al. [33] also characterizes the node-level cycle counting power of subgraph MPNNs and  $\mathrm{I}^2$ -GNN. Apart from counting cycles, there are also some works analyzing the general substructure counting power of GNNs. Chen et al. [15] discusses the ability of  $\mathrm{WL}(k)$  tests to count general subgraphs or induced subgraphs, but the result is loose. Tahmasebi et al. [48] analyzes the substructure counting power of Recursive Neighborhood Pooling, which can be seen as a subgraph GNN with recursive subgraph extraction procedures.
The trade-off between expressive power and efficiency of GNNs. Numerous methods have been proposed to boost the expressive power of MPNNs. Many of the provably powerful GNN models have direct correspondence to the Weisfeiler-Leman hierarchy [12, 28], such as higher-order GNNs [44] and IGNs [41-43]. Despite their simplicity in theory, those models require  $O(n^{k + 1})$  time and  $O(n^{k})$  space in order to achieve equal expressive power to FWL(k) tests, and thus do not scale to large graphs even for  $k = 2$ .
Recent works try to strike a balance between the expressive power of GNNs and their efficiency. Among the state-of-the-art GNN models with  $sub-O(n^3)$  time complexity, subgraph GNNs have gained much research interest [8, 17, 24, 33, 39, 46, 57-60]. Subgraph GNNs process a graph  $G$  by 1) extracting a bag of subgraphs  $\{G_i : i = 1, 2, \dots, p\}$  from  $G$ , 2) generating representations for every subgraph  $G_i$ ,  $i = 1, 2, \dots, p$  (often using a weak GNN such as MPNN), and 3) combining the representations of all subgraphs into a representation of  $G$ . Most commonly, the number  $p$  of subgraphs extracted is equal to the number of nodes  $n$  (called a node-based subgraph extraction policy [24]). In this case, the time complexity of subgraph GNNs is upper-bounded by  $O(nm)$ , where  $m$  is the number of edges. If we further adopt the  $K$ -hop ego-network policy, i.e., extracting a  $K$ -hop subgraph  $G_u$  around each node  $u$ , the time complexity becomes  $O(n \deg^{K+1})$ . Frasca et al. [24] and Zhang et al. [58] theoretically characterize the expressive power of subgraph GNNs, and prove that subgraph GNNs with node-based subgraph extraction policies lie strictly between WL(1) and FWL(2) in the Weisfeiler-Leman hierarchy. Despite the lower complexity, in practice subgraph GNNs still suffer from heavy preprocessing and a high GPU memory usage.
Apart from subgraph GNNs, there are also attempts to add sparsity to higher-order GNNs. Morris et al. [45] proposes  $\delta$ - $k$ -LWL, a variant of the WL( $k$ ) test that updates a  $k$ -tuple  $\mathbf{u}$  only from the  $k$ -tuples with a component connected to the corresponding component in  $\mathbf{u}$ . Zhang et al. [58] proposes LFWL(2) and SLFWL(2), which are variants of FWL(2) that update a 2-tuple  $(u,v)$  from nodes in either  $\mathcal{N}(v)$  or  $\mathcal{N}(u) \cup \mathcal{N}(v)$ . Our model can also be classified into this type of approaches, yet we not only sparsify neighbors of a 2-tuple, but also 2-tuples used in message passing, which results in much lower space complexity. A detailed comparison between our method and LFWL(2)/SLFWL(2) is included in Appendix D.
In this section, we empirically evaluate the performance of  $d$ -DRFWL(2) GNNs (especially for the case of  $d = 2$ ) and verify our theoretical results. To be specific, we focus on the following questions:
Q1: Can  $d$ -DRFWL(2) GNNs reach their theoretical counting power as stated in Theorems 4.3-4.8?
Q2: How do  $d$ -DRFWL(2) GNNs perform compared with other state-of-the-art GNN models on open benchmarks for graphs?
Q3: What are the time and memory costs of  $d$ -DRFWL(2) GNNs on various datasets?
Q4: Do  $d$ -DRFWL(2) GNNs with increasing  $d$  values construct a hierarchy in discriminative power (as shown in Theorem 3.2)? Further, does this hierarchy lie between WL(1) and FWL(2) empirically?
We answer Q1-Q3 in 6.1-6.3, as well as in Appendix F. The answer to Q4 is included in Appendix F.1. The details of our implementation of  $d$ -DRFWL(2) GNNs, along with the experimental settings, are included in Appendix E. Our code for all experiments, including those in Section 6 of the main paper and in Appendix F, is available at https://github.com/zml72062/DR-FWL-2.
6.1 Substructure counting
Datasets. To answer Q1, we perform node-level substructure counting tasks on the synthetic dataset in [33, 60]. The synthetic dataset contains 5,000 random graphs, and the training/validation/test splitting is  $0.3 / 0.2 / 0.5$ . The task is to perform regression on the node-level counts of certain substructures. Normalized MAE is used as the evaluation metric.
Tasks and baselines. To verify Theorems 4.4-4.6, we use 2-DRFWL(2) GNN to perform node-level counting task on 9 different substructures: 3-cycles, 4-cycles, 5-cycles, 6-cycles, tailed triangles, chordal cycles, 4-cliques, 4-paths and triangle-rectangles. We choose MPNN, node-based subgraph GNNs (ID-GNN, NGNN, GNNAK+), PPGN, and  $\mathrm{I}^2$ -GNN as our baselines. Results for all baselines are from [33].
To verify Theorem 4.8, we compare the performances of 2-DRFWL(2) GNN and 3-DRFWL(2) GNN on node-level cycle counting tasks, with the cycle length ranging from 3 to 7.
We also conduct ablation studies to investigate what kinds of message passing are essential (among those depicted in Figure 1) for 2-DRFWL(2) GNNs to successfully count up to 6-cycles and other substructures. The experimental details and results are given in Appendix F.2. The studies also serve as a verification of Theorem 4.3.
Table 1: Normalized MAE results of node-level counting cycles and other substructures on synthetic dataset. The colored cell means an error less than 0.01.
Method	Synthetic (norm. MAE)
3-Cyc.	4-Cyc.	5-Cyc.	6-Cyc.	Tail. Tri.	Chor. Cyc.	4-Cliq.	4-Path	Tri.-Rect.
MPNN	0.3515	0.2742	0.2088	0.1555	0.3631	0.3114	0.1645	0.1592	0.2979
ID-GNN	0.0006	0.0022	0.0490	0.0495	0.1053	0.0454	0.0026	0.0273	0.0628
NGNN	0.0003	0.0013	0.0402	0.0439	0.1044	0.0392	0.0045	0.0244	0.0729
GNNAK+	0.0004	0.0041	0.0133	0.0238	0.0043	0.0112	0.0049	0.0075	0.1311
PPGN	0.0003	0.0009	0.0036	0.0071	0.0026	0.0015	0.1646	0.0041	0.0144
I²-GNN	0.0003	0.0016	0.0028	0.0082	0.0011	0.0010	0.0003	0.0041	0.0013
2-DRFWL(2) GNN	0.0004	0.0015	0.0034	0.0087	0.0030	0.0026	0.0009	0.0081	0.0070
Results. From Table 1, we see that 2-DRFWL(2) GNN achieves less-than-0.01 normalized MAE on all 3, 4, 5 and 6-cycles, verifying Theorem 4.5; 2-DRFWL(2) GNN also achieves less-than-0.01 normalized MAE on tailed triangles, chordal cycles, 4-paths and triangle-rectangles, verifying Theorems 4.4 and 4.6.
It is interesting that 2-DRFWL(2) GNN has a very good performance on the task of node-level counting 4-cliques, which by Theorem 4.7 it cannot count in theory. A similar phenomenon happens for subgraph GNNs. This may be because 2-DRFWL(2) GNN and subgraph GNNs still learn some local structural biases that have strong correlation with the number of 4-cliques.
Table 2: Normalized MAE results of node-level counting  $k$ -cycles  $(3 \leqslant k \leqslant 7)$  on synthetic dataset.
Method	Synthetic (norm. MAE)
3-Cyc.	4-Cyc.	5-Cyc.	6-Cyc.	7-Cyc.
2-DRFWL(2) GNN	0.0004	0.0015	0.0034	0.0087	0.0362
3-DRFWL(2) GNN	0.0006	0.0020	0.0047	0.0099	0.0176
From Table 2, we see that 3-DRFWL(2) GNN achieves a comparable performance to 2-DRFWL(2) GNN on the tasks of counting 3, 4, 5 and 6-cycles. Yet when it comes to counting 7-cycles, 3-DRFWL(2) GNN greatly outperforms 2-DRFWL(2) GNN, verifying Theorems 4.7 and 4.8.
Finally, from the last row of Table 7 we see that 1-DRFWL(2) GNN achieves less-than-0.01 normalized MAE on 3-cycles, but performs badly on 4, 5 and 6-cycles. This result verifies Theorem 4.3.
6.2 Molecular property prediction
Datasets. To answer Q2, we evaluate the performance of  $d$ -DRFWL(2) GNNs on four popular molecular graph datasets—QM9, ZINC, ogbg-molhiv and ogbg-molpcba. QM9 contains 130k small molecules, and the task is regression on 12 targets. One can refer to the page for the meaning of those 12 targets. ZINC [20], including a smaller version (ZINC-12K) and a full version (ZINC-250K), is a dataset of chemical compounds and the task is graph regression. The ogbg-molhiv (containing 41k molecules) and ogbg-molpcba (containing 438k molecules) datasets belong to the Open Graph Benchmark (OGB) [31]; the task on both datasets is binary classification. Details of the four datasets are given in Appendix E.2.2.
Table 3: MAE results on QM9 (smaller the better). The top two are highlighted as First, Second.
Target	1-GNN	1-2-3-GNN	DTNN	Deep LRP	PPGN	NGNN	I²-GNN	2-DRFWL(2) GNN
μ	0.493	0.476	0.244	0.364	0.231	0.428	0.428	0.346
α	0.78	0.27	0.95	0.298	0.382	0.29	0.230	0.222
εhomo	0.00321	0.00337	0.00388	0.00254	0.00276	0.00265	0.00261	0.00226
εlumo	0.00355	0.00351	0.00512	0.00277	0.00287	0.00297	0.00267	0.00225
Δε	0.0049	0.0048	0.0112	0.00353	0.00406	0.0038	0.0038	0.00324
R²	34.1	22.9	17.0	19.3	16.07	20.5	18.64	15.04
ZPVE	0.00124	0.00019	0.00172	0.00055	0.0064	0.0002	0.00014	0.00017
U0	2.32	0.0427	2.43	0.413	0.234	0.295	0.211	0.156
U	2.08	0.111	2.43	0.413	0.234	0.361	0.206	0.153
H	2.23	0.0419	2.43	0.413	0.229	0.305	0.269	0.145
G	1.94	0.0469	2.43	0.413	0.238	0.489	0.261	0.156
Cv	0.27	0.0944	2.43	0.129	0.184	0.174	0.0730	0.0901
Baselines. For QM9, the baselines are chosen as 1-GNN, 1-2-3-GNN [44], DTNN [53], Deep LRP [15], PPGN, NGNN [59] and  $\mathrm{I}^2$ -GNN [33]. Methods [3, 26, 40, 47] utilizing geometric features or quantum mechanics theory are omitted to fairly compare the graph representation power of the models. For ZINC and ogbg-molhiv, we adopt GIN, PNA [16], DGN [7], HIMP [23], GSN [10], Deep LRP [15], CIN [9], NGNN, GNNAK+, SUN [24] and  $\mathrm{I}^2$ -GNN as our baselines. For ogbg-molpcba, the baselines are GIN, PNA, DGN, NGNN and GNNAK+. The experimental details are given in Appendix E.2.2. We leave the results on ZINC, ogbg-molhiv and ogbg-molpcba to Appendix F.3.
Results. On QM9, Table 3 shows that 2-DRFWL(2) GNN attains top two results on most (11 out of 12) targets. Moreover, 2-DRFWL(2) GNN shows a good performance on targets  $U_0, U, H$  and  $G$ , where subgraph GNNs like NGNN or  $\mathbf{I}^2$ -GNN have a poor performance. The latter fact actually reveals that our method has a stronger ability to capture long-range interactions on graphs than subgraph GNNs, since the targets  $U_0, U, H$  and  $G$  are macroscopic thermodynamic properties of
molecules and heavily depend on such long-range interactions. We leave the detailed analysis to Appendix F.4. Apart from QM9, Table 8 of Appendix F.3 shows that  $d$ -DRFWL(2) GNN outperforms CIN [9] on ZINC-12K; the performance on obgb-molhiv is also comparable to baseline methods. These results show that although designed for cycle counting,  $d$ -DRFWL(2) GNN is also highly competitive on general molecular tasks.
It is interesting to notice that  $d$ -DRFWL(2) GNN shows an inferior performance on ogbg-molpcba, compared with baseline methods. We conjecture that the results on ogbg-molpcba might be insensitive to the gain in cycle counting power, and might prefer simple model architectures rather than highly expressive ones.
6.3 Empirical efficiency
To answer Q3, we compare the time and memory costs of 2-DRFWL(2) GNN with MPNN, NGNN and  $\mathrm{I}^2$ -GNN on two datasets—QM9 and ogbg-molhiv. We use three metrics to evaluate the empirical efficiency of 2-DRFWL(2) GNNs: the maximal GPU memory usage during training, the preprocessing time, and the training time per epoch.
To make a fair comparison, we fix the number of 2-DRFWL(2) GNN layers and the number of message passing layers in all baseline methods to 5; we also fix the size of hidden dimension to 64 for QM9 and 300 for ogbg-molhiv. The subgraph heights for NGNN and  $\mathrm{I}^2$ -GNN are both 3. The batch size is always 64.
Table 4: Empirical efficiency of 2-DRFWL(2) GNN.
Method	QM9	ogbg-molhiv
Memory (GB)	Pre. (s)	Train (s/epoch)	Memory (GB)	Pre. (s)	Train (s/epoch)
MPNN	2.28	64	45.3	2.00	2.4	18.8
NGNN	13.72	2354	107.8	5.23	1003	42.7
I²-GNN	19.69	5287	209.9	11.07	2301	84.3
2-DRFWL(2) GNN	2.31	430	141.9	4.44	201	44.3
Results. From Table 4, we see that the preprocessing time of 2-DRFWL(2) GNN is much shorter than subgraph GNNs like NGNN or  $\mathrm{I}^2$ -GNN. Moreover, 2-DRFWL(2) GNN requires much less GPU memory while training, compared with subgraph GNNs. The training time of 2-DRFWL(2) GNN is comparable to NGNN (which can only count up to 4-cycles), and much shorter than  $\mathrm{I}^2$ -GNN.
We also evaluate the empirical efficiency of 2-DRFWL(2) GNN on graphs of larger sizes. The results, along with details of the datasets we use, are given in Appendix F.5. From Table 14 in Appendix F.5, we see that 2-DRFWL(2) GNN easily scales to graphs with  $\sim 500$  nodes, as long as the average degree is small.
Motivated by the analysis of why FWL(2) has a stronger cycle counting power than WL(1), we propose  $d$ -DRFWL(2) tests and  $d$ -DRFWL(2) GNNs. It is then proved that with  $d = 2$ ,  $d$ -DRFWL(2) GNNs can already count up to 6-cycles, retaining most of the cycle counting power of FWL(2). Because  $d$ -DRFWL(2) GNNs explicitly leverage the local nature of cycle counting, they are much more efficient than other existing GNN models that have comparable cycle counting power. Besides,  $d$ -DRFWL(2) GNNs also have an outstanding performance on various real-world tasks. Finally, we have to point out that our current implementation of  $d$ -DRFWL(2) GNNs, though being efficient most of the time, still has difficulty scaling to datasets with a large average degree such as ogbg-ppa, since the preprocessing time is too long ( $\sim 40$  seconds per graph on ogbg-ppa). This also makes our method unsuitable for node classification tasks, since these tasks typically involve graphs with large average degrees. We leave the exploration of more efficient  $d$ -DRFWL(2) implementation to future work.
The work is supported in part by the National Natural Science Foundation of China (62276003), the National Key Research and Development Program of China (No. 2021ZD0114702), and Alibaba Innovative Research Program.
[1] Ralph Abboud, Ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. arXiv preprint arXiv:2010.01179, 2020.
[2] Noga Alon, Raphael Yuster, and Uri Zwick. Finding and counting given length cycles. Algorithmica, 17(3):209-223, 1997.
[3] Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.
[4] V. Arvind, Frank Fuhlbrück, Johannes Köbler, and Oleg Verbitsky. On weisfeiler-lerman invariance: Subgraph counts and related graph properties. Journal of Computer and System Sciences, 113:42-59, 2020. ISSN 0022-0000. doi: https://doi.org/10.1016/j.jcss.2020.04.003. URL https://www.sciencedirect.com/science/article/pii/S0022000020300386.
[5] Vikraman Arvind, Frank Fuhlbrück, Johannes Köbler, and Oleg Verbitsky. On weisfeiler-lemann invariance: Subgraph counts and related graph properties. Journal of Computer and System Sciences, 113:42-59, 2020.
[6] Muhammet Balcilar, Pierre Héroux, Benoit Gauzere, Pascal Vasseur, Sébastien Adam, and Paul Honeine. Breaking the limits of message passing graph neural networks. In International Conference on Machine Learning, pages 599–608. PMLR, 2021.
[7] Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Directional graph networks. In International Conference on Machine Learning, pages 748-758. PMLR, 2021.
[8] Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. In International Conference on Learning Representations, 2022.
[9] Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Lio, Guido F Montufar, and Michael Bronstein. Weisfeiler and lehman go cellular: Cw networks. Advances in Neural Information Processing Systems, 34:2625-2640, 2021.
[10] Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael M Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):657-668, 2022.
[11] Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiv:1711.07553, 2017.
[12] Jin-Yi Cai, Martin Fürer, and Neil Immerman. An optimal lower bound on the number of variables for graph identification. Combinatorica, 12(4):389-410, 1992.
[13] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pages 1725-1735. PMLR, 2020.
[14] Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. Advances in neural information processing systems, 32, 2019.
[15] Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count substructures? Advances in neural information processing systems, 33:10383-10395, 2020.
[52] Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. NTI, Series, 2(9):12-16, 1968.
[53] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Molecularnet: a benchmark for molecular machine learning. Chemical science, 9(2):513-530, 2018.
[54] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4-24, 2020.
[55] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2018.
[56] Zuoyu Yan, Junru Zhou, Liangcai Gao, Zhi Tang, and Muhan Zhang. Efficiently counting substructures by subgraph gnns without running gnn on subgraphs. arXiv preprint arXiv:2303.10576, 2023.
[57] Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10737-10745, 2021.
[58] Bohang Zhang, Guhao Feng, Yiheng Du, Di He, and Liwei Wang. A complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests. arXiv preprint arXiv:2302.07090, 2023.
[59] Muhan Zhang and Pan Li. Nested graph neural networks. Advances in Neural Information Processing Systems, 34:15734-15747, 2021.
[60] Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. In International Conference on Learning Representations, 2022.
[61] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open, 1:57-81, 2020.
A Message passing neural networks
Message passing neural networks (MPNNs) [27, 36, 50, 55] are a class of GNNs that iteratively updates node representations  $h_u$ ,  $u \in \mathcal{V}_G$  by aggregating information from neighbors. At the  $t$ -th  $(t \geqslant 1)$  iteration, the update rule for MPNNs is
$$
h _ {u} ^ {(t)} = f ^ {(t)} \left(h _ {u} ^ {(t - 1)}, \bigoplus_ {v \in \mathcal {N} (u)} m ^ {(t)} \left(h _ {u} ^ {(t - 1)}, h _ {v} ^ {(t - 1)}, e _ {u v}\right)\right), \tag {13}
$$
where  $h_u^{(t)}$  is the representation of node  $u$  at the  $t$ -th iteration and  $e_{uv}$  is the edge feature of  $\{u, v\} \in \mathcal{E}_G$ .  $f^{(t)}$  and  $m^{(t)}$  are learnable functions, and  $\bigoplus$  is a permutation-invariant aggregation operator, such as sum, mean or max. The representation of graph  $G$  is obtained by
$$
h _ {G} = R \left(\left\{\left\{h _ {u}: u \in \mathcal {V} _ {G} \right\} \right\}\right), \tag {14}
$$
where  $R$  is a permutation-invariant function of multisets.
It is known [55] that the ability of MPNNs to discriminate between non-isomorphic graphs is upper-bounded by the WL(1) test. When it comes to counting cycles, MPNNs cannot graph-level count any cycles, as stated in [33].
B Proofs of theorems in Section 3
We restate Theorem 3.1 as following,
Theorem B.1. In terms of the ability to distinguish between non-isomorphic graphs, the  $d$ -DRFWL(2) test is strictly more powerful than  $WL(1)$ , for any  $d \geqslant 1$ .