SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junru Zhou $^{1}$  Jiarui Feng $^{2}$  Xiyuan Wang $^{1}$  Muhan Zhang $^{1}$
$^{2}$ Department of CSE, Washington University in St. Louis
zml72062@stu.pku.edu.cn, feng.jiarui@wustl.edu,
{wangxiyuan,muhan}@pku.edu.cn
The ability of graph neural networks (GNNs) to count certain graph substructures, especially cycles, is important for the success of GNNs on a wide range of tasks. It has been recently used as a popular metric for evaluating the expressive power of GNNs. Many of the proposed GNN models with provable cycle counting power are based on subgraph GNNs, i.e., extracting a bag of subgraphs from the input graph, generating representations for each subgraph, and using them to augment the representation of the input graph. However, those methods require heavy preprocessing, and suffer from high time and memory costs. In this paper, we overcome the aforementioned limitations of subgraph GNNs by proposing a novel class of GNNs— $d$ -Distance-Restricted FWL(2) GNNs, or  $d$ -DRFWL(2) GNNs, based on the well-known FWL(2) algorithm. As a heuristic method for graph isomorphism testing, FWL(2) colors all node pairs in a graph and performs message passing among those node pairs. In order to balance the expressive power and complexity,  $d$ -DRFWL(2) GNNs simplify FWL(2) by restricting the range of message passing to node pairs whose mutual distances are at most  $d$ . This way,  $d$ -DRFWL(2) GNNs exploit graph sparsity while avoiding the expensive subgraph extraction operations in subgraph GNNs, making both the time and space complexity lower. We theoretically investigate both the discriminative power and the cycle counting power of  $d$ -DRFWL(2) GNNs. Our most important finding is that  $d$ -DRFWL(2) GNNs have provably strong cycle counting power even with  $d = 2$ : they can count all 3, 4, 5, 6-cycles. Since 6-cycles (e.g., benzene rings) are ubiquitous in organic molecules, being able to detect and count them is crucial for achieving robust and generalizable performance on molecular tasks. Experiments on both synthetic datasets and molecular datasets verify our theory. To the best of our knowledge, 2-DRFWL(2) GNN is the most efficient GNN model to date (both theoretically and empirically) that can count up to 6-cycles.
Graphs are important data structures suitable for representing relational or structural data. As a powerful tool to learn node-level, link-level or graph-level representations for graph-structured data, graph neural networks (GNNs) have achieved remarkable successes on a wide range of tasks [54, 61]. Among the various GNN models, Message Passing Neural Networks (MPNNs) [27, 36, 50, 55] are a widely adopted class of GNNs. However, the expressive power of MPNNs has been shown to be upper-bounded by the Weisfeiler-Leman test [55]. More importantly, MPNNs even fail to detect some simple substructures (e.g., 3, 4-cycles) [33], thus losing great structural information in graphs.
The weak expressive power of MPNNs has aroused a search for more powerful GNN models. To evaluate the expressive power of these GNN models, there are usually two perspectives. One is to characterize their discriminative power, i.e., the ability to distinguish between non-isomorphic graphs. It is shown in [14] that an equivalence exists between distinguishing all non-isomorphic graph pairs and approximating any permutation-invariant functions on graphs. Although the discriminative power partially reveals the function approximation ability of a GNN model, it fails to tell what specific functions a model is able to approximate. Another perspective is to directly characterize what specific function classes a model can approximate. In particular, we can study the expressive power of a GNN model by asking what graph substructures it can count. This perspective is often more practical since the task of counting substructures (especially cycles) is closely related to a variety of domains such as chemistry [18, 35], biology [37], and social network analysis [34].
Despite the importance of counting cycles, the task gets increasingly difficult as the length of cycle increases [2]. To have an intuitive understanding of this difficulty, we first discuss why MPNNs fail to count any cycles. The reason is clear: MPNNs only keep track of the rooted subtree around each node [55], and without node identity, any neighboring node  $v$  of  $u$  has no idea which of its neighbors are also neighbors of  $u$ . Therefore, MPNNs cannot count even the simplest 3-cycles. In contrast, the 2-dimensional Folklore Weisfeiler-Leman, or FWL(2) test [12], uses 2-tuples of nodes instead of nodes as the basic units for message passing, and thus naturally encodes closed walks. For example, when a 2-tuple  $(u,v)$  receives messages from 2-tuples  $(w,v)$  and  $(u,w)$  in FWL(2),  $(u,v)$  gets aware of the walk  $u\rightarrow w\rightarrow v\rightarrow u$ , whose length is  $d(u,w) + d(w,v) + d(u,v)\eqqcolon l$ . As long as the intermediate nodes do not overlap, we immediately get an  $l$ -cycle passing both  $u$  and  $v$ . In fact, FWL(2) can provably count up to 7-cycles [5, 25].
However, FWL(2) has a space and time complexity of  $O(n^{2})$  and  $O(n^{3})$ , which makes it impractical for large graphs and limits its real-world applications. Inspired by the above discussion, we naturally raise a question: to what extent can we retain the cycle counting power of FWL(2) while reducing the time and space complexity? Answering this question requires another observation: cycle counting tasks are intrinsically local. For example, the number of 6-cycles that pass a given node  $u$  has nothing to do with the nodes that have a shortest-path distance  $\geqslant 4$  to  $u$ . Therefore, if we only record the embeddings of node pairs with mutual distance  $\leqslant d$  in FWL(2), where  $d$  is a fixed positive integer, then the ability of FWL(2) to count substructures with diameter  $\leqslant d$  should be retained. Moreover, given that most real-world graphs are sufficiently sparse, this simplified algorithm runs with complexities much lower than FWL(2), since we only need to store and update a small portion of the  $n^{2}$  embeddings for all 2-tuples. We call this simplified version  $d$ -Distance-Restricted FWL(2), or  $d$ -DRFWL(2), reflecting that only 2-tuples with restricted distances get updated.
Main contributions. Our main contributions are summarized as follows:
1. We propose  $d$ -DRFWL(2) tests as well as their neural versions,  $d$ -DRFWL(2) GNNs. We study how the hyperparameter  $d$  affects the discriminative power of  $d$ -DRFWL(2), and show that a strict expressiveness hierarchy exists for  $d$ -DRFWL(2) GNNs with increasing  $d$ , both theoretically and experimentally.
2. We study the cycle counting power of  $d$ -DRFWL(2) GNNs. Our major results include:
- 2-DRFWL(2) GNNs can already count up to 6-cycles, covering common structures like benzene rings in organic chemistry. The time and space complexities of 2-DRFWL(2) are  $O(n \deg^4)$  and  $O(n \deg^2)$  respectively, making it the most efficient one to date among other models with 6-cycle counting power such as  $I^2$ -GNN [33].
- With  $d \geqslant 3$ ,  $d$ -DRFWL(2) GNNs can count up to 7-cycles, fully retaining the cycle counting power of FWL(2) but with complexities strictly lower than FWL(2). This finding also confirms the existence of GNNs with the same cycle counting power as FWL(2), but strictly weaker discriminative power.
3. We compare the performance and empirical efficiency of  $d$ -DRFWL(2) GNNs (especially for  $d = 2$ ) with other state-of-the-art GNNs on both synthetic and real-world datasets. The results verify our theory on the counting power of  $d$ -DRFWL(2) GNNs. Additionally, for the case of  $d = 2$ , the amount of GPU memory required for training 2-DRFWL(2) GNNs is greatly reduced compared with subgraph GNNs like ID-GNN [57], NGNN [59] and  $\mathrm{I}^2$ -GNN [33]; the preprocessing time and training time of 2-DRFWL(2) GNNs are also much less than  $\mathrm{I}^2$ -GNN.
For a simple, undirected graph  $G$ , we use  $\mathcal{V}_G$  and  $\mathcal{E}_G$  to denote its node set and edge set respectively. For every node  $v \in \mathcal{V}_G$ , we define its  $k$ -th hop neighbors as  $\mathcal{N}_k(v) = \{u \in \mathcal{V}_G : d(u,v) = k\}$ , where  $d(u,v)$  is the shortest-path distance between nodes  $u$  and  $v$ . We further introduce the symbols  $\mathcal{N}_{\leqslant k}(v) = \bigcup_{i=1}^{k} \mathcal{N}_i(v)$  and  $\mathcal{N}(v) = \mathcal{N}_1(v)$ . For any  $n \in \mathbb{N}$ , we denote  $[n] = \{1,2,\ldots,n\}$ .
An  $\ell$ -cycle  $(\ell \geqslant 3)$  in the (simple, undirected) graph  $G$  is a sequence of  $\ell$  edges  $\{v_{1}, v_{2}\}, \{v_{2}, v_{3}\}, \ldots, \{v_{\ell}, v_{1}\} \in \mathcal{E}_{G}$  with  $v_{i} \neq v_{j}$  for any  $i \neq j$  and  $i, j \in [\ell]$ . The  $\ell$ -cycle is said to pass a node  $v$  if  $v$  is among the nodes  $\{v_{i}: i \in [\ell]\}$ . An  $\ell$ -path is a sequence of  $\ell$  edges  $\{v_{1}, v_{2}\}, \{v_{2}, v_{3}\}, \ldots, \{v_{\ell}, v_{\ell+1}\} \in \mathcal{E}_{G}$  with  $v_{i} \neq v_{j}$  for any  $i \neq j$  and  $i, j \in [\ell+1]$ , and it is said to start at node  $v_{1}$  and end at node  $v_{\ell+1}$ . An  $\ell$ -walk from  $v_{1}$  to  $v_{\ell+1}$  is a sequence of  $\ell$  edges  $\{v_{1}, v_{2}\}, \{v_{2}, v_{3}\}, \ldots, \{v_{\ell}, v_{\ell+1}\} \in \mathcal{E}_{G}$  but the nodes  $v_{1}, v_{2}, \ldots, v_{\ell+1}$  can coincide. An  $\ell$ -clique is  $\ell$  nodes  $v_{1}, \ldots, v_{\ell}$  such that  $\{v_{i}, v_{j}\} \in \mathcal{E}_{G}$  for all  $i \neq j$  and  $i, j \in [\ell]$ .
Let  $\mathcal{G}$  be the set of all simple, undirected graphs. If  $S$  is a graph substructure on  $\mathcal{G}$ , and  $G \in \mathcal{G}$  is a graph, we use  $C(S, G)$  to denote the number of inequivalent substructures  $S$  that occur as subgraphs of  $G$ . Similarly, if  $u$  is a node of  $G$ , we use  $C(S, u, G)$  to denote the number of inequivalent substructures  $S$  that pass node  $u$  and occur as subgraphs of  $G$ .
Following [15, 33], we give the definition of whether a function class  $\mathcal{F}$  on graphs can count a certain substructure  $S$ .
Definition 2.1 (Graph-level count). Let  $\mathcal{F}_{\mathrm{graph}}$  be a function class on  $\mathcal{G}$ , i.e.,  $f_{\mathrm{graph}}: \mathcal{G} \to \mathbb{R}$  for all  $f_{\mathrm{graph}} \in \mathcal{F}_{\mathrm{graph}}$ .  $\mathcal{F}_{\mathrm{graph}}$  is said to be able to graph-level count a substructure  $S$  on  $\mathcal{G}$  if for  $\forall G_1, G_2 \in \mathcal{G}$  such that  $C(S, G_1) \neq C(S, G_2)$ , there exists  $f_{\mathrm{graph}} \in \mathcal{F}_{\mathrm{graph}}$  such that  $f_{\mathrm{graph}}(G_1) \neq f_{\mathrm{graph}}(G_2)$ .
Definition 2.2 (Node-level count). Let  $\mathcal{G} \times \mathcal{V} = \{(G, u) : G \in \mathcal{G}, u \in \mathcal{V}_G\}$ . Let  $\mathcal{F}_{\mathrm{node}}$  be a function class on  $\mathcal{G} \times \mathcal{V}$ , i.e.,  $f_{\mathrm{node}}: \mathcal{G} \times \mathcal{V} \to \mathbb{R}$  for all  $f_{\mathrm{node}} \in \mathcal{F}_{\mathrm{node}}$ .  $\mathcal{F}_{\mathrm{node}}$  is said to be able to node-level count a substructure  $S$  on  $\mathcal{G}$  if for  $\forall (G_1, u_1), (G_2, u_2) \in \mathcal{G} \times \mathcal{V}$  such that  $C(S, u_1, G_1) \neq C(S, u_2, G_2)$ , there exists  $f_{\mathrm{node}} \in \mathcal{F}_{\mathrm{node}}$  such that  $f_{\mathrm{node}}(G_1, u_1) \neq f_{\mathrm{node}}(G_2, u_2)$ .
Notice that  $C(S, G)$  can be calculated by  $\sum_{u \in \mathcal{V}_G} C(S, u, G)$  divided by a factor only depending on  $S$ , for any given substructure  $S$ . (For example, for triangles the factor is 3.) Therefore, counting a substructure at node level is harder than counting it at graph level.
2.2 FWL  $(k)$  graph isomorphism tests
WL(1) test. The 1-dimensional Weisfeiler-Leman test, or WL(1) test, is a heuristic algorithm for the graph isomorphism problem [52]. For a graph  $G$ , the WL(1) test iteratively assigns a color  $W(v)$  to every node  $v \in \mathcal{V}_G$ . At the 0-th iteration, the color  $W^{(0)}(v)$  is identical for every node  $v$ . At the  $t$ -th iteration with  $t \geqslant 1$ ,
$$
W ^ {(t)} (v) = \operatorname {H A S H} ^ {(t)} \left(W ^ {(t - 1)} (v), \operatorname {P O O L} ^ {(t)} \left(\{\{W ^ {(t - 1)} (u): u \in \mathcal {N} (v) \}\}\right)\right), \tag {1}
$$
where  $\mathrm{HASH}^{(t)}$  and  $\mathrm{POOL}^{(t)}$  are injective hashing functions, and  $\{\cdot\}$  means a multiset (set with potentially identical elements). The algorithm stops when the node colors become stable, i.e.,  $\forall v, u \in \mathcal{V}_G, W^{(t+1)}(v) = W^{(t+1)}(v) \Leftrightarrow W^{(t)}(v) = W^{(t)}(u)$ . We denote the stable coloring of node  $v$  as  $W^{(\infty)}(v)$ ; then the representation for graph  $G$  is
$$
W (G) = \operatorname {R E A D O U T} \left(\{\{W ^ {(\infty)} (v): v \in \mathcal {V} _ {G} \} \}\right), \tag {2}
$$
where READOUT is an arbitrary injective multiset function.
$\mathbf{FWL}(k)$  tests. For  $k \geqslant 2$ , the  $k$ -dimensional Folklore Weisfeiler-Leman tests, or  $\mathrm{FWL}(k)$  tests, define a hierarchy of algorithms for graph isomorphism testing, as described in [12, 32, 42]. For a graph  $G$ , the  $\mathrm{FWL}(k)$  test assigns a color  $W(\mathbf{v})$  for every  $k$ -tuple  $\mathbf{v} = (v_1, \ldots, v_k) \in \mathcal{V}_G^k$ . At the 0-th iteration, the color  $W^{(0)}(\mathbf{v})$  is the atomic type of  $\mathbf{v}$ , denoted as  $\mathrm{atp}(\mathbf{v})$ . If we denote the
subgraphs of  $G$  induced by  $\mathbf{v}$  and  $\mathbf{v}'$  as  $G(\mathbf{v})$  and  $G(\mathbf{v}')$  respectively, then the function  $\mathrm{atp}(\cdot)$  can be any function on  $\mathcal{V}_G^k$  that satisfies the following condition:  $\mathrm{atp}(\mathbf{v}) = \mathrm{atp}(\mathbf{v}')$  iff the mapping  $v_i \mapsto v_i'$ ,  $i \in [k]$  (i.e., the mapping that maps each  $v_i$  to its corresponding  $v_i'$ , for  $i \in [k]$ ) induces an isomorphism from  $G(\mathbf{v})$  to  $G(\mathbf{v}')$ .
At the  $t$ -th iteration with  $t \geqslant 1$ ,  $\mathrm{FWL}(k)$  updates the color of  $\mathbf{v} \in \mathcal{V}_G^k$  as
$$
W ^ {(t)} (\mathbf {v}) = \operatorname {H A S H} ^ {(t)} \left(W ^ {(t - 1)} (\mathbf {v}), \operatorname {P O O L} ^ {(t)} \left(\{\{\text {s i f t} (W ^ {(t - 1)}, \mathbf {v}, w): w \in \mathcal {V} _ {G} \} \}\right)\right), \tag {3}
$$
where  $\mathrm{sift}(f,\mathbf{v},w)$  is defined as
$$
\operatorname {s i f t} (f, \mathbf {v}, w) = \left(f (\mathbf {v} [ 1 \rightarrow w ]), f (\mathbf {v} [ 2 \rightarrow w ]), \dots , f (\mathbf {v} [ k \rightarrow w ])\right). \tag {4}
$$
Here we use  $\mathbf{v}[j\to w]$  to denote  $(v_{1},\ldots ,v_{j - 1},w,v_{j + 1},\ldots ,v_{k})$  for  $j\in [k]$  and  $w\in \mathcal{V}_G$
$\mathrm{HASH}^{(t)}$  and  $\mathrm{POOL}^{(t)}$  are again injective hashing functions. The algorithm stops when all  $k$ -tuples receive stable colorings. The stable coloring of  $\mathbf{v} \in \mathcal{V}_G^k$  is denoted as  $W^{(\infty)}(\mathbf{v})$ . We then calculate the representation for graph  $G$  as
$$
W (G) = \operatorname {R E A D O U T} \left(\left\{\left\{W ^ {(\infty)} (\mathbf {v}): \mathbf {v} \in \mathcal {V} _ {G} ^ {k} \right\} \right\}\right), \tag {5}
$$
3  $d$ -Distance-Restricted FWL(2) GNNs
In this section, we propose the  $d$ -Distance Restricted FWL(2) tests/GNNs. They use 2-tuples like FWL(2), but restrict the distance between nodes in each 2-tuple to be  $\leqslant d$ , which effectively reduces the number of 2-tuples to store and aggregate while still retaining great cycle counting power.
3.1  $d$ -DRFWL(2) tests
We call a 2-tuple  $(u,v)\in \mathcal{V}_G^2$  a distance- $\pmb{k}$  tuple if the shortest-path distance between  $u$  and  $v$  is  $k$  in the following.  $d$ -Distance Restricted FWL(2) tests, or  $d$ -DRFWL(2) tests, assign a color  $W(u,v)$  for every distance- $\pmb{k}$  tuple  $(u,v)$  with  $0\leqslant k\leqslant d$ . Initially, the color  $W^{(0)}(u,v)$  only depends on  $d(u,v)$ . For the  $t$ -th iteration with  $t\geqslant 1$ ,  $d$ -DRFWL(2) updates the colors using the following rule,
For each  $k = 0,1,\ldots ,d$
$$
W ^ {(t)} (u, v) = \operatorname {H A S H} _ {k} ^ {(t)} \left(W ^ {(t - 1)} (u, v), \left(M _ {i j} ^ {k (t)} (u, v)\right) _ {0 \leqslant i, j \leqslant d}\right), \quad \text {i f} d (u, v) = k, \tag {6}
$$
where  $\mathrm{HASH}_k^{(t)}$  is an injective hashing function for distance  $k$  and iteration  $t$ , and  $M_{ij}^{k(t)}(u,v)$  is defined as
$$
M _ {i j} ^ {k (t)} (u, v) = \operatorname {P O O L} _ {i j} ^ {k (t)} \left(\left\{\left\{\left(W ^ {(t - 1)} (w, v), W ^ {(t - 1)} (u, w)\right): w \in \mathcal {N} _ {i} (u) \cap \mathcal {N} _ {j} (v) \right\} \right\}\right). \tag {7}
$$
The symbol  $\left(M_{ij}^{k(t)}(u,v)\right)_{0\leqslant i,j\leqslant d}$  stands for  $\left(M_{00}^{k(t)}(u,v),M_{01}^{k(t)}(u,v),\ldots ,M_{0d}^{k(t)}(u,v),\ldots , M_{dd}^{k(t)}(u,v)\right)$ . Each of the POOL with  $0\leqslant i,j,k\leqslant d$  is an injective multiset hashing function. Briefly speaking, the rules (6) and (7) update the color of a distance- $\pmb{k}$  tuple  $(u,v)$  using colors of distance- $\pmb{i}$  and distance- $\pmb{j}$  tuples.
When every distance- $k$  tuple  $(u,v)$  with  $0 \leqslant k \leqslant d$  receives its stable coloring, denoted as  $W^{(\infty)}(u,v)$ , the representation of  $G$  is calculated as
$$
W (G) = \operatorname {R E A D O U T} \left(\left\{\left\{W ^ {(\infty)} (u, v): (u, v) \in \mathcal {V} _ {G} ^ {2} \text {a n d} 0 \leqslant d (u, v) \leqslant d \right\} \right\}\right). \tag {8}
$$
Figure 1 illustrates how 2-DRFWL(2) updates the color of a distance-2 tuple  $(u,v)$ . Since only distance- $k$  tuples with  $0 \leqslant k \leqslant 2$  are colored in 2-DRFWL(2) tests, there are 7 terms of the form  $((W(w,v), W(u,w))$  (with  $w \in \{u,v,x,y,z,t,r\}$ ) contributing to the update of  $W(u,v)$ . The 7 nodes  $u,v,x,y,z,t,r$  are filled with different colors, according to their distances to  $u$  and to  $v$ . For example, the violet node  $u$  has distance 0 to  $u$  and distance 2 to  $v$ , thus contributing to  $M_{02}^2(u,v)$ ; the green nodes  $x$  and  $y$  have distance 1 to either  $u$  or  $v$ , thus contributing to  $M_{11}^2(u,v)$ . Analogously, nodes with red, blue, orange and pink colors contribute to  $M_{20}^2(u,v)$ ,  $M_{12}^2(u,v)$ ,  $M_{21}^2(u,v)$  and  $M_{22}^2(u,v)$ , respectively. Finally, the uncolored nodes  $q$  and  $s$  do not contribute to the update of  $W(u,v)$ , since
they have distance 3 (which is greater than 2) to  $u$ . From the figure, we can observe that by redefining neighbors and sparsifying 2-tuples of FWL(2),  $d$ -DRFWL(2) significantly reduces the complexity and focuses only on local structures, especially on sparse graphs.
Figure 1: Neighbor aggregation in 2-DRFWL(2) for 2-tuple  $(u, v)$ .
Now we study the expressive power of  $d$ -DRFWL(2) tests by comparing them with the WL hierarchy. First, we can prove that  $d$ -DRFWL(2) tests are strictly more powerful than WL(1), for every  $d$ .
Theorem 3.1. In terms of the ability to distinguish between non-isomorphic graphs, the  $d$ -DRFWL(2) test is strictly more powerful than  $WL(1)$ , for any  $d \geqslant 1$ .
Next, we compare  $d$ -DRFWL(2) tests with FWL(2). Actually, since it is easy for the FWL(2) test to compute distance between every pair of nodes (we can initialize  $W(u,v)$  as 0, 1 and  $\infty$  for  $u = v$ ,  $(u,v) \in \mathcal{E}_G$  and all other cases, and iteratively update  $W(u,v)$  with  $\min \{W(u,v), \min_w\{W(u,w) + W(w,v)\}\}$ ), the FWL(2) test can use its update rule to simulate (6) and (7) by applying different  $\mathrm{HASH}_k^{(t)}$  and  $\mathrm{POOL}_{ij}^{k(t)}$  functions to different  $(i,j,k)$  values. This implies that  $d$ -DRFWL(2) tests are at most as powerful as the FWL(2) test. Actually, this hierarchy in expressiveness is strict, due to the following theorem.
Theorem 3.2. In terms of the ability to distinguish between non-isomorphic graphs,  $FWL(2)$  is strictly more powerful than  $d$ -DRFWL(2), for any  $d \geqslant 1$ . Moreover,  $(d + 1)$ -DRFWL(2) is strictly more powerful than  $d$ -DRFWL(2).
The proofs of all theorems within this section are included in Appendix B.
3.2  $d$ -DRFWL(2) GNNs
Based on the  $d$ -DRFWL(2) tests, we now propose  $d$ -DRFWL(2) GNNs. Let  $G$  be a graph which can have node features  $f_v \in \mathbb{R}^{d_f}, v \in \mathcal{V}_G$  and (or) edge features  $e_{uv} \in \mathbb{R}^{d_e}, \{u, v\} \in \mathcal{E}_G$ . A  $d$ -DRFWL(2) GNN is defined as a function of the form
$$
f = M \circ R \circ L _ {T} \circ \sigma_ {T - 1} \circ \dots \circ \sigma_ {1} \circ L _ {1}. \tag {9}
$$
The input of  $f$  is the initial labeling  $h_{uv}^{(0)}, 0 \leqslant d(u,v) \leqslant d$ . Each  $L_{t}$  with  $t = 1,2,\dots,T$  in (9) is called a  $d$ -DRFWL(2) GNN layer, which transforms  $h_{uv}^{(t - 1)}$  into  $h_{uv}^{(t)}$  using rules
For each  $(u,v)\in \mathcal{V}_G^2$  with  $d(u,v) = k$
$$
a _ {u v} ^ {i j k (t)} = \bigoplus_ {w \in \mathcal {N} _ {i} (u) \cap \mathcal {N} _ {j} (v)} m _ {i j k} ^ {(t)} \left(h _ {w v} ^ {(t - 1)}, h _ {u w} ^ {(t - 1)}\right), \tag {10}
$$
$$
h _ {u v} ^ {(t)} = f _ {k} ^ {(t)} \left(h _ {u v} ^ {(t - 1)}, \left(a _ {u v} ^ {i j k (t)}\right) _ {0 \leqslant i, j \leqslant d}\right), \tag {11}
$$
where  $m_{ijk}^{(t)}$  and  $f_{k}^{(t)}$  are arbitrary learnable functions;  $\bigoplus$  denotes a permutation-invariant aggregation operator (e.g., sum, mean or max). (10) and (11) are simply counterparts of (7) and (6) with  $\mathrm{POOL}_{ij}^{k(t)}$  and  $\mathrm{HASH}_k^{(t)}$  replaced with continuous functions.  $\sigma_t, t = 1,\dots,T - 1$  are entry-wise activation functions.  $R$  is a permutation-invariant readout function, whose input is the multiset
$\{h_{uv}^{(T)}: (u,v) \in \mathcal{V}_G^2$  and  $0 \leqslant d(u,v) \leqslant d\}$ . Finally,  $M$  is an MLP that acts on the graph representation output by  $R$ .
We can prove that (i) the representation power of any  $d$ -DRFWL(2) GNN is upper-bounded by the  $d$ -DRFWL(2) test, and (ii) there exists a  $d$ -DRFWL(2) GNN instance that has equal representation power to the  $d$ -DRFWL(2) test. We leave the formal statement and proof to Appendix B.
4 The cycle counting power of  $d$ -DRFWL(2) GNNs
By Theorem 3.2, the expressive power of  $d$ -DRFWL(2) tests (or  $d$ -DRFWL(2) GNNs) strictly increases with  $d$ . However, the space and time complexities of  $d$ -DRFWL(2) GNNs also increase with  $d$ . On one hand, since there are  $O(n \deg^k)$  distance- $k$  tuples in a graph  $G$ , at least  $O(n \deg^k)$  space is necessary to store the representations for all distance- $k$  tuples. For  $d$ -DRFWL(2) GNNs, this results in a space complexity of  $O(n \deg^d)$ . On the other hand, since there are at most  $O(\deg^{\min\{i,j\}})$  nodes in  $\mathcal{N}_i(u) \cap \mathcal{N}_j(v), \forall 0 \leqslant i, j \leqslant d$ , there are at most  $O(\deg^d)$  terms at the RHS of (10). Therefore, it takes  $O(d^2 \deg^d)$  time to update a single representation vector  $h_{uv}^{(t)}$  using (10) and (11). This implies the time complexity of  $d$ -DRFWL(2) GNNs is  $O(nd^2 \deg^{2d})$ .
For scalability,  $d$ -DRFWL(2) GNNs with a relatively small value of  $d$  are used in practice. But how to find the  $d$  value that best strikes a balance between expressive power and efficiency? To answer this question, we need a practical, quantitative metric of expressive power. In the following, we characterize the cycle counting power of  $d$ -DRFWL(2) GNNs. We find that 2-DRFWL(2) GNNs are powerful enough to node-level count up to 6-cycles, as well as many other useful graph substructures. Since for  $d = 2$ , the time and space complexities of  $d$ -DRFWL(2) GNNs are  $O(n \deg^4)$  and  $O(n \deg^2)$  respectively, our model is much more efficient than  $\mathrm{I}^2$ -GNN which requires  $O(n \deg^5)$  time and  $O(n \deg^4)$  space to count up to 6-cycles [33]. Moreover, with  $d \geqslant 3$ ,  $d$ -DRFWL(2) GNNs are able to node-level count up to 7-cycles, already matching the cycle counting power of FWL(2).