SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiarui Feng $^{1}$  Lecheng Kong $^{1}$  Hao Liu $^{1}$  Dacheng Tao $^{2}$  Fuhai Li $^{1}$  Muhan Zhang $^{3*}$  Yixin Chen $^{1}$
{feng.jiarui, jerry.kong, liuhao, fuhai.li, ychen25}@wustl.edu, dacheng.tao@gmail.com, muhan@pku.edu.cn
$^{1}$ Washington University in St. Louis  $^{2}$ JD Explore Academy  $^{3}$ Peking University
Message passing neural networks (MPNNs) have emerged as the most popular framework of graph neural networks (GNNs) in recent years. However, their expressive power is limited by the 1-dimensional Weisfeiler-Lehman (1-WL) test. Some works are inspired by  $k$ -WL/FWL (Folklore WL) and design the corresponding neural versions. Despite the high expressive power, there are serious limitations in this line of research. In particular, (1)  $k$ -WL/FWL requires at least  $O(n^{k})$  space complexity, which is impractical for large graphs even when  $k = 3$ ; (2) The design space of  $k$ -WL/FWL is rigid, with the only adjustable hyper-parameter being  $k$ . To tackle the first limitation, we propose an extension,  $(k, t)$ -FWL. We theoretically prove that even if we fix the space complexity to  $O(n^{k})$  (for any  $k \geq 2$ ) in  $(k, t)$ -FWL, we can construct an expressiveness hierarchy up to solving the graph isomorphism problem. To tackle the second problem, we propose  $k$ -FWL+, which considers any equivariant set as neighbors instead of all nodes, thereby greatly expanding the design space of  $k$ -FWL. Combining these two modifications results in a flexible and powerful framework  $(k, t)$ -FWL+. We demonstrate  $(k, t)$ -FWL+ can implement most existing models with matching expressiveness. We then introduce an instance of  $(k, t)$ -FWL+ called Neighborhood²-FWL (N²-FWL), which is practically and theoretically sound. We prove that N²-FWL is no less powerful than 3-WL, and can encode many substructures while only requiring  $O(n^{2})$  space. Finally, we design its neural version named N²-GNN and evaluate its performance on various tasks. N²-GNN achieves record-breaking results on ZINC-Subset (0.059) outperforming previous SOTA results by 10.6%. Moreover, N²-GNN achieves new SOTA results on the BREC dataset (71.8%) among all existing high-expressive GNN methods.
In recent years, graph neural networks (GNNs) have become one of the most popular and powerful methods for graph representation learning, following a message passing framework [1-4]. However, the expressive power of message passing GNNs is bounded by the one-dimensional Weisfeiler-Lehman (1-WL) test [5, 6]. As a result, numerous efforts have been made to design GNNs with higher expressive power. We provide a more detailed discussion in Section 5.
Several works have drawn inspiration from the  $k$ -dimensional Weisfeiler-Lehman ( $k$ -WL) or Folklore Weisfeiler-Lehman ( $k$ -FWL) test [7] and developed corresponding neural versions [6, 8-10]. However,  $k$ -WL/FWL has two inherent limitations. First, while the expressive power increases
with higher values of  $k$ , the space and time complexity also grows exponentially, requiring  $O(n^{k})$  space complexity and  $O(n^{k + 1})$  time complexity, which makes it impractical even when  $k = 3$ . Thus, the question arises: Can we retain high expressiveness without exploding both time and space complexities? Second, the design space of WL-based algorithms is rigid, with the only adjustable hyper-parameter being  $k$ . However, there is a significant gap in expressive power even between consecutive values of  $k$ , making it hard to fine-tune the tradeoffs. Moreover, increasing the expressive power does not necessarily translate into better real-world performance, as it may lead to overfitting [8, 11]. Although some works try to tackle this problem [8, 10], there is still limited understanding of how to expand the design space of the original  $k$ -FWL to a broader space that enables us to identify the most appropriate instance to match the complexity of real-world tasks.
To tackle the first limitation, we notice that  $k$ -FWL and  $k$ -WL have the same space complexity but  $k$ -FWL can achieve the same expressive power as  $(k + 1)$ -WL. We found the key component that allows  $k$ -FWL to have stronger power is the tuple aggregation style. Enlightened by this observation, we propose  $(k, t)$ -FWL, which extends the tuple aggregation style in  $k$ -FWL. Specifically, in the original  $k$ -FWL, a neighbor of a  $k$ -tuple is defined by iteratively replacing its  $i$ -th element with a node  $u$ , and  $u$  traverses all nodes in the graph. In  $(k, t)$ -FWL, we extend a single node  $u$  to a  $t$ -tuple of nodes and carefully design a replacement scheme to insert the  $t$ -tuple into a  $k$ -tuple to define its neighbor. We demonstrate that even with a fixed space complexity of  $O(n^{k})$  (for any  $k \geq 2$ ),  $(k, t)$ -FWL can construct an expressive hierarchy capable of solving the graph isomorphism problem. To deal with the second limitation, we revisit the definition of neighborhood in  $k$ -FWL. Inspired by previous works [8, 12] which consider only local neighbors (i.e.,  $u$  must be connected to the  $k$ -tuple) instead of global neighbors in  $k$ -WL/FWL, we find that the neighborhood (i.e., which  $u$  are used to construct a  $k$ -tuple's neighbors) can actually be extended to any equivariant set related to the  $k$ -tuple, resulting in  $k$ -FWL+. Combining the two modifications leads to a novel and powerful FWL-based algorithm  $(k, t)$ -FWL+.  $(k, t)$ -FWL+ is highly flexible and can be used to design different versions to fit the complexity of real-world tasks. Based on the proposed  $(k, t)$ -FWL+ framework, we implement many different instances that are closely related to existing powerful GNN/WL models, further demonstrating the flexibility of  $(k, t)$ -FWL+.
Finally, we propose an instance of (2, 2)-FWL+ named Neighborhood²-FWL that is both theoretically expressive and practically powerful. It considers the local neighbors of both two nodes in a 2-tuple. Despite having a space complexity of  $O(n^{2})$ , which is lower than 3-WL's  $O(n^{3})$  space complexity, this instance can still partially outperform 3-WL and is able to count many substructures. We implement a neural version named Neighborhood²-GNN ( $\mathbf{N}^2$ -GNN) and evaluate its performance on various synthetic and real-world datasets. Our results demonstrate that  $\mathbf{N}^2$ -GNN outperforms existing SOTA methods across most tasks. Particularly, it achieves 0.059 in ZINC-Subset, surpassing existing state-of-the-art models by significant margins. Meanwhile, it achieves 71.8% on the BREC dataset, the new SOTA among all existing high-expressive GNN methods.
Notations. Let  $\{\cdot\}$  denote a set,  $\{\cdot\}$  denote a multiset (a set that allows repetition), and  $(\cdot)$  denote a tuple. As usual, let  $[n] = \{1,2,\dots,n\}$ . Let  $G = (V(G),E(G),l_{G})$  be an undirected, colored graph, where  $V(G) = [n]$  is the node set with  $n$  nodes,  $E(G) \subseteq V(G) \times V(G)$  is the edge set, and  $l_{G} \colon V(G) \to C$  is the graph coloring function with  $C = \{c_1,\ldots ,c_d\}$  denote a set of  $d$  distinct colors. Let  $\mathcal{N}_k(v)$  denote a set of nodes within  $k$  hops of node  $v$  and  $Q_{k}(v)$  denote the  $k$ -th hop neighbors of node  $v$  and we have  $\mathcal{N}_k(v) = \bigcup_{i=0}^k Q_i(v)$ . Let  $\mathrm{SPD}(u,v)$  denote the shortest path distance between  $u$  and  $v$ . We use  $x_v \in \mathbb{R}^{d_n}$  to denote attributes of node  $v \in V(G)$  and  $e_{uv} \in \mathbb{R}^{d_e}$  to denote attributes of edge  $(u,v) \in E(G)$ . They are usually the one-hot encoding of the node and edge color respectively. We say that two graphs  $G$  and  $H$  are isomorphic (denoted as  $G \simeq H$ ) if there exists a bijection  $\varphi \colon V(G) \to V(H)$  such that  $\forall u,v \in V(G),(u,v) \in E(G) \Leftrightarrow (\varphi(u),\varphi(v)) \in E(H)$  and  $\forall v \in V(G),l_G(v) = l_H(\varphi(v))$ . Denote  $V(G)^k$  the set of  $k$ -tuples of vertices and  $\mathbf{v} = (v_1,\dots,v_k) \in V(G)^k$  a  $k$ -tuple of vertices. Let  $S_n$  denote the permutation group of  $[n]$  and  $g \in S_n : [n] \to [n]$  be a particular permutation. When a permutation  $g \in S_n$  operates on any target  $X$ , we denote it by  $g \cdot X$ . Particularly, a permutation operating on an edge set  $E(G)$  is  $g \cdot E(G) = \{(g(u),g(v))|(u,v) \in E(G)\}$ . A permutation operating on a  $k$ -tuple  $\mathbf{v}$  is  $g \cdot \mathbf{v} = (g(v_1),\dots,g(v_k))$ . A permutation operating on a graph is  $g \cdot G = (g \cdot V(G),g \cdot E(G),g \cdot l_G)$ .
$k$ -dimensional Weisfeiler-Lehman test. The  $k$ -dimensional Weisfeiler-Lehman ( $k$ -WL) test is a family of algorithms used to test graph isomorphism. There are two variants of the  $k$ -WL test:  $k$ -WL and  $k$ -FWL (Folklore WL). We first describe the procedure of 1-WL, which is also called the color refinement algorithm [7]. Let  $\mathcal{C}_{1wl}^{0}(v) = l_{G}(v)$  be the initial color of node  $v \in V(G)$ . At the  $l$ -th iteration, 1-WL updates the color of each node using the following equation:
$$
\mathcal {C} _ {1 w l} ^ {l} (v) = \operatorname {H A S H} \left(\mathcal {C} _ {1 w l} ^ {l - 1} (v), \{\{\mathcal {C} _ {1 w l} ^ {l - 1} (u) | u \in Q _ {1} (v) \}\}\right). \tag {1}
$$
After the algorithm converges, a color histogram is constructed using the colors assigned to all nodes. If the color histogram is different for two graphs, then the two graphs are non-isomorphic. However, if the color histogram is the same for two graphs, they can still be non-isomorphic.
The  $k$ -WL and  $k$ -FWL, for  $k \geq 2$ , are generalizations of the 1-WL, which do not color individual nodes but node tuples  $\mathbf{v} \in V(G)^k$ . Let  $\mathbf{v}_{w/j}$  be a  $k$ -tuple obtained by replacing the  $j$ -th element of  $\mathbf{v}$  with  $w$ . That is  $\mathbf{v}_{w/j} = (v_1, \ldots, v_{j-1}, w, v_{j+1}, \ldots, v_k)$ . The main difference between  $k$ -WL and  $k$ -FWL lies in their aggregation way. For  $k$ -WL, the set of  $j$ -th neighbors of tuple  $\mathbf{v}$  is denoted as  $Q_j(\mathbf{v}) = \{\mathbf{v}_{w/j} | w \in V(G)\}$ ,  $j \in [k]$ . Instead, the  $w$ -neighbor of tuple  $\mathbf{v}$  for  $k$ -FWL is denoted as  $Q_w^F(\mathbf{v}) = (\mathbf{v}_{w/1}, \mathbf{v}_{w/2}, \ldots, \mathbf{v}_{w/k})$ . Let  $\mathcal{C}_{kwl}^0(\mathbf{v}) = \mathcal{C}_{kflw}^0(\mathbf{v})$  be the initial color for  $k$ -WL and  $k$ -FWL, respectively. They are usually the isomorphism types of tuple  $\mathbf{v}$ . At the  $l$ -th iteration,  $k$ -WL and  $k$ -FWL update the color of each tuple according to the following equations:
$$
\mathbf {W L}: \quad \mathcal {C} _ {k w l} ^ {l} (\mathbf {v}) = \operatorname {H A S H} \left(\mathcal {C} _ {k w l} ^ {l - 1} (\mathbf {v}), \left(\{\{\mathcal {C} _ {k w l} ^ {l - 1} (\mathbf {u}) | \mathbf {u} \in Q _ {j} (\mathbf {v}) \} \} \mid j \in [ k ]\right)\right), \tag {2}
$$
$$
\mathbf {F W L}: \quad \mathcal {C} _ {k f w l} ^ {l} (\mathbf {v}) = \operatorname {H A S H} \left(\mathcal {C} _ {k f w l} ^ {l - 1} (\mathbf {v}), \{\{\left(\mathcal {C} _ {k f w l} ^ {l - 1} (\mathbf {u}) | \mathbf {u} \in Q _ {w} ^ {F} (\mathbf {v})\right) | w \in V (G) \} \}\right). \tag {3}
$$
The procedure described above is repeated until convergence, resulting in a color histogram of the graph that can be compared with the histograms of other graphs. Prior research has shown that  $(k + 1)$ -WL and  $(k + 1)$ -FWL are strictly more powerful than  $k$ -WL and  $k$ -FWL, respectively, except in the case where 1-WL is equivalent to 2-WL. Additionally, it has been shown that  $k$ -FWL is equivalent to  $(k + 1)$ -WL [13, 14, 6]. We leave the additional discussion on  $k$ -WL/FWL in Appendix A.
Message passing neural networks. Message Passing Neural Networks (MPNNs) are a type of GNNs that update node representations by iteratively aggregating information from their direct neighbors. Let  $h_v^l$  be the output of MPNNs of node  $v \in V(G)$  after  $l$  layers and we let  $h_v^0 = x_v$ . At  $l$ -th layer, the representation is updated by:
$$
\left. h _ {v} ^ {l} = \mathbf {U} ^ {l} \left(h _ {v} ^ {l - 1}, \{\{m _ {v u} ^ {l} | u \in Q _ {1} (v) \} \}\right), \quad m _ {v u} ^ {l} = \mathbf {M} ^ {l} \left(h _ {v} ^ {l - 1}, h _ {u} ^ {l - 1}, e _ {v u}\right), \right. \tag {4}
$$
where  $\mathbf{U}^l$  and  $\mathbf{M}^l$  are learnable update and message functions, usually parameterized by multi-layer perceptrons (MLPs). After  $L$  layers, MPNNs output the final representation  $h_v^L$  for all nodes  $v\in V(G)$ . The graph-level representation is obtained by:
$$
h _ {G} = \mathbf {R} \left(\left\{\left\{h _ {v} ^ {L} \mid v \in V (G) \right\} \right\}\right), \tag {5}
$$
where  $\mathbf{R}$  is a readout function. The expressive power of MPNNs is at most as powerful as 1-WL [5, 6].
3 Rethinking and extending the Folklore Weisfeiler-Lehman test
3.1 Rethinking and extending the aggregation style in  $k$ -FWL
Increasing  $k$  can increase the expressive power of  $k$ -FWL for distinguishing graph structures. However, increasing  $k$  brings significant memory costs, as  $k$ -FWL requires  $O(n^{k})$  memory. It becomes impractical even when  $k = 3$  [9]. Therefore, it is natural to ask:
Can we achieve higher expressive power without exploding the memory cost?
To achieve this goal, we first notice that  $k$ -FWL has a higher expressive power than  $k$ -WL but only requires the same  $O(n^{k})$  memory cost. To understand why, let's use 2-WL and 2-FWL as examples and rewrite Equation (2) and Equation (3):
2-WL:  $\mathcal{C}_{2wl}^{l}(v_{1},v_{2}) = \mathrm{HASH}\left(\mathcal{C}_{2wl}^{l - 1}(v_{1},v_{2}),\{\{\mathcal{C}_{2wl}^{l - 1}(v_{1},w)|w\in V(G)\} ,\{\{\mathcal{C}_{2wl}^{l - 1}(w,v_{2})|w\in V(G)\} \}\right),$
2-FWL:  $\mathcal{C}_{2fwl}^{l}(v_{1},v_{2}) = \mathrm{HASH}\left(\mathcal{C}_{2fwl}^{l - 1}(v_{1},v_{2}),\{\{\left(\mathcal{C}_{2fwl}^{l - 1}(v_{1},w),\mathcal{C}_{2fwl}^{l - 1}(w,v_{2})\right)|w\in V(G)\}\}\right),$
Figure 1: Illustration of the construction of neighborhood tuple  $Q_{(w_1, w_2)}^F(v_1, v_2)$  in (2, 2)-FWL. We sequentially select 0, 1, 2 elements from  $(w_1, w_2)$  to replace 0, 1, 2 elements in  $(v_1, v_2)$ , resulting in three sub-tuples of length 1, 4, 1, respectively. The final neighborhood tuple is the concatenation of three sub-tuples. We can easily recover high-order graph structures from the constructed neighborhood tuple with the isomorphism type of 2-tuples.
where  $\mathcal{C}_{2wl}^{l}(v_{1},v_{2})$  and  $\mathcal{C}_{2fwl}^{l}(v_{1},v_{2})$  are the color of tuple  $(v_{1},v_{2})$  at iteration  $l$  for 2-WL and 2-FWL, respectively. We can see the key difference between 2-WL and 2-FWL is that: in 2-FWL, a tuple of color  $(\mathcal{C}_{2fwl}^{l-1}(v_{1},w),\mathcal{C}_{2fwl}^{l-1}(w,v_{2}))$  is aggregated. While in 2-WL, the colors of nodes are considered in separate multisets. To understand why the first aggregation is more powerful, we can further rewrite the update equation of 2-FWL as follows:
$$
\mathcal {C} _ {2 f w l} ^ {l} (v _ {1}, v _ {2}) = \operatorname {H A S H} \left(\{\{\left(\mathcal {C} _ {2 f w l} ^ {l - 1} (v _ {1}, v _ {2}), \mathcal {C} _ {2 f w l} ^ {l - 1} (v _ {1}, w), \mathcal {C} _ {2 f w l} ^ {l - 1} (w, v _ {2})\right) | w \in V (G) \}\}\right).
$$
It is easy to verify that the above equation is equivalent to the original one. This means that 2-FWL updates the color of tuple  $(v_{1}, v_{2})$  by aggregating different tuples of  $((v_{1}, v_{2}),(v_{1}, w), (w, v_{2}))$ , which can be viewed as aggregating information of 3-tuples  $(v_{1}, v_{2}, w)$ . For example,  $\mathrm{HASH}(\mathcal{C}_{2fwl}^{0}(v_{1}, v_{2}), \mathcal{C}_{2fwl}^{0}(v_{1}, w), \mathcal{C}_{2fwl}^{0}(w, v_{2}))$  can fully recover the isomorphism type of tuple  $(v_{1}, v_{2}, w)$  used in 3-WL. The above observations can be easily extended to  $k$ -FWL. The key insight here is that tuple-style aggregation is the key to lifting the expressive power of  $k$ -FWL. Since aggregating  $k$ -tuple can boost the expressive power of  $k$ -FWL to be equivalent to  $(k+1)$ -WL without the increase of space complexity, we may wonder, can we further extend the size of the tuple?
We first introduce the neighborhood tuple to extend the original  $w$ -neighbor used in  $k$ -FWL. Let  $Q_{\mathbf{w}}^{F}(\mathbf{v})$  denote a neighborhood tuple related to  $k$ -tuple  $\mathbf{v}$  and  $t$ -tuple  $\mathbf{w}$ . The neighborhood tuple contains all possible results such that we sequentially select  $m \in [0,1,\dots,min(k,t)]$  elements from  $\mathbf{w}$  to replace  $m$  elements in  $\mathbf{v}$ . For each  $m$ , we first select all possible combinations of  $m$ -tuples (no repeated elements) from tuple  $\mathbf{w}$  with a pre-defined order to form a new tuple, denoted as  $P_{m}(\mathbf{w})$ . At the same time, we select all possible combinations of  $m$  indices from  $\mathbf{i} = (1,2,\dots,k)$  with a pre-defined order to form another new tuple  $P_{m}(\mathbf{i})$ . Finally, we iterate all  $(\mathbf{w}',\mathbf{i}') \in P_{m}(\mathbf{w}) \times P_{m}(\mathbf{i})$  to get a  $|P_{m}(\mathbf{w})||P_{m}(\mathbf{i})|$ -sub-tuple, where for each  $(\mathbf{w}',\mathbf{i}')$  we replace the elements with indices  $\mathbf{i}'$  in the original  $\mathbf{v}$  with  $\mathbf{w}'$ . The final neighborhood tuple is the concatenation of all sub-tuples. Note that the pre-defined orders can be flexible as long as they are consistent for any  $k$  and  $t$ .
Here is an example of a possible construction of a neighborhood tuple when  $k = t = 2$ . Let  $Q_{(w_1, w_2)}^F(v_1, v_2)$  denote the neighborhood tuple. In this case, we need to enumerate  $m$  in  $[0, 1, 2]$ . First, when  $m = 0$ , we select 0 elements from  $(w_1, w_2)$  to replace  $(v_1, v_2)$ , resulting in a sub-tuple with only one element  $((v_1, v_2))$ . Next, when  $m = 1$ , we sequentially select  $w_1$  and  $w_2$ , resulting in  $P_1(\mathbf{w}) = (w_1, w_2)$ . Similarly, we can get  $P_1(\mathbf{i}) = (2, 1)$ . Thus, the final sub-tuple is  $((v_1, w_1), (v_1, w_2), (w_1, v_2), (w_2, v_2))$ . The case of  $m = 2$  is simple and we have the result sub-tuple be  $((w_1, w_2))$ . By concatenating all three sub-tuples, we have  $Q_{(w_1, w_2)}^F(v_1, v_2) = ((v_1, v_2), (v_1, w_1), (v_1, w_2), (w_1, v_2), (w_2, v_2), (w_1, w_2))$ . The construction procedure is shown in Figure 1. By default, we adopt the above order in  $Q_{\mathbf{w}}^F(\mathbf{v})$  in the rest of the paper if  $k = t = 2$ . We leave a more formal definition of the neighborhood tuple  $Q_{\mathbf{w}}^F(\mathbf{v})$  in Appendix A.
With the neighborhood tuple, we are ready to introduce  $(k,t)$ -FWL, which extends  $k$ -FWL by extending the tuple size in the update function. Let  $\mathcal{C}_{ktfwl}^{l}(\mathbf{v})$  be the color of tuple  $\mathbf{v}$  at iteration  $l$  for  $(k,t)$ -FWL, the update function of  $(k,t)$ -FWL is:
$$
(k, t) \text {- F W L}: \mathcal {C} _ {k t f w l} ^ {l} (\mathbf {v}) = \operatorname {H A S H} \left(\mathcal {C} _ {k t f w l} ^ {l - 1} (\mathbf {v}), \{\{\left(\mathcal {C} _ {k t f w l} ^ {l - 1} (\mathbf {u}) | \mathbf {u} \in Q _ {\mathbf {w}} ^ {F} (\mathbf {v})\right) | \mathbf {w} \in V ^ {t} (G) \} \} _ {t}\right), \tag {6}
$$
where  $\{\{\cdot\}\}_{t}$  is hierarchical multiset over  $t$ -tuples. In a hierarchical multiset  $\{\{\mathbf{v}|\mathbf{v}\in V^t (G)\} _t\}$ , elements are grouped hierarchically according to the node order of the tuple. For example, to construct  $\{(v_{1},v_{2},v_{3})|(v_{1},v_{2},v_{3})\in V^{3}(G)\}_{3}$  from  $\{(v_1,v_2,v_3)|(v_1,v_2,v_3)\in V^3 (G)\}$ , we first group together all elements with the same  $v_{2}$  and  $v_{3}$ . That is,  $\forall v_{2},v_{3}\in V(G)$ , we denote  $t(v_{2},v_{3}) = \{(v_{1},v_{2},v_{3})|v_{1}\in V(G)\}$  as the grouped result. Next, use the similar procedure, we have  $\forall v_{3}\in V(G)$ ,  $t(v_{3}) = \{(t(v_{2},v_{3})|v_{2}\in V(G)\}$ . Finally, we group all possible  $v_{3}\in V(G)$  to get  $\{(v_{1},v_{2},v_{3})|(v_{1},v_{2},v_{3})\in V^{3}(G)\}_{3} = \{(t(v_{3})|v_{3}\in V(G)\}$ .
It is easy to see  $(k,1)$ -FWL is equivalent to  $k$ -FWL under our definition of  $Q_{\mathbf{w}}^{F}(\mathbf{v})$ . Further, as we only need to maintain representations of all  $k$ -tuples,  $(k,t)$ -FWL has a fixed space complexity of  $O(n^{k})$ . Here we show the expressive power of  $(k,t)$ -FWL.
Proposition 3.1. For  $k \geq 2$  and  $t \geq 1$ , if  $t \geq n - k$ ,  $(k, t)$ -FWL can solve the graph isomorphism problems with the size of the graph less than or equal to  $n$ .
Theorem 3.2. For  $k \geq 2, t \geq 1$ ,  $(k, t)$ -FWL is at most as powerful as  $(k + t)$ -WL. In particular,  $(k, 1)$ -FWL is as powerful as  $(k + 1)$ -WL.
Proposition 3.3. For  $k \geq 2, t \geq 1$ ,  $(k, t + 1)$ -FWL is strictly more powerful than  $(k, t)$ -FWL;  $(k + 1, t)$ -FWL is strictly more powerful than  $(k, t)$ -FWL.
We leave all formal proofs and complexity analysis in Appendix B. Briefly speaking, even if we fix the size of  $k$ ,  $(k,t)$ -FWL can still construct an expressive hierarchy by varying  $t$ . Further, if  $t$  is large enough,  $(k,t)$ -FWL can actually enumerate all possible combinations of tuples with the size of the graph, and thus equivalent to the relational pooling on graph [15]. It is worth noting that the size of  $Q_{\mathbf{w}}^{F}(\mathbf{v})$  will grow exponentially with an increase in the size of  $t$ . However, the key contribution of  $(k,t)$ -FWL is that even when  $k = 2$ ,  $(k,t)$ -FWL can still construct an expressive hierarchy for solving graph isomorphism problems. Therefore, high-order embedding may not be necessary for building high-expressivity WL algorithms. Note that our  $(k,t)$ -FWL is also different from subgraph GNNs such as  $k,l$ -WL [16] and  $l$ -OSAN [17], where  $l$ -tuples are labeled independently to enable learning  $k$ -tuple representations in all  $l$ -tuples' subgraphs, resulting in  $O(n^{k + l})$  space complexity.
3.2 Rethinking and extending the aggregation scope of  $k$ -FWL
Another problem of  $k$ -FWL is its limited design space, as the only adjustable hyperparameter is  $k$ . It is well known that there is a huge gap in expressive power even if we increase from  $k$  to  $k + 1$ . For example, 1-WL cannot count any cycle even with a length of 3, but 2-FWL can already count up to 7-cycle [18-20]. Moreover, increasing the expressive power does not always bring better performance when designing the corresponding neural version as it quickly leads to overfitting [6, 9, 11]. Therefore, we ask another question:
Can we extend the  $k$ -FWL to a more flexible and fine-grained design space?
To address this issue, we identify that the inflexibility of  $k$ -FWL's design space arises from the definition of the neighbor used in the aggregation step. Unlike 1-WL,  $k$ -FWL lacks the concept of local neighbors and instead requires the aggregation of all  $|V(G)|$  global neighbors to update the color of each tuple  $\mathbf{v}$ . Recently, some works have extended  $k$ -WL by incorporating local information [8, 12]. Inspired by previous works, we find that the definition of neighbor can actually be much more flexible than just considering local neighbors or global neighbors. Specifically, for each  $k$ -tuple  $\mathbf{v}$  in graph  $G$ , we define equivariant set  $ES(\mathbf{v})$  to be the neighbors set of tuple  $\mathbf{v}$  and propose  $k$ -FWL+.
Definition 3.4. An equivariant set  $ES(\mathbf{v})$  is a set of nodes related to  $\mathbf{v}$  and equivariant given the permutation  $g \in S_n$ . That is,  $\forall w \in ES(\mathbf{v})$  in graph  $G$  implies  $g(w) \in ES(g \cdot \mathbf{v})$  in graph  $g \cdot G$ .
Some nature equivariant sets  $ES(v)$  including  $V(G), \mathcal{N}_k(v)$ , and  $Q_{k}(v)$ , etc. Let  $\mathcal{C}_{kfwl + }^{l}(\mathbf{v})$  be the color of tuple  $\mathbf{v}$  at iteration  $l$  for  $k$ -FWL+, we have: