SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
k - \mathbf {F W L} +: \quad \mathcal {C} _ {k f w l +} ^ {l} (\mathbf {v}) = \operatorname {H A S H} \left(\mathcal {C} _ {k f w l +} ^ {l - 1} (\mathbf {v}), \{\{\left(\mathcal {C} _ {k f w l +} ^ {l - 1} (\mathbf {u}) | \mathbf {u} \in Q _ {w} ^ {F} (\mathbf {v})\right) | w \in E S (\mathbf {v}) \}\}\right). \tag {7}
$$
The key of  $k$ -FWL+ is that the equivariant set  $ES(\mathbf{v})$  can be any set of nodes as long as it is equivariant to any permutation  $g \in S_n$ . For example, if  $ES(\mathbf{v}) = V(G)$ , the  $k$ -FWL+ will reduce to the original  $k$ -FWL. Instead, if  $ES(\mathbf{v}) = \bigcup_{i=0}^{k} \mathcal{N}_1(v_i)$ , it becomes the localized  $k$ -FWL [12]. We can also design other more innovative equivariant sets  $ES(\mathbf{v})$ . For example, denote  $\mathcal{SP}(\mathbf{v})$  to be a set that contains all nodes in the shortest paths between any  $v_i, v_j \in \mathbf{v}$ . It is still a valid equivariant set of tuple  $\mathbf{v}$ . We can see that the design space of  $k$ -FWL+ is much broader than  $k$ -FWL.
3.3  $(k,t)$ -FWL+: A flexible and powerful extension of  $k$ -FWL
In this section, we propose  $(k,t)$ -FWL+, which combines both  $(k,t)$ -FWL in Equation (6) and  $k$ -FWL+ in Equation (7). Let  $\mathcal{C}_{ktf+}^{l}(\mathbf{v})$  be the color of tuple  $\mathbf{v}$  at iteration  $l$  for  $(k,t)$ -FWL+, the color update equation of  $(k,t)$ -FWL+ is:
$$
(k, t) \text {- F W L + :} \quad \mathcal {C} _ {k t f +} ^ {l} (\mathbf {v}) = \operatorname {H A S H} \left(\mathcal {C} _ {k t f +} ^ {l - 1} (\mathbf {v}), \{\{\left(\mathcal {C} _ {k t f +} ^ {l - 1} (\mathbf {u}) | \mathbf {u} \in Q _ {\mathbf {w}} ^ {F} (\mathbf {v})\right) | \mathbf {w} \in E S ^ {t} (\mathbf {v}) \} \} _ {t}\right), \tag {8}
$$
where  $ES^t(\mathbf{v}) = ES_1(\mathbf{v}) \times ES_2(\mathbf{v}) \times \dots \times ES_t(\mathbf{v})$  and  $ES_i(\mathbf{v})$  can be any  $ES(\mathbf{v})$ .  $(k, t)$ -FWL+ integrates advantages from both  $(k, t)$ -FWL and  $k$ -FWL+. First, by extending  $V^t(G)$  to  $ES^t(\mathbf{v})$ , we can hugely reduce the time complexity of  $(k, t)$ -FWL as  $ES^t(\mathbf{v})$  usually contains much less elements than  $V^t(G)$ . Second, by extending the tuple size in aggregation, we only require a small  $k$  (even with  $k = 2$ ) to achieve high expressive power instead of being bounded by  $k$ -FWL. Finally, the most important advantage is that the design space of  $(k, t)$ -FWL+ is much more flexible than  $k$ -FWL and we can design the most suitable instance for different tasks. However, as the expressive power of  $(k, t)$ -FWL+ highly relies on the choice of  $ES^t(\mathbf{v})$ , it is hard to analyze the expressive power of  $(k, t)$ -FWL+ in a systematic way. Instead, in the next section, we provide some practical implementations of  $(k, t)$ -FWL+ that can cover many existing powerful models.
3.4 Gallery of  $(k,t)$ -FWL+ instances
In this section, we provide some practical designs of  $ES^t(\mathbf{v})$  that are closely correlated to many existing powerful GNN/WL models. Note that any model directly follows the original  $k$ -FWL like PPGN [9] can obviously be implemented by  $(k, t)$ -FWL+ as  $(k, t)$ -FWL+ also includes the original  $k$ -FWL. We specify that  $k$ -tuple  $\mathbf{v} = (v_1, v_2, \ldots, v_k)$  to avoid any confusion.
Proposition 3.5. Let  $t = 1, k = 2$ , and  $ES(\mathbf{v}) = \mathcal{N}_1(v_1) \cup \mathcal{N}_1(v_2)$ , the corresponding  $(k, t)$ -FWL+ instance is equivalent to SLFWL(2) [12] and strictly more powerful than any existing node-based subgraph GNNs.
Proposition 3.6. Let  $t = 1$ ,  $k = k$ , and  $ES(\mathbf{v}) = \bigcup_{i=1}^{k} Q_1(v_i)$ , the corresponding  $(k, t)$ -FWL+ instance is more powerful than  $\delta$ - $k$ -LWL [8].
Proposition 3.7. Let  $t = 2$ ,  $k = 2$ , and  $ES^2(\mathbf{v}) = (Q_1(v_1) \cap Q_1(v_2)) \times (Q_1(v_1) \cap Q_1(v_2))$ , the corresponding  $(k, t)$ -FWL+ instance is more powerful than GraphSNN [21].
Proposition 3.8. Let  $t = 2$ ,  $k = 2$ , and  $ES^2(\mathbf{v}) = Q_1(v_2) \times Q_1(v_1)$ , the corresponding  $(k, t)$ -FWL+ instance is more powerful than edge-based subgraph GNNs like  $I^2$ -GNN [19].
Proposition 3.9. Let  $t = 1$ ,  $k = 2$ , and  $ES(\mathbf{v}) = Q_{SPD(v_1,v_2)}(v_1) \cap Q_1(v_2)$ , the corresponding  $(k,t)$ -FWL+ instance is at most as powerful as KP-GNN [22] with the peripheral subgraph encoder as powerful as 1-WL.
Proposition 3.10. Let  $t = 2$ ,  $k = 2$ , and  $ES^2(\mathbf{v}) = Q_1(v_2) \times \mathcal{SP}(v_1, v_2)$ , the corresponding  $(k, t)$ -FWL+ is more powerful than GDGNN [23].
We leave all formal proofs and discussion on complexity in Appendix C. Significantly, Proposition 3.8 indicates that  $(k,t)$ -FWL+ can be more powerful than edge-based subgraph GNNs but only require  $O(n^{2})$  space, strictly lower than  $O(nm)$  space used in edge-based subgraph GNNs, where  $m$  is the number of edges in the graph. However, most existing works employ MPNNs as their basic encoder, which differs from the tuple aggregation in  $(k,t)$ -FWL+. Thus it is still non-trivial to find an instance that can exactly fit many existing works. Nevertheless, given the strict hierarchy between node-based subgraph GNNs and SLFWL(2) [12], we conjecture there also exists a strict hierarchy between the  $(k,t)$ -FWL+ instances and corresponding existing works based on MPNNs and leave the detailed proof in our future works.
4 Neighborhood $^2$ -GNN: a practical and powerful (2, 2)-FWL+ implementation
Due to the extremely flexible and broad design space of  $(k, t)$ -FWL+, it is hard to evaluate all possible instances. Therefore, we focus on a particular instance that is both theoretically and practically sound. All detailed proof and complexity analysis related to this section can be found in Appendix D.
4.1 Neighborhood2-FWL
In this section, we propose a practical and powerful implementation of (2,2)-FWL+ named Neighborhood $^2$ -FWL  $(\mathbf{N}^2$ -FWL). Let  $\mathcal{C}_{n^2 fwl}^l (\mathbf{v})$  denote the color of tuple  $\mathbf{v}$  at iteration  $l$  for  $\mathbf{N}^2$ -FWL, the color is updated by:
$$
\mathbf {N} ^ {2} - \mathbf {F W L}: \quad \mathcal {C} _ {n ^ {2} f w l} ^ {l} (\mathbf {v}) = \operatorname {H A S H} \left(\mathcal {C} _ {n ^ {2} f w l} ^ {l - 1} (\mathbf {v}), \{\{\left(\mathcal {C} _ {n ^ {2} f w l} ^ {l - 1} (\mathbf {u}) | \mathbf {u} \in Q _ {\mathbf {w}} ^ {F} (\mathbf {v})\right) | \mathbf {w} \in N ^ {2} (\mathbf {v}) \} \} _ {2}\right), \tag {9}
$$
where  $N^2 (\mathbf{v}) = (\mathcal{N}_1(v_2)\times \mathcal{N}_1(v_1))\cap (\mathcal{N}_h(v_1)\cap \mathcal{N}_h(v_2))^2$  and  $h$  is the number of hop. Briefly speaking,  $\mathbf{N}^2$ -FWL considers neighbors of both node  $v_{1}$  and  $v_{2}$  within the overlapping  $h$ -hop subgraph between  $v_{1}$  and  $v_{2}$ . In the following, we show the expressive power and counting power of  $\mathbf{N}^2$ -FWL.
Theorem 4.1. Given  $h$  is large enough,  $N^2$ -FWL is more powerful than SLFWL(2) [12] and edge-based subgraph GNNs.
Corollary 4.2.  $N^2$ -FWL is strictly more powerful than all existing node-based subgraph GNNs and no less powerful than 3-WL.
Theorem 4.3.  $N^2$ -FWL can count up to (1) 6-cycles; (2) all connected graphlets with size 4; (3) 4-paths at node level.
4.2 Neighborhood $^{2}$ -GNN
In this section, we provide a neural version of  $\mathbf{N}^2$ -FWL called Neighborhood $^2$ -GNN ( $\mathbf{N}^2$ -GNN). Let  $h_{\mathbf{v}}^{l}$  be the output of  $\mathbf{N}^2$ -GNN of tuple  $\mathbf{v}$  at layer  $l$  and  $h_{\mathbf{v}}^{0}$  encode the isomorphism type of tuple  $\mathbf{v}$ . At the  $l$ -th layer, the representation of each tuple is updated by:
$$
\left. h _ {\mathbf {v}} ^ {l} = \mathbf {U} ^ {l} \left(h _ {\mathbf {v}} ^ {l - 1}, \{\left\{m _ {\mathbf {v w}} ^ {l} \mid \mathbf {w} \in N ^ {2} (\mathbf {v}) \right\} \} _ {2}\right), \quad m _ {\mathbf {v w}} ^ {l} = \mathbf {M} ^ {l} \left(\left(h _ {\mathbf {u}} ^ {l - 1} \mid \mathbf {u} \in Q _ {\mathbf {w}} ^ {F} (\mathbf {v})\right)\right). \right. \tag {10}
$$
After  $L$  layers of message passing, the final graph representation is obtained by:
$$
h _ {G} = \mathbf {R} \left(\left\{\left\{h _ {\mathbf {v}} ^ {L} \mid \mathbf {v} \in V ^ {2} (G) \right\} \right\} _ {2}\right). \tag {11}
$$
We leave the detailed implementation in Appendix D. Finally, we show the expressive power of  $\mathrm{N}^2$ -GNN as follows:
Proposition 4.4. If  $\mathbf{U}^l$ ,  $\mathbf{M}^l$ , and  $\mathbf{R}$  are injective functions, there exist a  $N^2$ -GNN instance that can be as powerful as  $N^2$ -FWL.
After the expressive power of MPNNs has been demonstrated to be limited by the 1-WL test [5, 6], significant research efforts have focused on breaking the limitations of MPNNs.
Subgraph-based methods. Subgraph-based methods aim to address the limitation of MPNNs by leveraging the fact that although some graphs may be difficult to distinguish, they contain distinct substructures that can be easily differentiated by MPNNs. To this end, various subgraph selection policies and aggregation strategies have been proposed to enhance the expressive power of MPNNs. These include K-ego subgraphs of nodes [24-28], distance labeling [29], graph elements deletion [30, 31], edge-based subgraph selection [19, 32, 33], etc. ESAN [34] summarizes different subgraph selection policies and proposes a general framework for subgraph-based GNNs.  $l$ -OSAN [17] further extends the subgraph-based model to  $l$ -tuples and  $k$ ,  $l$ -WL [16] considers running  $k$ -WL on  $l$ -tuples subgraphs. KP-GNN [22] takes a slightly different view by considering the peripheral subgraph at each hop of nodes. Recently, Frasca et al. [27] proved that all node-based subgraph GNNs are bounded by 3-WL test. Zhang et al. [12] further show that they are even strictly less powerful than localized 2-FWL, highlighting the inherent limitations of neighbor aggregation-based methods compared to tuple-based aggregation.
WL-based methods. WL-based methods try to simulate  $k$ -WL/FWL test. In particular, Morris et al. [6] proposed  $k$ -GNNs to approximate  $k$ -WL test. To reduce the complexity of  $k$ -GNNs, Morris et al. [8] considered only local neighbors. Similarly, PPGN [9] adopted  $k$ -FWL instead of  $k$ -WL and proposed a tensor-based model with 3-WL power but requiring only  $O(n^2)$  space. To further reduce complexity, KC-SetGNN [10] leveraged node sets rather than tuples and considered only
connected node sets. However, these approaches can still be impractical despite careful design. Our work hugely extends the design space of FWL-based methods and demonstrates that the resulting model can be expressive even with a lower space complexity cost.
Permutation equivariant and invariant networks. This line of research aims to leverage the permutation equivariant and invariant properties of graphs and devise corresponding architectures. For example, Maron et al. [35] proposed  $k$ -IGNs, which employ permutational equivariant linear layers with point-wise activation functions to the adjacency matrix.  $k$ -IGNs have been proven to have the same expressive power as  $k$ -WL [9, 36, 37]. Another approach utilizes the Reynold operator to consider all possible permutations on either global [15] or local [20] levels. However, they consider all possible permutations in order to achieve universality, which may lead to overfitting in real-world tasks and are more valuable from a theoretical perspective. In contrast, our framework is highly flexible and can be tailored to fit the complexity of real-world tasks.
Feature-based methods. Feature-based methods aim to incorporate graph-related features that cannot be computed by MPNNs. For instance, some methods add random features to nodes to break the symmetry [38, 39]. GSN [40] introduces substructure counting features into MPNNs. GD-WL [41] applies general distance features to enhance the expressiveness of MPNNs and shows it can solve the graph biconnectivity problem. GDGNN [23] integrates the geodesic information between node pairs and achieves a great balance between performance and efficiency. ESC-GNN [42] employs subgraph features to simulate subgraph GNNs without running them explicitly. Puny et al. [43] computes graph polynomial features and injects them into PPGN to achieve greater expressive power than 3-WL within  $O(n^2)$  space. Although feature-based methods are highly efficient, they often suffer from overfitting in real-world tasks and produce suboptimal results.
In this section, we conduct various experiments on synthetic and real-world tasks to verify the effectiveness of  $\mathbf{N}^2$ -GNN. The details of all experiments can be found in Appendix E. Additional experimental results and ablation studies are included in Appendix F. The source code is provided in https://github.com/JiaruiFeng/N2GNN.
6.1 Expressive power
Table 1: Expressive power verification (Accuracy).
Datasets	EXP	CSL	SR25
N2-GNN	100	100	100
Datasets. To verify the expressive power of  $\mathbf{N}^2$ -GNN, we select four synthetic datasets, all containing graphs that cannot be distinguished by 1-WL/MPNNs. The datasets we selected are (1) EXP [38], which consists of 600 pairs of non-isomorphic graphs that 1-WL fails to distinguish; (2) CSL [15], which contains 150 4-regular graphs divided into 10 isomorphism classes; and (3) SR25 [44], which
contains 15 non-isomorphic strongly regular graphs, each has 25 nodes. Even 3-WL is unable to distinguish between these graphs. (4) BREC [45], which contains 400 non-isomorphic graph pairs that range from 1-WL-indistinguishable to 4-WL-indistinguishable.
Models. For EXP, CSL, and SR25, we only report results for  $\mathrm{N}^2$ -GNN. For BREC, we compare  $\mathrm{N}^2$ -GNN with  $\mathrm{I}^2$ -GNN [19] as it is the current SOTA on BREC among all GNN methods.
Results. We report results in Table 1 and Table 2. For EXP and CSL, we report the average accuracy for 10-time cross-validation; For SR25, we report single-time accuracy. We can see  $\mathbf{N}^2$ -GNN achieves perfect results on all three datasets, empirically verifying its expressive power. Notably,  $\mathbf{N}^2$ -GNN is able to distinguish strongly regular graphs. This empirically verified Corollary 4.2. Moreover,  $\mathbf{N}^2$ -GNN achieves the best result on the BREC dataset among all GNNs methods, surpassing the previous SOTA,  $\mathbf{I}^2$ -GNN (See the complete comparison and further discussion in Appendix F). Despite this improved performance,  $\mathbf{N}^2$ -GNN only requires at most  $O(n^{2})$  space complexity, which is less than 3-WL. This further demonstrates the superiority of  $\mathbf{N}^2$ -GNN.
Table 2: Pair distinguishing accuracies on BREC
Model	Basic Graphs (60)	Regular Graphs (140)	Extension Graphs (100)	CFI Graphs (100)	Total (400)
Number	Accuracy	Number	Accuracy	Number	Accuracy	Number	Accuracy	Number	Accuracy
I2-GNN	60	100%	100	71.4%	100	100%	21	21%	281	70.2%
N2-GNN	60	100%	100	71.4%	100	100%	27	27%	287	71.8%
Table 3: Evaluation on Counting Substructures (norm MAE), cells with MAE less than 0.01 are colored.
Target	ID-GNN [28]	NGNN [24]	GIN-AK+ [25]	PPGN [9]	I²-GNN [19]	N²-GNN
Tailed Triangle	0.1053	0.1044	0.0043	0.0026	0.0011	0.0025
Chordal Cycle	0.0454	0.0392	0.0112	0.0015	0.0010	0.0019
4-Clause	0.0026	0.0045	0.0049	0.1646	0.0003	0.0005
4-Path	0.0273	0.0244	0.0075	0.0041	0.0041	0.0042
Tri.-Rec.	0.0628	0.0729	0.1311	0.0144	0.0013	0.0055
3-Cycles	0.0006	0.0003	0.0004	0.0003	0.0003	0.0002
4-Cycles	0.0022	0.0013	0.0041	0.0009	0.0016	0.0024
5-Cycles	0.0490	0.0402	0.0133	0.0036	0.0028	0.0039
6-Cycles	0.0495	0.0439	0.0238	0.0071	0.0082	0.0075
6.2 Substructure counting
Datasets. To verify the substructure counting power of  $\mathbf{N}^2$ -GNN, we select the synthetic dataset from [25, 19]. The dataset consists of 5000 randomly generated graphs from different distributions, which are split into training, validation, and test sets with a ratio of  $0.3 / 0.2 / 0.5$ . The task is to perform node-level counting regression. Specifically, we choose tailed-triangle, chordal cycles, 4-cliques, 4-paths, triangle-rectangle, 3-cycles, 4-cycles, 5-cycles, and 6-cycles as target substructures.
Models. We compare  $\mathbf{N}^2$ -GNN with the following baselines: Identity-aware GNN (ID-GNN) [28], Nested GNN (NGNN) [24], GNN-AK+ [25], PPGN [9], I²-GNN [19]. Results for all baselines are reported from [19].
Results. We report the results for counting substructures in Table 3. All results are the average normalized test MAE of three runs with different random seeds. We colored all results that are less than 0.01, which is an indication of successful counting, the same as in [19]. We can see that  $\mathrm{N}^2$ -GNN achieves MAE less than 0.01 among all different substructures/cycles, which empirically verified Theorem 4.3. Specifically,  $\mathrm{N}^2$ -GNN achieves the best result on 4-path counting and comparable performance to  $\mathrm{I}^2$ -GNN on most substructures. Moreover,  $\mathrm{N}^2$ -GNN can count 4-clique extremely well, which is theoretically and empirically infeasible for 3-WL [42] and PPGN [9].
Table 4: MAE results on QM9 (smaller the better).
Target	DTNN [46]	MPNN [46]	PPGN [9]	NGNN [24]	KP-GIN' [22]	I²-GNN [19]	N²-GNN
μ	0.244	0.358	0.231	0.433	0.358	0.428	0.333
α	0.95	0.89	0.382	0.265	0.233	0.230	0.193
εHOMO	0.00388	0.00541	0.00276	0.00279	0.00240	0.00261	0.00217
εLUMO	0.00512	0.00623	0.00287	0.00276	0.00236	0.00267	0.00210
Δε	0.0112	0.0066	0.00406	0.00390	0.00333	0.00380	0.00304
〈R²〉	17.0	28.5	16.7	20.1	16.51	18.64	14.47
ZPVE	0.00172	0.00216	0.00064	0.00015	0.00017	0.00014	0.00013
U0	2.43	2.05	0.234	0.205	0.0682	0.211	0.0247
U	2.43	2.00	0.234	0.200	0.0696	0.206	0.0315
H	2.43	2.02	0.229	0.249	0.0641	0.269	0.0182
G	2.43	2.02	0.238	0.253	0.0484	0.261	0.0178
Cv	0.27	0.42	0.184	0.0811	0.0869	0.0730	0.0760
6.3 Molecular properties prediction
Datasets. To evaluate the performance of  $\mathbf{N}^2$ -GNN on real-world tasks, we select two popular molecular graphs datasets: QM9 [46, 50] and ZINC [51]. QM9 dataset contains over 130K molecules with 12 different molecular properties as the target regression task. The dataset is split into training, validation, and test sets with a ratio of  $0.8 / 0.1 / 0.1$ . The ZINC dataset has two variants: ZINC-subset (12k graphs) and ZINC-full (250k graphs), and the task is graph regression. The training, validation, and test splits for the ZINC datasets are provided.
Models. For QM9, we report baseline results of DTNN and MPNN from [46]. We further adopt PPGN [9], NGNNs [24], KP-GIN/ [22], I²-GNN [19]. We report results of PPGN, NGNN, and KP-GIN/ from [22] and results of I²-GNN from [19]. For ZINC, the baseline including CIN [47], δ-2-GNN [8], KC-SetGNN [10], PPGN [9], Graphormer-GD [41], GPS [48], Specformer [49], NGNN [24], GNN-AK-ctx [25], ESAN [34], SUN [27], KP-GIN/ [22], I²-GNN [19], SSWL+ [12]. We report results of all baselines from [12, 22, 43, 10, 48, 49, 19].
Table 5: MAE results on ZINC (smaller the better).
Model	#Param	ZINC-Subset Test MAE	ZINC-Full Test MAE
CIN [47]	~100k	0.079 ± 0.006	0.022 ± 0.002
δ-2-GNN [8]	-	-	0.042 ± 0.003
KC-SetGNN [10]	-	0.075 ± 0.003	-
PPGN [9]	-	0.079 ± 0.005	0.022 ± 0.003
GD-WL [41]	503k	0.081 ± 0.009	0.025 ± 0.004
GPS [48]	424k	0.070 ± 0.004	-
Specformer [49]	~500k	0.066 ± 0.003	-
NGNN [24]	~500k	0.111 ± 0.003	0.029 ± 0.001
GNN-AK [25]	~500k	0.093 ± 0.002	-
ESAN [34]	446k	0.097 ± 0.006	0.025 ± 0.003
SUN [27]	526k	0.083 ± 0.003	0.024 ± 0.003
KP-GIN' [22]	489k	0.093 ± 0.007	-
I²-GNN [19]	-	0.083 ± 0.001	0.023 ± 0.001
SSWL+ [12]	387k	0.070 ± 0.005	0.022 ± 0.002
N²-GNN	316k/414k	0.059 ± 0.002	0.022 ± 0.002
Results. For QM9, we report the single-time final test MAE in Table 4. We can see that  $\mathrm{N}^2$ -GNN outperforms all other methods in almost all targets with a large margin. Particularly, we achieve  $63.8\%$ ,  $54.7\%$ ,  $71.6\%$ , and  $63.2\%$  improvement comparing to the second-best results on target  $U_0$ ,  $U$ ,  $H$ , and  $G$  respectively. For ZINC-Subset and ZINC-Full, we report the average test MAE and standard deviation of 10 runs with different random seeds in Table 5. We can see  $\mathrm{N}^2$ -GNN still surpasses all previous methods with only the smallest model parameter size among all baselines. It achieves  $10.6\%$  improvement over the second-best methods on ZINC-Subset and the same performance over the previous SOTA on ZINC-full. All these results demonstrate the superiority of  $\mathrm{N}^2$ -GNN on real-world tasks.
7 Limitations, future works, and conclusions
Limitations: For  $(k,t)$ -FWL, although we can control the space complexity by fixing the  $k$ , the time complexity of  $(k,t)$ -FWL will grow exponentially with the increase of  $t$  (as discussed in Appendix B). Therefore, the resulting model can still be impractical when  $t$  is large. We believe this result is evidence of the "no free lunch" in developing a more expressive GNN model. For  $(k,t)$ -FWL+, although we demonstrate its capability by implementing many instances corresponding to different existing GNN models, it is still unclear what is the whole space of  $(k,t)$ -FWL+, especially the space of  $ES(\mathbf{v})$ . Moreover, the quantitative expressiveness analysis of  $(k,t)$ -FWL+ is still unexplored. Finally, for  $\mathrm{N}^2$ -GNN, the practical complexity can still be unbearable, especially for dense graphs if the aggregation is implemented in a parallel way, as we need to aggregate many more neighbors for each tuple than normal MPNN. It also introduces the optimization issue on  $\mathrm{N}^2$ -GNN, which makes it hard to achieve its theoretical expressiveness. In Appendix B, C, and D, we provide more detailed discussion on the limitation of  $(k,t)$ -FWL,  $(k,t)$ -FWL+, and  $\mathrm{N}^2$ -GNN, respectively.
Future works: There are several directions that are worth further exploration. First, can we characterize the whole space of  $ES(\mathbf{v})$  and thus can theoretically and quantitatively analyze the expressive power of  $(k,t)$ -FWL+. Further, how to design advanced model architecture to achieve the theoretical power of  $(k,t)$ -FWL+. Finally, can we provide a systematic implementation code package and practical usage of different downstream tasks such that researchers can easily develop an instance of  $(k,t)$ -FWL+ that can best fit their downstream tasks? We leave these to our future work.
Conclusions: In this work, we propose  $(k,t)$ -FWL+, a flexible and powerful extension of  $k$ -FWL. First,  $(k,t)$ -FWL+ expands the tuple aggregation style in  $k$ -FWL. We theoretically prove that given any fixed pace complexity of  $O(n^{k})$ ,  $(k,t)$ -FWL+ can still construct an expressiveness hierarchy up to solving the graph isomorphism problem. Second,  $(k,t)$ -FWL+ extends the global neighborhood definition in  $k$ -FWL to any equivariant set, which enables a finer-grained design space. We show that the  $(k,t)$ -FWL+ framework can implement many existing powerful GNN models with matching expressiveness. We further implement an instance named  $\mathbf{N}^2$ -GNN which is both practically powerful and theoretically expressive. Theoretically, it partially outperforms 3-WL but only requires  $O(n^{2})$  space. Empirically, it achieves new SOTA on BREC, ZINC-Subset, and ZINC-Full datasets. We envision  $(k,t)$ -FWL+ can serve as a promising general framework for designing highly effective GNNs tailored to real-world problems.
Jiarui Feng, Lecheng Kong, Hao Liu, and Yixin Chen are supported by NSF grant CBE-2225809. Muhan Zhang is partially supported by the National Natural Science Foundation of China (62276003) and Alibaba Innovative Research Program.