SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model: We set the number of hop  $h$  to be 8, the number of layers to be 4, the hidden size to 64, and the inner size  $d$  to 32. Meanwhile, the output channel is set to 16 for computing the similarity. We use the sparse version of data preprocessing and we do not add root aggregation (Equation (50)-(51)).
Training. We follow the same training and evaluation procedure as described by BREC [45]. The initial learning rate is 0.001 and the weight decay is set to 0.0001. The batch size is set to 4 and the number of epochs is 20. Due to the resource limitation, we only ran the experiment on 340 graph pairs (removed 20 4-vertex-condition graphs and 40 CFI graphs), which follow the same protocol as the reported result for  $\mathrm{I}^2$ -GNN.
F Additional experimental results
F.1 Ablation studies for  $(k,t)$ -FWL+
In this section, we provide an additional ablation study on  $(2,t)$ -FWL+ by varying different  $t$  and  $ES$ . We select two important  $ES$  mentioned in our paper— $Q_{1}(v)$  and  $\mathcal{SP}(v_{1}, v_{2})$ . We conduct experiments on both Expressiveness verification datasets and counting datasets. The experiment setup is the same as what is discussed in the above section. The results can be seen in Table 7 and Table 8. Notes that we find the performance of  $\mathcal{SP}$  or  $\mathcal{SP} \times \mathcal{SP}$  on counting dataset is similar to MPNN and thus omit it in Table 8.
We can see that as long as  $t = 2$ , we can have a perfect performance on SR25 no matter if we use  $\mathcal{SP}$  or  $Q_{1}(v_{1})$ , this aligns with our theoretical results. Further,  $\mathcal{SP}(v_{1}, v_{2}) \times Q_{1}(v_{1})$  achieve great results on all counting tasks and expressiveness verification. This indicates the  $\mathcal{SP}$  can also be a great choice in the practical scenario.
Table 7: Expressive power verification (Accuracy) for different  $(2,t)$ -FWL+.
Datasets	t=1,Q1(v1)	t=1,Q1(v1)∪Q1(v2)	t=1,SP(v1,v2)	t=2,SP(v1,v2)×Q1(v1)	t=2,SP(v1,v2)×SP(v1,v2)
EXP	100	100	100	100	100
CSL	100	100	100	100	100
SR25	6.67	6.67	6.67	100	100
Table 8: Evaluation of different  $(2,t)$ -FWL+ variants on Counting Substructures (norm MAE), cells with MAE greater than 0.01 are colored.
Target	t=1,Q1(v1)	t=1,Q1(v1) ∪ Q1(v2)	t=2,SP(v1,v2) × Q1(v1)	N2-GNN
Tailed Triangle	0.0033	0.0031	0.0022	0.0025
Chordal Cycle	0.0019	0.0017	0.0021	0.0019
4-Clause	0.0013	0.0014	0.0016	0.0005
4-Path	0.0046	0.0043	0.0076	0.0042
Tri.-Rec.	0.0043	0.0053	0.0081	0.0055
3-Cycles	0.0005	0.0006	0.0004	0.0002
4-Cycles	0.0027	0.0022	0.0037	0.0024
5-Cycles	0.0051	0.0042	0.0037	0.0039
6-Cycles	0.0113	0.0097	0.0097	0.0075
F.2 Practical complexity of  $\mathbf{N}^2$ -GNN
In this section, we provide practical complexity analysis for  $\mathbf{N}^2$ -GNN. Here we use the cycle counting datasets with a similar parameter size for all models and a batch size of 256. To ensure a fair comparison, in  $\mathbf{N}^2$ -GNN, we use the sparse version for data preprocessing (which is true for all compared models), and set the number of workers as 0 and random seed as 1 for all models. The experiments are run on a single TeslaV100 32GB GPU. We report the training time, maximum training memory, inference time, and maximum inference memory usage during inference for both  $h = 1$  and  $h = 2$  for an average of 20 epochs. The results is shown in Table 9 and Table 10.
We can see that the empirical memory usage of  $\mathbf{N}^2$ -GNN is much less than  $I^2$ -GNN when  $h = 1$  and comparable when  $h = 2$ . We conjecture the reason why  $\mathbf{N}^2$ -GNN require more training memory than  $\mathbf{I}^2$ -GNN when  $h = 2$  is that when  $h$  increase, each tuple will need to aggregate much more neighbors than each node in  $\mathbf{I}^2$ -GNN and each aggregation requires a constant size (5 in  $\mathbf{N}^2$ -GNN) of more memory for saving the gradients. Meanwhile, As we mentioned in limitations (Appendix D), to enjoy some extent of parallelism, the current implementation of  $N^2$ -GNN needs to save all neighbor indices, which introduces a constant factor of more memory on saving. But the inference memory of  $\mathbf{N}^2$ -GNN is less than  $\mathbf{I}^2$ -GNN in both  $h = 1$  and  $h = 2$ . The training time and inference time of  $\mathbf{N}^2$ -GNN are also more efficient than  $\mathbf{I}^2$ -GNN. However, currently, the time and memory cost by  $\mathbf{N}^2$ -GNN are higher than node-based subgraph GNNs. We do want to highlight several points: (1) By changing the way of implementation,  $N^2$ -GNN can be implemented strictly within  $O(n^{2})$  space. Therefore, the merit of  $N^2$ -GNN still holds. (2) The experiment is conducted on the same parameter budget level. However, in real-world tasks,  $N^2$ -GNN needs much fewer parameters to achieve SOTA results (Table 5). (3) In this work, we not only aim to propose a new model but also mean to introduce a new flexible framework,  $(k,t)$ -FWL+. The empirical success of  $N^2$ -GNN demonstrates the great potential of this framework for designing new expressive GNN variants.
Table 9: Practical usage comparison for  $h = 1$  .
h=1	# Params	Training time (ms/epoch)	Training memory (GB)	Inference time (ms/epoch)	Inference Memory (GB)
NGNN	186k	478.1	0.398	463.4	0.089
ID-GNN	135k	455.5	0.375	432.8	0.094
GNN-AK+	251k	546.5	0.479	490.1	0.090
I2-GNN	194k	666.9	1.326	613.5	0.292
N2-GNN	202k	368.3	0.744	311.8	0.140
Table 10: Practical usage comparison for  $h = 2$
h=2	# Params	Training time (ms/epoch)	Training memory (GB)	Inference time (ms/epoch)	Inference Memory (GB)
NGNN	186k	516.5	0.916	470.7	0.198
ID-GNN	135k	521.7	0.858	471.5	0.211
GNN-AK+	251k	617.4	1.082	556.1	0.200
I2-GNN	194k	926.8	3.012	770.1	0.675
N2-GNN	202k	892.9	4.026	709.8	0.660
F.3 Additional discussion on BREC experiment
In Table 11, we provide a complete comparison of all baseline methods on the BREC dataset. Please see the detailed description of all baseline methods in BREC [45]. We can see that  $\mathbf{N}^2$ -GNN achieve the best result among all GNN baselines and the second best result among all baselines, even if we only test on 340 pair of graphs. Particularly, we surpass  $\mathrm{I}^2$ -GNN on all parts. This result empirically verified Theorem 4.1 that the theoretical power of  $\mathrm{N}^2$ -GNN is more powerful than  $\mathrm{I}^2$ -GNN. Meanwhile, we notice that SSWL_P can distinguish more CFI graph pairs than ours, which may contradict Theorem 4.1. This may be due to the fact that we remove 40 CFI graph pairs in the experiments given the resource limitation and SSWL_P uses 8 layers in the experiment and we only use 4 layers. Meanwhile, we don't implement the hierarchical multiset in the current version of  $\mathrm{N}^2$ -GNN, which slightly compromises the expressive power.
Table 11: Complete comparison on BREC dataset
Model	Basic Graphs (60)	Regular Graphs (140)	Extension Graphs (100)	CFI Graphs (100)	Total (400)
Number	Accuracy	Number	Accuracy	Number	Accuracy	Number	Accuracy	Number	Accuracy
3-WL	60	100%	50	35.7%	100	100%	60	60.0%	270	67.5%
SPD-WL	16	26.7%	14	11.7%	41	41%	12	12%	83	20.8%
S3	52	86.7%	48	34.3%	5	5%	0	0%	105	26.2%
S4	60	100%	99	70.7%	84	84%	0	0%	243	60.8%
N1	60	100%	99	85%	93	93%	0	0%	252	63%
N2	60	100%	138	98.6%	100	100%	0	0%	298	74.5%
M1	60	100%	50	35.7%	100	100%	41	41%	251	62.8%
NGNN	59	98.3%	48	34.3%	59	59%	0	0%	166	41.5%
DE+NGNN	60	100%	50	35.7%	100	100%	21	21%	231	57.8%
DS-GNN	58	96.7%	48	34.3%	100	100%	16	16%	222	55.5%
DSS-GNN	58	96.7%	48	34.3%	100	100%	15	15%	221	55.2%
SUN	60	100%	50	35.7%	100	100%	13	13%	223	55.8%
SSWL_P	60	100%	50	35.7%	100	100%	38	38%	248	62%
GNN-AK	60	100%	50	35.7%	97	97%	15	15%	222	55.5%
KP-GNN	60	100%	106	75.7%	98	98%	11	11%	275	68.8%
I2-GNN	60	100%	100	71.4%	100	100%	21	21%	281	70.2%
PPGN	60	100%	50	35.7%	100	100%	23	23%	233	58.2%
δ-k-LGNN	60	100%	50	35.7%	100	100%	6	6%	216	54%
KC-SetGNN	60	100%	50	35.7%	100	100%	1	1%	211	52.8%
GSN	60	100%	99	70.7%	95	95%	0	0%	254	63.5%
DropGNN	52	86.7%	41	29.3%	82	82%	2	2%	177	44.2%
OSAN	56	93.3%	8	5.7%	79	79%	5	5%	148	37%
Graphormer	16	26.7%	12	10%	41	41%	10	10%	79	19.8%
N2-GNN	60	100%	100	71.4%	100	100%	27	27%	287	71.8%
F.4 Ablation studies for  $\mathbf{N}^2$ -GNN
In this subsection, we perform ablation studies to investigate each component in  $\mathbf{N}^2$ -GNN. All ablation studies are performed on ZINC-Subset and target  $C_v$  in QM9. Except hyper-parameter mentioned later, all other setting is exactly the same as what is described in Section E.2.
In the first ablation study, we investigate different components of the model. Specifically, we analyze two components. The first component is aggregation  $h^l (v_1,v_2)$  described in Equation (50)-(51). The second component is the dense/sparse embedding described in Section D.3. In the ablation study, we verify all different combinations of two components. For the model without Equation (50)-(51), to ensure that the size of the model is comparable, we increase the model hidden size to 120 and 136 for
ZINC-Subset and QM9, respectively. The results are shown in Table 12. First, we can see adding Equation (50)-(51) indeed improve the performance across different tasks. Second, by explicitly keeping  $O(n^{2})$  embedding the performance, of  $\mathbf{N}^2$ -GNN on ZINC-Subset improved from 0.079 to 0.059. This may indicate that even though the two models have the same expressive power from the theoretical side, adding additional feature embedding can still boost the performance significantly in real-world tasks as it allows a better mixture of both feature and structure information. Note that these two components were used in many previous models like ESAN [43], SUN [27], and SSWL+ [12], etc. However, our model still outperforms them by a large margin. This indicates that the expressive power of the model is still the most critical ingredient, adding these additional components is just a way to unleash all the potential of the model for real-world tasks.
Table 12: Ablation study of model components.
Model design	ZINC-Subset ↓	QM9 (Cv) ↓
Full	0.059 ± 0.002	0.0760
w/o hl(v2, v2)	0.064 ± 0.003	0.0811
sparse	0.079 ± 0.003	0.0876
w/o hl(v2, v2) + sparse	0.084 ± 0.003	0.0847
Table 13: Ablation study of  $h$  .
h in N2-GNN	ZINC-Subset ↓	QM9 (Cv) ↓
h=1	0.150 ± 0.009	0.0823
h=2	0.127 ± 0.005	0.0808
h=3	0.059 ± 0.002	0.0760
h=4	0.063 ± 0.004	0.0826
In the second ablation study, we evaluate the effect of the number of hop  $h$ . We vary  $h$  from 1 to 4 and results are shown in Table 13. We can see an increase of  $h$  can boost the performance of the model, which has already been verified in previous subgraph-based methods. Particularly, for ZINC-Subset, we find that the third hop is critical to the performance. However, if we continue to increase the number of hops, the performance drops, which is a sign of overfitting.