SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For experiments on both datasets, we use 5 2-DRFWL(2) GNN layers with hidden size 120. We train 200 epochs using Adam optimizer with initial learning rate 0.001, and plateau scheduler with patience 10, decay factor 0.5, and minimal learning rate  $10^{-5}$ . The batch size is 64. We also list all baseline results given in [21], including MPNNs like GCN [36], GCNII [13], GINE [30, 55] and GatedGCN [11], as well as Transformer-based models like fully connected Transformer [49] with Laplacian PE (LapPE) [19, 20] and SAN [38]. The results are shown in Table 10.
From Table 10, we see that 2-DRFWL(2) GNN outperforms almost all MPNNs. Surprisingly, on Peptides-struct the performance of 2-DRFWL(2) GNN is even comparable to Transformer-based methods (which inherently capture long-range information by treating the graph as fully connected). Therefore, we conclude that although designed to capture local structural information, our model still possesses the ability to learn long-range dependencies.
F.5 Additional experiments on cycle counting power and scalability
Datasets. To further compare the cycle counting power and scalability of  $d$ -DRFWL(2) GNNs (especially for the case of  $d = 2$ ) with other powerful GNNs, we perform node-level cycle counting tasks on two protein datasets, ProteinsDB and HomologyTAPE, collected from [29]. We evaluate the performance and empirical efficiency of 2-DRFWL(2) GNN as well as baseline models such as MPNN, NGNN,  $\mathrm{I}^2$ -GNN and PPGN, on both datasets.
The official code of [29] provides raw data for three protein datasets—"Enzymes vs Non-Enzymes", "Scope 1.75" and "Protein Function". All three datasets contain graphs that represent protein molecules, in which nodes represent amino acids, and edges represent the peptide bonds or hydrogen bonds between amino acids. Each of the three datasets contains a number of directed graphs, in which only a small portion of edges are directed.
We collect the first two datasets from the three and convert all graphs to undirected graphs by adding inverse edges for the directed edges. Since we are only interested in cycle counting tasks, we remove node features (e.g., amino acid types, spatial positions of amino acids) and edge features (e.g., whether the bond is a peptide bond or a hydrogen bond). The node-level cycle counts are generated by the same method as the one described in Appendix E.2.1. We then rename the two processed datasets as ProteinsDB and HomologyTAPE respectively. (The new names of the datasets correspond to their original .zip file names.)
Some statistics of the two processed datasets are shown in Table 11. We also show the statistics of some other datasets we use in Table 12. From Table 11 and Table 12 we can see that the graphs in ProteinsDB and HomologyTAPE datasets are much larger than those in QM9, ZINC, ogbg-molhiv, ogbg-molpcba or the synthetic dataset, thus suitable for evaluating the scalability of GNN models.
Table 11: Statistics of two protein datasets.
Dataset	Number of graphs	Average number of nodes	Average number of edges
ProteinsDB	1,178	475.9	714.8
HomologyTAPE	16,292	167.3	256.7
Table 12: Statistics of other datasets we use.
Dataset	Number of graphs	Average number of nodes	Average number of edges
Synthetic	5,000	18.8	31.3
QM9	130,831	18.0	18.7
ZINC-12K	12,000	23.2	24.9
ZINC-250K	249,456	23.2	24.9
ogbg-molhiv	41,127	25.5	27.5
ogbg-molpcba	437,929	26.0	28.1
Table 13: Average normalized MAE results of node-level counting cycles on ProteinsDB and HomologyTAPE datasets. The colored cell means an error less than 0.002 (for ProteinsDB) or 0.0002 (for HomologyTAPE). "Out of GPU Memory" means the method takes an amount of GPU memory more than 24 GB.
Method	ProteinsDB	HomologyTAPE
3-Cycle	4-Cycle	5-Cycle	6-Cycle	3-Cycle	4-Cycle	5-Cycle	6-Cycle
MPNN	0.422208	0.223064	0.254188	0.203454	0.185634	0.162396	0.211261	0.158633
NGNN	0.000070	0.000189	0.003395	0.009955	0.000003	0.000016	0.003706	0.002541
I²-GNN	Out of GPU Memory	0.000002	0.000010	0.000069	0.000177
PPGN	Out of GPU Memory	Out of GPU Memory
2-DRFWL(2) GNN	0.000077	0.000192	0.000317	0.001875	0.000001	0.000007	0.000021	0.000178
Table 14: Empirical efficiency on ProteinsDB and HomologyTAPE datasets. "OOM" means the method takes an amount of GPU memory more than 24 GB.
Method	ProteinsDB	HomologyTAPE
Memory (GB)	Pre. (s)	Train (s/epoch)	Memory (GB)	Pre. (s)	Train (s/epoch)
MPNN	2.60	235.7	0.597	1.99	243.8	5.599
NGNN	16.94	941.8	2.763	8.44	1480.7	15.249
I²-GNN	OOM	1293.4	OOM	21.97	3173.6	38.201
PPGN	OOM	235.7	OOM	OOM	243.8	OOM
2-DRFWL(2) GNN	8.11	1843.7	3.809	3.82	2909.3	30.687
For ProteinsDB, we use 10-fold cross validation and report the average normalized MAE for each counting task. We use the splitting provided in the raw data.
For HomologyTAPE, there are three test splits in the raw data, called "test_fold", "test_family" and "test superfamily" respectively. The training/validation/test_fold/test_family/test superfamily splitting is 12,312/736/718/1,272/1,254. We report the weighted average normalized MAE on the three test splits for each counting task.
Models. We adopt MPNN, NGNN,  $\mathrm{I}^2$ -GNN and PPGN as our baselines. The implementation details of 2-DRFWL(2) GNN as well as all baseline methods follow Appendix E.2.1, except that (i) for NGNN and  $\mathrm{I}^2$ -GNN, the subgraph height is 3 for all tasks, and (ii) for PPGN, the number of PPGN layers is 5 and the embedding size is 64.
Training settings. We use Adam optimizer with initial learning rate 0.001, and use plateau scheduler with patience 10, decay factor 0.9 and minimum learning rate  $10^{-5}$ . We train all models for 1,500 epochs on ProteinsDB and 2,000 epochs on HomologyTAPE. The batch size is 32.
Results. From Table 13, we see that 2-DRFWL(2) GNN achieves relatively low errors (average normalized MAE  $< 0.002$  for ProteinsDB, and  $< 0.0002$  for HomologyTAPE) for counting all 3, 4, 5 and 6-cycles on both datasets. This again verifies our theoretical result (Theorem 4.5). Although  $\mathrm{I}^2$ -GNN and PPGN are also provably capable of counting up to 6-cycles, PPGN fails to scale to either ProteinsDB or HomologyTAPE dataset, while  $\mathrm{I}^2$ -GNN fails to scale to ProteinsDB. Therefore, 2-DRFWL(2) GNN maintains good scalability compared with other existing GNNs that can count up to 6-cycles.
We also measure (i) the maximal GPU memory usage during training, (ii) the preprocessing time, and (iii) the training time per epoch, for all models on both datasets. The result is shown in Table 14. We again see that 2-DRFWL(2) GNN uses much less GPU memory while training, compared with subgraph GNNs. Besides, the training time of 2-DRFWL(2) GNN is shorter than that of  $\mathrm{I}^2$ -GNN, while the preprocessing time is comparable.
From Table 14, we also see one limitation of our implementation of 2-DRFWL(2) GNN, which we have mentioned in Section 7—the preprocessing of 2-DRFWL(2) GNN is not efficient enough on larger graphs or graphs with a larger average degree. We believe this limitation can be addressed by
developing more efficient, parallelized operators for the preprocessing of 2-DRFWL(2) GNN, and we leave the exploration of such implementations to future work.