SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now compare  $d$ -DRFWL(2) with LFWL(2)/SLFWL(2) in terms of discriminative power. We have the following results:
- No  $d$ -DRFWL(2) with a fixed  $d$  value can be more powerful than LFWL(2) or SLFWL(2). This is because for any finite  $d$ , we can construct graph pairs that are separable by LFWL(2) and SLFWL(2) but not separable by  $d$ -DRFWL(2). One of such graph pairs can be two  $(3d + 1)$ -cycles and one  $(6d + 2)$ -cycle, between which  $d$ -DRFWL(2) cannot discriminate (as shown in the proof of Theorem 3.2). However, since both LFWL(2) and SLFWL(2) can calculate distance between every pair of nodes (following a procedure constructed in the proof of Theorem 3.2), it is easy for LFWL(2) or SLFWL(2) to distinguish between the aforementioned pair of graphs, since they have different diameters.
- With sufficiently large  $d$ ,  $d$ -DRFWL(2) is not less powerful than LFWL(2) or SLFWL(2). This is because for graphs with diameter  $\leqslant d$ ,  $d$ -DRFWL(2) has equal power to FWL(2). Since it is shown in [58] that there exist graph pairs separable by FWL(2) but not separable by LFWL(2) or SLFWL(2), once the value of  $d$  grows larger than the diameters of such graph pairs, those graph pairs can be separated by  $d$ -DRFWL(2) but not LFWL(2) or SLFWL(2).
Therefore, for sufficiently large  $d$ , the discriminative power of  $d$ -DRFWL(2) is neither stronger nor weaker than LFWL(2) or SLFWL(2). However, it remains unknown the relation in discriminative power between LFWL(2)/SLFWL(2) and  $d$ -DRFWL(2) with a practical  $d$  value, such as  $d = 2$  or  $d = 3$ . Are  $d$ -DRFWL(2) with those smaller  $d$  values less powerful than LFWL(2)/SLFWL(2), or are they incomparable? We leave this question open for future research.
E Experimental details
E.1 Model implementation
Let  $G$  be a graph with node features  $f_{u}, u \in \mathcal{V}_{G}$  and edge features  $e_{uv}, \{u, v\} \in \mathcal{E}_{G}$ . In most of our experiments, we implement  $d$ -DRFWL(2) GNNs as following: we generate  $h_{uv}^{(0)}$  as
$$
h _ {u v} ^ {(0)} = \left\{ \begin{array}{l l} \operatorname {L I N} _ {0} \left(f _ {u}\right), & \text {i f} d (u, v) = 0, \\ \operatorname {L I N} _ {1} \left(f _ {u} + f _ {v}, e _ {u v}\right), & \text {i f} d (u, v) = 1, \\ \operatorname {L I N} _ {k} \left(f _ {u} + f _ {v}\right), & \text {i f} d (u, v) = k \geqslant 2, \end{array} \right. \tag {136}
$$
where  $\mathrm{LIN}_i, i = 0,1,\ldots ,d$  are linear functions. When node and edge features are absent, we assign identical values to  $h_{uv}^{(0)}$  with the same  $d(u,v)$ .
The  $m_{ijk}^{(t)}$  and  $f_{k}^{(t)}$  functions in (10) and (11) are chosen as
$$
m _ {i j k} ^ {(t)} \left(h _ {w v} ^ {(t - 1)}, h _ {u w} ^ {(t - 1)}\right) = \operatorname {R e L U} \left(\operatorname {L I N} ^ {(t)} \left(h _ {w v} ^ {(t - 1)} + h _ {u w} ^ {(t - 1)}\right)\right), \tag {137}
$$
$$
\begin{array}{l} f _ {k} ^ {(t)} \left(h _ {u v} ^ {(t - 1)}, \left(a _ {u v} ^ {i j k (t)}\right) _ {0 \leqslant i, j \leqslant d}\right) = h _ {u v} ^ {(t - 1)} + \mathrm {M L P} _ {k} ^ {(t)} \left((1 + \epsilon) h _ {u v} ^ {(t - 1)} \right. \\ \left. + \sum_ {\substack {| i - j | \leqslant k \leqslant i + j \\ 0 \leqslant i, j \leqslant d}} \operatorname {LIN} _ {\{i, j, k \}} ^ {(t)} \left(a _ {u v} ^ {i j k (t)}\right)\right), \tag{138} \\ \end{array}
$$
where  $\mathrm{LIN}^{(t)}$  in (137) is a linear module whose parameters are shared among all  $i,j,k$  combinations;  $\mathrm{LIN}_{\{i,j,k\}}^{(t)}$  in (138) is a linear module whose parameters are shared among all  $i,j,k$  that form the same multiset  $\{\{i,j,k\}\}$  (for example, the contributions  $a_{uv}^{122}, a_{uv}^{212}$  and  $a_{uv}^{221}$  are all linearly transformed by an identical  $\mathrm{LIN}_{\{1,2,2\}}^{(t)}$ );  $\mathrm{MLP}_k^{(t)}$  is a multilayer perceptron, and can depend on  $k$ ;  $\epsilon$  can be a constant or a learnable parameter. After the  $d$ -DRFWL(2) GNN layers, we apply a sum-pooling layer
$$
R \left(\{\{h _ {u v} ^ {(T)}: (u, v) \in \mathcal {V} _ {G} ^ {2} \text {a n d} 0 \leqslant d (u, v) \leqslant d \}\}\right) = \sum_ {0 \leqslant d (u, v) \leqslant d} h _ {u v} ^ {(T)}, \tag {139}
$$
and then a final MLP.
We point out that our implementation of  $d$ -DRFWL(2) GNNs has the property  $h_{uv}^{(t)} = h_{vu}^{(t)}$ , for any  $u, v \in \mathcal{V}_G$  with  $0 \leqslant d(u, v) \leqslant d$ , and for any  $t = 0, 1, \dots, T$ . We call an instance of  $d$ -DRFWL(2) GNN symmetrized, if it preserves the above property. It is obvious that symmetrized  $d$ -DRFWL(2) GNNs only constitute a subspace of the function space of all  $d$ -DRFWL(2) GNNs.
Nevertheless, we remark that for the case of  $d = 2$ , symmetrized 2-DRFWL(2) GNNs have equal node-level cycle counting power to 2-DRFWL(2) GNNs. Actually, in the proofs of Lemmas C.7-C.19 and Theorems 4.4-4.6 (see Appendix C for details), all the 2-DRFWL(2) tests we constructed have the property  $W^{(t)}(u,v) = W^{(t)}(v,u)$ , for any  $u,v\in \mathcal{V}_G$  with  $0\leqslant d(u,v)\leqslant 2$  and any  $t$ ; on the other hand, it is easy to establish the equivalence between such kind of 2-DRFWL(2) tests and symmetrized 2-DRFWL(2) GNNs. Similar facts can be verified for  $d$ -DRFWL(2) GNNs with  $d = 1$  or  $d = 3$ . Therefore, at least for all our experiments (where  $d\leqslant 3$ ), our symmetrized implementation of  $d$ -DRFWL(2) GNNs does no harm to their theoretical cycle counting power, although the model is greatly simplified.
E.2 Experimental settings
E.2.1 Substructure counting
Datasets. The synthetic dataset is provided by open-source code of GNNAK on github. The node-level substructure counts are calculated by simple DFS algorithms, which we implement in C language. (This part of code is also available in our repository.)
Models. Implementations of all baseline methods (MPNN, ID-GNN, NGNN, GNNAK+, PPGN and I²-GNN) follow [33]. For  $d$ -DRFWL(2) GNN ( $d = 1, 2, 3$ ), we use 5  $d$ -DRFWL(2) GNN layers. In each layer (numbered by  $t = 1, \ldots, 5$ ),  $\mathrm{MLP}_k^{(t)}$  is a 2-layer MLP, for  $k = 0, 1, \ldots, d$ . (See (138) for the definition of  $\mathrm{MLP}_k^{(t)}$ .) The embedding size is 64.
Training settings. We use Adam optimizer with initial learning rate 0.001, and use plateau scheduler with patience 10, decay factor 0.9 and minimum learning rate  $10^{-5}$ . We train our model for 2,000 epochs. The batch size is 256.
E.2.2 Molecular property prediction
Datasets. The QM9 and ZINC datasets are provided by PyTorch Geometric package [22]. The ogbg-molhiv and ogbg-molpcba datasets are provided by Open Graph Benchmark (OGB) [31]. We rewrite preprocessing code for all four datasets.
The training/validation/test splitting for QM9 is 0.8/0.1/0.1. The training/validation/test splittings for ZINC, ogbg-molhiv and ogbg-molpcba are provided in the original releases.
Models. For QM9, we adopt a 2-DRFWL(2) GNN with 5 2-DRFWL(2) GNN layers. In each layer (numbered by  $t = 1,\dots ,5$ ),  $\mathrm{MLP}_k^{(t)}$  is a 2-layer MLP, for  $k = 0,1,2$ . The embedding size is 64.
For ZINC, we adopt a simplified version of 3-DRFWL(2) GNN, which only adds terms  $a_{uv}^{313}$ ,  $a_{uv}^{133}$  and  $a_{uv}^{331}$  to the update rules (10) and (11) of a 2-DRFWL(2) GNN. Notice that the adopted model architecture is slightly different from the one described in Appendix E.1. This is because we observe that including other kinds of message passing in 3-DRFWL(2) GNN does not improve the performance on ZINC. The adopted 3-DRFWL(2) GNN has 6 3-DRFWL(2) GNN layers. In each layer (numbered by  $t = 1, \ldots, 6$ ),  $\mathrm{MLP}_k^{(t)}$  is a 2-layer MLP, for  $k = 0, 1, 2, 3$ . The embedding size is 64. We apply batch normalization between every two 3-DRFWL(2) GNN layers.
For ogbg-molhiv, we also adopt a simplified version of 2-DRFWL(2) GNN; namely, we remove the term  $a_{uv}^{222}$  in the update rules of the original 2-DRFWL(2) GNN. The adopted 2-DRFWL(2) GNN has 5 2-DRFWL(2) GNN layers. In each layer (numbered by  $t = 1,\dots ,5$ ),  $\mathrm{MLP}_k^{(t)}$  is a 2-layer MLP, for  $k = 0,1,2$ . The embedding size is 300. We apply a dropout layer with  $p = 0.2$  after every 2-DRFWL(2) GNN layer.
For ogbg-molpcba, we adopt a 2-DRFWL(2) GNN with 3 2-DRFWL(2) GNN layers. In each layer (numbered by  $t = 1,2,3$ ),  $\mathrm{MLP}_k^{(t)}$  is a 2-layer MLP, with layer normalization applied after every MLP layer, for  $k = 0,1,2$ . The embedding size is 256. We apply layer normalization between every two 2-DRFWL(2) GNN layers, and also apply a dropout layer with  $p = 0.2$  after every 2-DRFWL(2) GNN layer.
Training settings. For QM9, we use the Adam optimizer, and use plateau scheduler with patience 10, decay factor 0.9 and minimum learning rate  $10^{-5}$ . For each of the 12 targets on QM9, we search hyperparameters from the following space: (i) initial learning rate  $\in \{0.001, 0.002, 0.005\}$ ; (ii) whether to apply layer normalization between every two 2-DRFWL(2) GNN layers (yes/no). We train our model for 400 epochs. The batch size is 64.
For both ZINC-12K and ZINC-250K, we use Adam optimizer with initial learning rate 0.001. For ZINC-12K, we use plateau scheduler with patience 20, decay factor 0.5 and minimum learning rate  $10^{-5}$ ; for ZINC-250K, we use plateau scheduler with patience 25, decay factor 0.5 and minimum learning rate  $5 \times 10^{-5}$ . We train our model for 500 epochs on ZINC-12K and 800 epochs on ZINC-250K. The batch size is 128.
For both ogbg-molhiv and ogbg-molpcba, we use Adam optimizer with initial learning rate 0.0005. For ogbg-molhiv, we use step scheduler with step size 20 and decay factor 0.5. We train our model for 100 epochs and the batch size is 64. For ogbg-molpcba, we use step scheduler with step size 10 and decay factor 0.8. We train our model for 150 epochs and the batch size is 256.
In Appendix F, we present
- Experiments that answer Q4 in Section 6 of the main paper;
- Ablation studies on the substructure counting power of 2-DRFWL(2) GNNs;
- Additional experiments that study the performance of  $d$ -DRFWL(2) GNNs on molecular datasets;
- Experiments that study the ability of  $d$ -DRFWL(2) GNNs to capture long-range interactions;
- Additional experiments that study the cycle counting power and empirical efficiency of 2-DRFWL(2) GNNs on larger graphs.
F.1 Discriminative power
Datasets. To answer Q4, we evaluate the discriminative power on three synthetic datasets: (1) EXP [1], containing 600 pairs of non-isomorphic graphs that cannot be distinguished by the WL(1) test; (2) SR25 [6], containing 15 non-isomorphic strongly regular graphs that the FWL(2) test fails to distinguish; (3) BREC [51], containing 400 pairs of non-isomorphic graphs generated from a variety of sources.
For EXP, we follow the evaluation process in [59], use 10-fold cross validation, and report the average binary classification accuracy; for SR25, we follow [60], treat the task as a 15-way classification, and report the accuracy. Raw data for both EXP and SR25 datasets are provided by open-source code of GNNAK on github.
For BREC, we use Reliable Paired Comparison (RPC) as the evaluation method, following [51]. For every graph pair  $(G,H)$ , the RPC procedure consists of two stages: major procedure and reliability check.
- In the major procedure, we generate  $q = 32$  copies  $(G_i, H_i)$  of  $(G, H)$ ,  $i = 1, \ldots, q$ . The copies  $G_i$  and  $H_i$  ( $i = 1, \ldots, q$ ) are obtained by randomly relabeling nodes in  $G$  and  $H$ , respectively. We then apply a Hotelling's  $T^2$  test to check whether we should reject the null hypothesis  $H_0: \mathbb{E}_{\mathrm{relabel}}[f(G) - f(H)] = 0$ , meaning that the model  $f$  cannot distinguish between  $G$  and  $H$ .
- In the reliability check, we replace  $H_{i}$  in each copy  $(G_{i},H_{i})$  with  $G_{i}^{\pi}$ , which is obtained by randomly relabeling nodes in  $G_{i}$ , for  $i = 1,\dots ,q$ . We then apply a second  $T^{2}$  test to check whether we should reject the null hypothesis  $H_0^\prime :\mathbb{E}_{\mathrm{relabel}}[f(G) - f(G^\pi)] = 0$ , meaning that the representations of two isomorphic graphs will not deviate too much from one another due to numerical errors.
Finally,  $G$  and  $H$  are considered separable by  $f$  iff we should reject  $H_0$  but should not reject  $H_0'$ . The detailed description of the RPC procedure is provided in [51].
Table 5: Accuracy on EXP/SR25.
Method	EXP	SR25
GIN	50%	6.67%
Nested GIN	99.9%	6.67%
GIN-AK+	100%	6.67%
PPGN	100%	6.67%
3-GCN	99.7%	6.67%
I²-GNN	100%	100%
2-DRFWL(2) GNN	99.8%	6.67%
3-DRFWL(2) GNN	100%	6.67%
Baselines. For EXP and SR25, baseline methods are chosen from: (1) Basic MPNNs. These methods have discriminative power upper-bounded by the WL(1) test, and we choose GIN [55], which can achieve WL(1) expressive power theoretically. (2) Subgraph MPNNs. These methods are strictly more powerful than WL(1) but strictly upper-bounded by the FWL(2) test [58]. We choose Nested GIN [59] and GIN-AK+ [60]. (3) Higher-order GNNs with expressive power equal to FWL(2). We choose PPGN [42] and 3-GCN [1]. (4) Methods with discriminative power partially stronger than FWL(2), e.g.  $\mathrm{I}^2$ -GNN [33].
For BREC, the complete list of baseline results is given in Table 2 of [51]. We select 3-WL, NGNN, NGNN with distance encoding (DE), SUN [24], SSWL_P [58], GNN-AK,  $\mathrm{I}^2$ -GNN and PPGN as our baselines.
Models. We evaluate both 2-DRFWL(2) GNN and 3-DRFWL(2) GNN on the three datasets. On all three datasets, we adopt a  $d$ -DRFWL(2) GNN with  $5d$ -DRFWL(2) GNN layers ( $d = 2$  or 3). In each layer (numbered by  $t = 1, \ldots, 5$ ),  $\mathrm{MLP}_k^{(t)}$  is a 2-layer MLP, for  $k = 0, \ldots, d$ . The embedding size is 64 for EXP/SR25, and 32 for BREC.
Training settings. For both EXP and SR25 datasets, we use Adam optimizer with learning rate 0.001. We train our model for 10 epochs on EXP and 100 epochs on SR25. The batch size is 20 on EXP and 64 on SR25. For BREC, we use Adam optimizer with learning rate 0.0001. We train for 10 epochs and the batch size is 32.
Results. We can see from Table 5 that 2-DRFWL(2) GNN already achieves  $99.8\%$  accuracy on the EXP dataset, while 3-DRFWL(2) GNN achieves  $100\%$  accuracy. This complies with our theoretical result (Theorem 3.1) that  $d$ -DRFWL(2) GNNs are strictly more powerful than WL(1); moreover, 3-DRFWL(2) GNN achieves higher accuracy than 2-DRFWL(2) GNN, verifying the expressiveness hierarchy we obtain in Theorem 3.2. Table 5 also shows that both 2-DRFWL(2) GNN and 3-DRFWL(2) GNN fail on SR25 with a 1/15 accuracy. This verifies our assertion that  $d$ -DRFWL(2) GNNs are strictly less powerful than FWL(2).
Table 6: Numbers of distinguishable graph pairs on BREC.
Method	Basic (60)	Regular (140)	Extension (100)	CFI (100)	Total (400)
3-WL	60	50	100	60	270
NGNN	59	48	59	0	166
NGNN+DE	60	50	100	21	231
SUN	60	50	100	13	223
SSWL_P	60	50	100	38	248
GNN-AK	60	50	97	15	222
I²-GNN	60	100	100	21	281
PPGN	60	50	100	23	233
2-DRFWL(2) GNN	60	50	99	0	209
3-DRFWL(2) GNN	60	50	100	13	223
The experimental results on BREC are shown in Table 6. Since the 400 graph pairs in BREC are categorized into four classes—basic graphs (60 pairs), regular graphs (140 pairs), extension graphs (100 pairs) and CFI graphs (100 pairs), we report the number of distinguishable graph pairs in each class, and also the total number. From Table 6, it is easy to observe the expressiveness gap between  $d$ -DRFWL(2) and 3-WL (which has equal discriminative power to FWL(2)), and the gap between 2-DRFWL(2) and 3-DRFWL(2). This again verifies Theorem 3.2.
F.2 Ablation studies on substructure counting
We study the impact of different kinds of message passing in 2-DRFWL(2) GNNs on their substructure (especially, cycle) counting power, by comparing the full 2-DRFWL(2) GNN with modified versions in which some kinds of message passing are forbidden. The dataset, model hyperparameters and training settings are the same as those in Appendix E.2.1.
Table 7: Ablation studies of node-level counting substructures on synthetic dataset. The colored cell means an error less than 0.01.
Method	Synthetic (norm. MAE)
3-Cyc.	4-Cyc.	5-Cyc.	6-Cyc.	Tail. Tri.	Chor. Cyc.	Tri.-Rect.
2-DRFWL(2) GNN	0.0004	0.0015	0.0034	0.0087	0.0030	0.0026	0.0070
2-DRFWL(2) GNN (w/o distance-0 tuples)	0.0006	0.0017	0.0035	0.0092	0.0030	0.0030	0.0070
2-DRFWL(2) GNN (w/o a222)	0.0006	0.0013	0.0033	0.0448	0.0029	0.0029	0.0067
2-DRFWL(2) GNN (w/o a222, a122, a212, a221)	0.0004	0.0009	0.0852	0.0735	0.0020	0.0020	0.0051
1-DRFWL(2) GNN	0.0002	0.0379	0.1078	0.0948	0.0037	0.0013	0.0650
Results. From Table 7, we see that by removing the term  $a_{uv}^{222}$  in (11) (i.e., by forbidding a distance-2 tuple  $(u,v)$  to receive messages from other two distance-2 tuples  $(u,w)$  and  $(w,v)$ ), we make 2-DRFWL(2) GNN unable to node-level count 6-cycles. This complies with our intuitive discussion at the end of Section 4 that such kind of message passing is essential for identifying closed 6-walks, and thus for counting 6-cycles.
Similarly, if we further remove  $a_{uv}^{122}$ ,  $a_{uv}^{212}$  and  $a_{uv}^{221}$ , then 2-DRFWL(2) GNN will be unable to count 5-cycles. If  $a_{uv}^{121}$ ,  $a_{uv}^{211}$  and  $a_{uv}^{112}$  are also removed, the 2-DRFWL(2) GNN actually degenerates into 1-DRFWL(2) GNN, and becomes unable to count 4-cycles and triangle-rectangles (the latter also containing a 4-cycle as its subgraph). Finally, we can see from Table 7 that even 1-DRFWL(2) GNN can count 3-cycles, tailed triangles and chordal cycles, since the numbers of these substructures only depend on the number of closed 3-walks.
We also study the effect of distance-0 tuples in 2-DRFWL(2) GNN. Removing embeddings for those tuples does not affect the theoretical counting power of 2-DRFWL(2) GNN, but we observe that the empirical performance on counting tasks slightly drops, possibly due to optimization issues.
F.3 Additional experiments on molecular datasets
In the following, we present the results on ZINC, ogbg-molhiv and ogbg-molpcba datasets.
Table 8: Ten-runs MAE results on ZINC-12K (smaller the better), four-runs MAE results on ZINC-250K (smaller the better), ten-runs ROC-AUC results on ogbg-molhiv (larger the better), and four-runs AP results on ogbg-molpcba (larger the better). The * indicates the model uses virtual node on ogbg-molhiv and ogbg-molpcba.
Method	ZINC-12K (MAE)	ZINC-250K (MAE)	ogbg-molhiv (AUC)	ogbg-molpcba (AP)
GIN*	0.163±0.004	0.088±0.002	77.07±1.49	27.03±0.23
PNA	0.188±0.004	-	79.05±1.32	28.38±0.35
DGN	0.168±0.003	-	79.70±0.97	28.85±0.30
HIMP	0.151±0.006	0.036±0.002	78.80±0.82	-
GSN	0.115±0.012	-	80.39±0.90	-
Deep LRP	-	-	77.19±1.40	-
CIN-small	0.094±0.004	0.044±0.003	80.05±1.04	-
CIN	0.079±0.006	0.022±0.002	80.94±0.57	-
Nested GIN*	0.111±0.003	0.029±0.001	78.34±1.86	28.32±0.41
GNNAK+	0.080±0.001	-	79.61±1.19	29.30±0.44
SUN (EGO)	0.083±0.003	-	80.03±0.55	-
I²-GNN	0.083±0.001	0.023±0.001	78.68±0.93	-
d-DRFWL(2) GNN	0.077±0.002	0.025±0.003	78.18±2.19	25.38±0.19
F.4 Long-range interactions
In 6.2 of the main paper, we mention that 2-DRFWL(2) GNN greatly outperforms subgraph GNNs like NGNN or  $\mathrm{I}^2$ -GNN on the targets  $U_0, U, H$  and  $G$  of the QM9 dataset, a phenomenon we ascribe to the stronger ability of 2-DRFWL(2) GNN to capture long-range interactions. Actually, a bit knowledge from physical chemistry tells us that the targets  $U_0, U, H$  and  $G$  not only depend on the chemical bonds between atoms, but also the intermolecular forces (such as hydrogen bonds). Such intermolecular forces can exist between atoms that are distant from each other on the molecular graph (since the two atoms belong to different molecules). However, most subgraph GNNs process the input graphs by extracting  $k$ -hop subgraphs around each node, usually with a small  $k$ . This operation thus prevents information from propagating between nodes that are more than  $k$ -hop away from one another, and makes subgraph GNNs fail to learn the long-range interactions that are vital for achieving good performance on the targets  $U_0, U, H$  and  $G$ . In contrast, 2-DRFWL(2) GNN directly performs message passing between distance-restricted 2-tuples on the raw graph, resulting in its capability to capture such long-range interactions.
So far, it remains an interesting question whether 2-DRFWL(2) GNNs' ability to capture long-range interactions is comparable with that of globally expressive models, such as  $k$ -GNNs [44] or GNNs based on SSWL, LFWL(2) or SLFWL(2) [58]. Those models keep embeddings for all possible  $k$ -tuples  $\mathbf{v} \in \mathcal{V}_G^k$  (instead of only distance-restricted ones). In this sense, they treat the input graphs as fully-connected ones, and presumably have stronger capability to learn long-range dependencies than 2-DRFWL(2) GNNs. We verify this assertion by comparing the performance of 2-DRFWL(2)
Table 9: MAE results on QM9 (smaller the better).
Target	1-GNN	1-2-GNN	1-2-3-GNN	SSWL+	LFWL(2)	SLFWL(2)	2-DRFWL(2) GNN
μ	0.493	0.493	0.476	0.418	0.439	0.435	0.346
α	0.78	0.27	0.27	0.271	0.315	0.289	0.222
εhomo	0.00321	0.00331	0.00337	0.00298	0.00332	0.00308	0.00226
εlumo	0.00355	0.00350	0.00351	0.00291	0.00332	0.00322	0.00225
Δε	0.0049	0.0047	0.0048	0.00414	0.00455	0.00447	0.00324
R²	34.1	21.5	22.9	18.36	19.10	18.80	15.04
ZPVE	0.00124	0.00018	0.00019	0.00020	0.00022	0.00020	0.00017
U0	2.32	0.0357	0.0427	0.110	0.144	0.083	0.156
U	2.08	0.107	0.111	0.106	0.143	0.121	0.153
H	2.23	0.070	0.0419	0.120	0.164	0.124	0.145
G	1.94	0.140	0.0469	0.115	0.164	0.103	0.156
Cv	0.27	0.0989	0.0944	0.1083	0.1192	0.1167	0.0901
Table 10: Four-runs results on Long Range Graph Benchmark. Methods are categorized into: MPNNs, Transformer-based methods, our method.
Method	Peptides-func (AP ↑)	Peptides-struct (MAE ↓)
GCN	0.5930 ± 0.0023	0.3496 ± 0.0013
GCNII	0.5543 ± 0.0078	0.3471 ± 0.0010
GINE	0.5498 ± 0.0079	0.3547 ± 0.0045
GatedGCN	0.5864 ± 0.0077	0.3420 ± 0.0013
GatedGCN+RWSE	0.6069 ± 0.0035	0.3357 ± 0.0006
Transformer+LapPE	0.6326 ± 0.0126	0.2529 ± 0.0016
SAN+LapPE	0.6384 ± 0.0121	0.2683 ± 0.0043
SAN+RWSE	0.6439 ± 0.0075	0.2545 ± 0.0012
2-DRFWL(2) GNN	0.5953 ± 0.0048	0.2594 ± 0.0038
GNN on QM9 with that of 1-2-GNN, 1-2-3-GNN, SSWL+ GNN, LFWL(2) GNN and SLFWL(2) GNN, among which the first two are proposed by Morris et al. [44], and the others by Zhang et al. [58]. We also include the result of 1-GNN for reference.
We present the results in Table 9. The baseline results for 1-GNN, 1-2-GNN and 1-2-3-GNN are from [44]. For experiments on SSWL+ GNN, LFWL(2) GNN and SLFWL(2) GNN, we implement the three models following the style of PPGN [42]; namely, we use a  $B \times d \times n_{\max} \times n_{\max}$  matrix to store the embeddings of all 2-tuples within a batch, with  $B$  being the batch size,  $d$  being the size of the embedding dimension, and  $n_{\max}$  being the maximal number of nodes among all graphs in the batch.
For each of SSWL+, LFWL(2) and SLFWL(2), we use 5 layers with embedding size 64, and apply layer normalization after each layer. We train for 400 epochs using Adam optimizer with initial learning rate searched from  $\{0.001, 0.002\}$ , and plateau scheduler with patience 10, decay factor 0.9 and minimum learning rate  $10^{-5}$ . The batch size is 64. We report the best MAE (mean absolute error) for each target. The experimental details for 2-DRFWL(2) GNN follow Appendix E.2.2.
From Table 9, we see that although 2-DRFWL(2) GNN outperforms all baseline methods on 8 out of the 12 targets, its performance is inferior to globally expressive models on the targets  $U_{0}, U, H$  and  $G$ . This result implies that the ability of 2-DRFWL(2) GNN to capture long-range interactions is weaker than globally expressive models.
We also evaluate the performance of 2-DRFWL(2) GNN on the Long Range Graph Benchmark (LRGB) [21]. LRGB is a collection of graph datasets suitable for testing GNN models' power to learn from long-range interactions. We select the two graph-level property prediction datasets, Peptides-function and Peptides-struct, from LRGB. The task of Peptides-function is multi-label binary classification, and the average precision (AP) is adopted as the metric. The task of Peptides-struct is multi-label regression, and the mean absolute error (MAE) is adopted as the metric.