SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {\mathcal {C}} ^ {l} \left(v _ {1}, v _ {2}\right) = \operatorname {H A S H} \left(\tilde {\mathcal {C}} ^ {l - 1} \left(v _ {1}, v _ {2}\right), \{\{\tilde {\mathcal {C}} ^ {l - 1} \left(v _ {1}, w\right) \mid w \in Q _ {1} \left(v _ {2}\right) \} \}, \{\{\tilde {\mathcal {C}} ^ {l - 1} \left(v _ {1}, w\right) \mid w \in \mathscr {S P} \left(v _ {1}, v _ {2}\right) \} \}\right). \tag {44}
$$
Note that Equation (44) is slightly different from the original version of GDGNN [23] stated in their paper. In the original paper, the author use MPNNs in  $O(n)$  space. Meanwhile, they only consider the direct neighbor of node  $v_{1}$  and  $v_{2}$  in  $\mathcal{SP}(v_1,v_2)$ , and they only add it in the last layer. However, it is easy to verify that Equation (44) is at least as powerful as the original version and we show that the  $(k,t)$ -FWL+ can be more powerful than this version. Now we prove Proposition 3.10. We restate it here:
Proposition C.12. Let  $t = 2$ ,  $k = 2$ , and  $ES^2(\mathbf{v}) = Q_1(v_2) \times \mathcal{SP}(v_1, v_2)$ , the corresponding  $(k, t)$ -FWL+ is more powerful than GDGNN [23].
Proof. Given  $t = 2$ ,  $k = 2$ , and  $ES^2(\mathbf{v}) = Q_1(v_2) \times \mathcal{SP}(v_1, v_2)$ , the color update equation of  $(k, t)$ -FWL+ can be written as:
$$
\begin{array}{l} \hat {\mathcal {C}} ^ {l} \left(v _ {1}, v _ {2}\right) = \operatorname {H A S H} \left(\hat {\mathcal {C}} ^ {l - 1} \left(v _ {1}, v _ {2}\right), \{\{\{\{\left(\hat {\mathcal {C}} ^ {l - 1} \left(u _ {1}, u _ {2}\right) \mid \left(u _ {1}, u _ {2}\right) \in Q _ {\left(w _ {1}, w _ {2}\right)} ^ {F} \left(v _ {1}, v _ {2}\right)\right)\right.\left. \right. \tag {45} \\ \left. \left| w _ {1} \in Q _ {1} \left(v _ {2}\right) \right\rbrace \right\rbrace \left| w _ {2} \in \mathcal {S P} \left(v _ {1}, v _ {2}\right) \right\rbrace \bigg). \\ \end{array}
$$
First, as  $(v_{1},w_{1})\in Q_{(w_{1},w_{2})}^{F}(v_{1},v_{2})$ , we can directly conclude that  $(k,t)$ -FWL+ can injectively encode  $\{\{\hat{C}^l (v_1,w_1)|w_1\in Q_1(v_2)\}\}$  for any  $l$ .
Second, we also have  $(v_{1},w_{2})\in Q^{F}_{(w_{1},w_{2})}(v_{1},v_{2})$ . Further, as in the inner multiset, each element has the same  $w_{2}$ . This means that the result of the inner multiset will not affect the information in the outer multiset and it is easy to verify  $(k,t)$ -FWL+ can injectively encode  $\{\hat{C}^l (v_1,w_2)|w_2\in \mathcal{SP}(v_1,v_2)\}$ .
Now, given any two graphs  $G$ ,  $H$  with  $\hat{\mathcal{C}}^{\infty}(G) = \hat{\mathcal{C}}^{\infty}(H)$ . For any  $(v_{11}, v_{12}) \in V^{2}(G)$  and  $(v_{21}, v_{22}) \in V^{2}(H)$  with  $\hat{\mathcal{C}}^{\infty}(v_{11}, v_{12}) = \hat{\mathcal{C}}^{\infty}(v_{21}, v_{22})$  we must have  $\{\{\hat{\mathcal{C}}^{\infty}(v_{11}, w_{11})|w_{11} \in Q_1(v_{12})\}\} = \{\{\hat{\mathcal{C}}^{\infty}(v_{21}, w_{21})|w_{21} \in Q_1(v_{22})\}\}$  and  $\{\{\hat{\mathcal{C}}^{\infty}(v_{11}, w_{12})|w_{12} \in \mathcal{SP}(v_{11}, v_{12})\}\} = \{\{\hat{\mathcal{C}}^{\infty}(v_{21}, w_{22})|w_{22} \in \mathcal{SP}(v_{21}, v_{22})\}\}$  based on two statements. Then, it is easy to verify that if we use the stable color from  $(k, t)$ -FWL+ as the initial color to do one more iteration of GDGNN, the color of  $(v_{11}, v_{12})$  and  $(v_{21}, v_{22})$  cannot be further refined so as the final graph color histogram. This concludes the proof.
C.2 Discussion on the complexity
The space complexity of  $(k,t)$ -FWL+ is still  $O(n^{k})$ . However, the time complexity of  $(k,t)$ -FWL+ is dependent on the choice of  $ES(\mathbf{v})$  and it is hard to give a formal analysis.
C.3 Limitations
In the paper, we implement several different instances of  $(k,t)$ -FWL+ which expressive power is closely related to existing expressive GNNs. However, the whole space of  $(k,t)$ -FWL+ is less explored. Especially, how many different  $ES(\mathbf{v})$  exists? Moreover, how do theoretically and quantitatively analyze the expressive power of  $(k,t)$ -FWL+? Finally, although  $(k,t)$ -FWL+ can be arbitrarily powerful even within  $O(n^2)$  space from the theoretical side, we observe it is sometimes hard to optimize the model to achieve its theoretical power by using fewer embedding. Specifically, how to ensure the injectiveness of the multiset function? For example, if we use large  $t$  and small  $k$  for  $(k,t)$ -FWL+ to conduct a graph isomorphism test, for each  $k$ -tuple, the hierarchical multiset  $\{\{\}\}_{t}$  contains exponentially increased elements. So far, we only use the summation as the function to encode this hierarchical multiset. Although it is an injective function from a theoretical view, we find it works badly with information loss when the number of elements is too large. It is worth investigating how to design model architecture that can unleash the full power of  $(k,t)$ -FWL+. We leave all these to our future work.
D Additional discussion on  $\mathbf{N}^2$ -FWL and  $\mathbf{N}^2$ -GNN
D.1 Detailed proofs for  $\mathbf{N}^2$ -FWL
In this section, we provide all detailed proofs for  $\mathbf{N}^2$ -FWL. To make it clear, we denote  $\hat{\mathcal{C}}^l(v_1, v_2)$  as the color of tuple  $(v_1, v_2)$  at iteration  $l$  for  $\mathbf{N}^2$ -FWL. We rewrite Equation 9 as:
$$
\begin{array}{l} \hat {\mathcal {C}} ^ {l} \left(v _ {1}, v _ {2}\right) = \operatorname {H A S H} \left(\hat {\mathcal {C}} ^ {l - 1} \left(v _ {1}, v _ {2}\right), \{\{\left(\hat {\mathcal {C}} ^ {l - 1} \left(u _ {1}, u _ {2}\right) \mid \left(u _ {1}, u _ {2} \in Q _ {\left(w _ {1}, w _ {2}\right)} ^ {F} \left(v _ {1}, v _ {2}\right)\right) \right. \right. \tag {46} \\ \left. \left(w _ {1}, w _ {2}\right) \in \left(\mathcal {N} _ {1} (v _ {2}) \times \mathcal {N} _ {1} (v _ {1})\right) \cap \left(\mathcal {N} _ {h} (v _ {1}) \cap \mathcal {N} _ {h} (v _ {2})\right) ^ {2} \} \right\}. \\ \end{array}
$$
We start with the first lemma:
Lemma D.1. Given  $h$  large enough,  $N^2$ -FWL is more powerful than SLFWL(2) [12].
Proof. Given  $h$  large enough, it is clearly that the neighbor of  $\mathbf{N}^2$ -FWL becomes  $\mathcal{N}_1(v_2) \times \mathcal{N}_1(v_1)$ . For any  $(w_1, w_2) \in \mathcal{N}_1(v_2) \times \mathcal{N}_1(v_1)$ , it is easy to verify that  $Q_{(w_1, w_2)}^F(v_1, v_2)$  contains all tuples in the aggregation of SLFWL(2). Further, the elements in  $\mathcal{N}_1(v_2) \times \mathcal{N}_1(v_1)$  include all elements in  $Q_1(v_2) \cup Q_1(v_1)$ . Therefore, it is easy to prove that  $\mathbf{N}^2$ -FWL is more powerful than SLFWL(2) by doing induction on  $l$ . Here we omit the detailed proof.
Lemma D.2. Given  $h$  large enough,  $N^2$ -FWL is more powerful than edge-based subgraph GNNs.
Proof. Given  $h$  large enough, it is clear that the neighbor of  $\mathbf{N}^2$ -FWL becomes  $\mathcal{N}_1(v_2) \times \mathcal{N}_1(v_1)$ . The  $\mathbf{N}^2$ -FWL exactly matches the instance in Proposition 3.8 with additional root nodes. As the initial feature of root tuples  $((v_1, v_1), (v_2, v_2))$  are different from other tuples, its information will not affect other neighbors. This concludes the proof.
It is intuitive to conjecture that  $\mathbf{N}^2$ -FWL with the number of  $h$  is more powerful than edge-based subgraph GNNs with  $h$ -hop subgraph and we leave the detailed proof in the future. Now, we can prove Theorem 4.1 in the main paper. We restate it here:
Theorem D.3. Given  $h$  is large enough,  $N^2$ -FWL is more powerful than SLFWL(2) [12] and edge-based subgraph GNNs.
Proof. Lemma D.1 and Lemma D.2 directly implies the theorem. This concludes the proof.
Next, we prove Theorem 4.3 in the main paper. We restate it here:
Theorem D.4.  $N^2$ -FWL can count up to (1) 6-cycles; (2) all connected graphlets with size 4; (3) 4-paths at node level.
Proof. Given Theorem 4.1,  $\mathbf{N}^2$ -FWL is more powerful than  $\mathrm{I}^2$ -GNN [19]. Given Theorem 4.1, 4.2, 4.3 in [19],  $\mathrm{I}^2$ -GNN [19] is able to count up to (1) 6-cycles; (2) all connected graphlets with size 4; (3) 4-paths at node level. This concludes the proof.
Then, we prove Corollary 4.2 in the main paper. We restate it here:
Corollary D.5.  $N^2$ -FWL is strictly more powerful than all existing node-based subgraph GNNs and no less powerful than 3-WL.
Proof. Given Theorem 4.1, this corollary can be directly concluded given Theorem 7.1 in [12] and Proposition 4.1 in [19]. But we further give an example to show that even with  $h = 1$ ,  $\mathbf{N}^2$ -FWL is already powerful enough to distinguish some non-isomorphic strongly regular graph pairs that 3-WL cannot distinguish. When  $h = 1$ , the neighbors of  $\mathbf{N}^2$ -FWL is  $(\mathcal{N}_1(v_2) \times \mathcal{N}_1(v_1)) \cap (\mathcal{N}_1(v_1) \cap \mathcal{N}_1(v_2))^2$ . Consider the Shrikhande graph and  $4 \times 4$  Rook's graph in Figure 2. It is well known this pair of graphs cannot be distinguished by 3-WL. First, look at node  $v_1$  and  $v_2$  in Shrikhande graph,  $\mathcal{N}_1(v_1) \cap \mathcal{N}_1(v_2)$  includes two green nodes,  $v_1$ , and  $v_2$ . This means that  $(w_1, w_2) \in (\mathcal{N}_1(v_2) \times \mathcal{N}_1(v_1)) \cap (\mathcal{N}_1(v_1) \cap \mathcal{N}_1(v_2))^2$  contains a pair of  $(w_1, w_2)$  such that  $w_1$  is one green node and  $w_2$  is the other. Given Definition A.2,  $(w_1, w_2) \in Q_{(w_1, w_2)}^F(v_1, v_2)$ . This means that the information of  $\hat{\mathcal{C}}^0(w_1, w_2)$  will be encoded into  $\hat{\mathcal{C}}^1(v_1, v_2)$  and we have  $(w_1, w_2)$  is not an
Figure 2: The Shrikhande graph and  $4 \times 4$  Rook's graph.
edge. Similarly, in  $4 \times 4$  Rook's graph,  $\mathcal{N}_1(v_1) \cap \mathcal{N}_1(v_2)$  includes two green nodes,  $v_1$ , and  $v_2$ . Thus,  $\hat{\mathcal{C}}^0(w_1, w_2)$  will be encoded into  $\hat{\mathcal{C}}^1(v_1, v_2)$ . However,  $(w_1, w_2)$  here is an edge, which will have a different isomorphism type than  $(w_1, w_2)$  in the Shrikhande graph, thus resulting in different  $\hat{\mathcal{C}}^1(v_1, v_2)$  and further graph color histogram. This concludes the proof.
Finally, we prove Proposition 4.4 in the main paper. We restate it here:
Proposition D.6. If  $\mathbf{U}^l$ ,  $\mathbf{M}^l$ , and  $\mathbf{R}$  are injective functions, there exist a  $N^2$ -GNN instance that can be as powerful as  $N^2$ -FWL.
Proof. If  $\mathbf{U}^l$ ,  $\mathbf{M}^l$ , and  $\mathbf{R}$  are injective, they are exactly the same as HASH used in  $\mathrm{N}^2$ -FWL. Therefore, it is easy to prove it by doing induction on  $l$  to show that if  $\mathrm{N}^2$ -GNN outputs the same color for two tuples,  $\mathrm{N}^2$ -FWL also does for any  $l \geq 0$  and vice versa. We omit the detailed proof here.
D.2 Complexity analysis
In this section, we provide a complexity analysis of  $\mathbf{N}^2$ -GNN. Let  $m$  denote the number of edges in the graph and  $d$  denote the average degree of the graph. First, we consider  $h$  is large enough, where the neighbor of  $\mathbf{N}^2$ -GNN becomes  $\mathcal{N}_1(v_2) \times \mathcal{N}_1(v_1)$ . As  $\mathbf{N}^2$ -GNN is an instance of (2,2)-FWL, it is straightforward that the space complexity of  $\mathbf{N}^2$ -GNN is upper-bounded by  $O(n^2)$ . Further, each tuple needs to aggregate  $(d + 1)^2$  different neighbors, which results in a total time complexity of  $O(n^2 d^2) = O(m^2)$ . Notes that the time complexity of  $\mathbf{N}^2$ -GNN is the same as edge-based subgraph GNNs. However, the space complexity of  $\mathbf{N}^2$ -GNN is  $O(n^2)$ , which is far less than edge-based subgraph GNNs with a  $O(nm)$  space complexity.
If we consider a pratical  $h$ , the neighbors of  $\mathbf{N}^2$ -GNN is  $(\mathcal{N}_1(v_2) \times \mathcal{N}_1(v_1)) \cap (\mathcal{N}_h(v_1) \cap \mathcal{N}_h(v_2))^2$ . This means, only  $(w_1, w_2)$  that within the overlapping  $h$ -hop subgraph of  $v_1$  and  $v_2$  can send message to  $v_1, v_2$ . Therefore, the time complexity is bounded by  $O(nd^{h + 2})$ , which is still the same as edge-based subgraph GNNs with  $h$ -hop subgraph selection policy. Further, it is easy to verify that only tuple  $(v_1, v_2)$  with  $\mathrm{SPD}(v_1, v_2) \leq h$  can update its representation. Thus, in memory-sensitive tasks, we can only keep representation for tuples that can be updated, resulting in a space complexity of  $O(nd^h)$ .
D.3 Model implementation details
Here we provide the details about model implementation. As  $\mathbf{N}^2$ -GNN is an instance of (2, 2)-FWL+, we are dealing with a tuple of size 6 in the aggregation process. Note that for any tuple  $(v_{1}, v_{2})$ ,  $Q_{(w_{1}, w_{2})}^{F}(v_{1}, v_{2})$  will also include itself. However, it does not increase the expressive power as
we already encode its information. Therefore, in the implementation, we remove it from the tuple. Thus, we implement the neighbor tuple as follows: for any 2-tuple  $(v_{1}, v_{2})$ , the message from each neighbor  $(w_{1}, w_{2}) \in N^{2}(v_{1}, v_{2})$  is defined as  $((v_{1}, w_{1}), (v_{1}, w_{2}), (w_{1}, v_{2}), (w_{2}, v_{2}), (w_{1}, w_{2}))$ . Note the order of tuples is flexible as long as it is uniform across different neighbors, according to Definition A.2. We implement  $\mathbf{N}^2$ -GNN in a memory-saving way. To make it more clear, we slightly redefine some notations. Let  $h^{l}(v_{1}, v_{2})$  be the representation of tuple  $(v_{1}, v_{2})$ ,  $m_{(v_{1}, v_{2})}^{l}(w_{1}, w_{2})$  be the message from the neighbor  $(w_{1}, w_{2})$ ,  $N^{2}(v_{1}, v_{2})$  be the set of all neighborhoods of  $(v_{1}, v_{2})$  in  $\mathbf{N}^2$ -GNN with the output embedding hidden size of  $hs$ . We let the initial feature of tuple  $h^{0}(v_{1}, v_{2}) = \mathrm{MLP}(l_{G}(v_{2}) + \mathrm{SPD}(v_{1}, v_{2}))$ . The message function is implemented as:
$$
u _ {i} ^ {l} \left(v _ {1}, v _ {2}\right) = \operatorname {L I N} _ {1} \left(h ^ {l - 1} \left(v _ {1}, v _ {2}\right)\right) * \operatorname {T a n h} \left(\mathbf {W} _ {i}\right), \tag {47}
$$
$$
m _ {\left(v _ {1}, v _ {2}\right)} ^ {l} \left(w _ {1}, w _ {2}\right) = \operatorname {L I N} _ {2} \left(\operatorname {C O N} \left(u _ {1} ^ {l} \left(v _ {1}, w _ {1}\right), u _ {2} ^ {l} \left(v _ {1}, w _ {2}\right), u _ {3} ^ {l} \left(w _ {1}, v _ {2}\right), u _ {4} ^ {l} \left(w _ {2}, v _ {2}\right), u _ {5} ^ {l} \left(w _ {1}, w _ {2}\right)\right)\right), \tag {48}
$$
where  $\mathrm{LIN}_1$  is a linear layer project the embedding from hidden size  $h$  to a lower value  $d$ , named as inner size,  $\mathbf{W}_i \in R^d$ ,  $i \in [5]$  are learnable weight vectors with size  $d$  and  $*$  is element-wise product, CON is the concatenation, and  $\mathrm{LIN}_2$  is another linear layer project the concatenated embedding from size  $5 * d$  back to hidden size  $h$ . It is easy to validate that this implementation is injective to tuple input. Next, the update function  $\mathbf{U}^l$  is implemented as:
$$
h ^ {l} \left(v _ {1}, v _ {2}\right) = \mathbf {M L P} ^ {l} \left(\left(1 + \epsilon^ {l}\right) * h ^ {l - 1} \left(v _ {1}, v _ {2}\right) + \sum_ {\left(w _ {1}, w _ {2}\right) \in N ^ {2} \left(v _ {1}, v _ {2}\right)} m _ {\left(v _ {1}, v _ {2}\right)} ^ {l} \left(w _ {1}, w _ {2}\right)\right), \tag {49}
$$
where  $\epsilon^l$  is a learnable scalar and  $\mathrm{MLP}^l$  is a one-hidden-layer MLP with hidden size and output size equal to  $h$  and activation function be ReLu. Note that in the implementation we do not consider the hierarchical multiset but directly pooling all neighbors together. We observe it already achieves superior results in all real-world tasks and is more parameter-saving.
As discussed in [12, 27], for each  $h^{l - 1}(v_1,v_2)$ , add the representation of  $h^{l - 1}(v_2,v_2)$  can boost the performance in real-world tasks. Therefore, we adopt a similar way for all real-world experiments:
$$
a ^ {l} \left(v _ {1}, v _ {2}\right) = \mathrm {M L P} _ {a} ^ {l} \left(\left(1 + \epsilon_ {a} ^ {l}\right) * h ^ {l - 1} \left(v _ {1}, v _ {2}\right) + h ^ {l - 1} \left(v _ {2}, v _ {2}\right)\right), \tag {50}
$$
$$
\hat {h} ^ {l} \left(v _ {1}, v _ {2}\right) = h ^ {l} \left(v _ {1}, v _ {2}\right) + a ^ {l} \left(v _ {1}, v _ {2}\right). \tag {51}
$$
If add this additional aggregation, we will use the  $\hat{h}^l (v_1,v_2)$  instead of  $h^l (v_1,v_2)$  as the output representation. We conjecture this works as virtual nodes in real-work tasks, which allows a better mixture of node features and structure information. As indicated by [12], adding this term will not increase the expressive power from the theoretical side. Finally, the readout function is implemented in a hierarchical way:
$$
h \left(v _ {1}\right) = \operatorname {M L P} \left(\operatorname {P O O L} _ {1} \left(\{\left\{h ^ {L} \left(v _ {1}, w\right) \mid w \in V (G) \right\}\right)\right), \left. \right. \tag {52}
$$
$$
h _ {G} = \operatorname {P O O L} _ {2} \left(\{\{h (w) | w \in V (G) \}\}\right), \tag {53}
$$
where  $\mathrm{POOL}_1$  and  $\mathrm{POOL}_2$  are two different pooling function.
In addition, we adopted two implementation strategies for  $\mathbf{N}^2$ -GNN. The first strategy is to keep all  $O(n^{2})$  tuples even if it has a distance greater than  $h$ . Notes that it will not aggregate information from other tuples but is only updated in Equation (50) and Equation (51). We observe it has great performance in real-world tasks combined with Equation (52) and Equation (53). The second implementation is used for memory-sensitive tasks. We remove all tuples that have a distance greater than  $h$ . We refer to the first strategy as the dense version and the second strategy as the sparse version. We will specify which version we used in our experiments later on. The code is implemented using PyG [52]. We provide open-sourced code in https://github.com/JiaruiFeng/N2GNN for reproducing all results reported in the main paper.
D.4 Limitations
Although  $\mathbf{N}^2$ -GNN has efficient theoretical space complexity, the empirical space complexity can still be high as we need to keep neighbors index for all tuple  $(u_1, u_2) \in Q_{(w_1, w_2)}^F(v_1, v_2)$ . This introduces
a constant factor (5 in the implementation) on the saving edge index and corresponding gradients during training (see Table 9 and Table 10 for the practical time and memory comparison). It is still worth studying how to improve the empirical space complexity of  $\mathrm{N}^2$ -GNN. Further, in the current implementation, we didn't consider the hierarchical set but directly pooled all elements together to save the parameters and computational cost. Additionally, each tuple needs to aggregate many more neighbors than normal MPNNs, which makes the sum aggregation hard to achieve injectiveness during the aggregation, introducing the optimization issue. These hinder the model from achieving its theoretical power. Finally, although  $\mathrm{N}^2$ -GNN has  $O(nd^{h + 2})$  time complexity, which is practical for most real-world tasks, the complexity still goes exponentially if the graph is dense (like a strongly regular graph). But as the design space of  $(k,t)$ -FWL+ is extremely flexible, it is easy to design more practical and expressive instances suitable for dense graphs.
This section presents additional information on the experimental settings.
E.1 Dataset statistics
Table 6: Dataset statistics.
Dataset	#Graphs	Avg. #Nodes	Avg. #Edges	Task type
EXP	1200	44.4	110.2	Graph classification
CSL	150	41.0	164.0	Graph classification
SR25	15	25.0	300.0	Graph classification
BREC	800	34.8	143.4	Pair distinguishment
Counting	5,000	18.8	31.3	Node regression
QM9	130,831	18.0	18.7	Graph regression
ZINC-12k	12,000	23.2	24.9	Graph regression
ZINC-250k	249,456	23.1	24.9	Graph regression
E.2 Experimental details
In this section, we provide all the experimental details. First, we discuss some general experimental settings that, if not specified further, are consistent across all experiments.
Model. We adopt  $\mathrm{POOL}_1$  as summation and  $\mathrm{POOL}_2$  as mean pooling in Equation (52) and Equation (53). We set the  $\epsilon^l$  in Equation (49),  $\epsilon_a^l$  in Equation (50) to be 0.0 and fix it during training and inference. At each MLP, we add batch normalization.
Training. For all experiments, we use Adam as the optimizer and ReduceLRonPlateau as the learning rate scheduler. We set 12 weight decay to 0.0 across all experiments. All experiments are conducted on a single NVIDIA A100 80GB GPU.
E.2.1 Graph structures counting
Model. For the substructure counting dataset, we adopt a similar setting as  $\mathrm{I}^2$ -GNN [19]. Specifically, we set the number of hop  $h$  to be 1, 2, 2, 3, 2, 2, 1, 4, 2 for 3-cycles, 4-cycles, 5-cycles, 6-cycles, tailed triangles, chordal cycles, 4-cliques, 4-paths, and triangle-rectangle respectively. For all substructures, we set the number of layers to 5, the hidden size  $h$  to 96, and the inner size  $d$  to 32. In the substructure count dataset, we remove all normalization layers. Finally, we use the sparse version of data preprocessing.
Training. The training/validation/test splitting ratio is 0.3/0.2/0.5. The initial learning rate is 0.001 and the minimum learning rate is 1e-5. The patience and factor of the scheduler are 10 and 0.9 respectively. The batch size is set to 256 and the number of epochs is 2000. For all substructures, we run the experiments 3 times and report the mean results on the test dataset.
E.2.2 ZINC
Model. The setting is the same for both ZINC-Subset and ZINC-Full except for the inner size. We set the number of hop  $h$  to be 3. We set the number of layers to 6, and the hidden size  $h$  to 96. The inner size  $d$  is 20 for the ZINC-subset and 48 for the ZINC-full.
Training. The initial learning rate is 0.001 and the minimum learning rate is 1e-6. The patience and factor of the scheduler are 20 and 0.5 respectively. The batch size is set to 128 and the number of epochs is 500. For both ZINC-Subset and ZINC-Full, we run experiments 10 times and report the mean results and standard deviation on the test dataset.
E.2.3 QM9
Model. For all targets, we adopt the same setting. We set the number of hop  $h$  to be 3. We set the number of layers to 6, the hidden size  $h$  to 120, and the inner size  $d$  to 32. We further add the resistance distance feature into the model, similar to the implementation in Feng et al. [22], Huang et al. [19].
Training. The initial learning rate is 0.001. The patience and factor of the scheduler are 5 and 0.9 respectively. The batch size is set to 128 and the number of epochs is 350. we run experiments 1 time and report the test result with the model of the best validation result.
E.2.4 CSL
Model: We set the number of hop  $h$  to be 4, the number of layers to be 4, the hidden size  $h$  to 48, and the inner size  $d$  to 16. We use the sparse version of data preprocessing and we do not add root aggregation (Equation (50)-(51)).
Training. The initial learning rate is 0.001 and the minimum learning rate is 1e-6. The patience and factor of the scheduler are 20 and 0.5 respectively. The batch size is set to 32 and the number of epochs is 80. We use 10-fold cross-validation and report the average results.
E.2.5 SR25
Model: We set the number of hop  $h$  to be 1, the number of layers to be 6, the hidden size  $h$  to 64, and the inner size  $d$  to 16. We use the sparse version of data preprocessing and we do not add root aggregation (Equation (50)-(51)). Further, the normalization layer in SR25 is set to Layer normalization to avoid test/training mismatch.
Training. The initial learning rate is 0.001 and the minimum learning rate is 1e-6. The patience and factor of the scheduler are 200 and 0.5 respectively. The batch size is set to 15 and the number of epochs is 800. We report the single-time results.
E.2.6 EXP
Model: We set the number of hop  $h$  to be 3, the number of layers to be 4, the hidden size to 48, and the inner size  $d$  to 24. We use the sparse version of data preprocessing and we do not add root aggregation (Equation (50)-(51)).
Training. The initial learning rate is 0.001 and the minimum learning rate is 1e-6. The patience and factor of the scheduler are 20 and 0.5 respectively. The batch size is set to 32 and the number of epochs is 200. We use 10-fold cross-validation and report the average results.
E.2.7 BREC