Title: Extending the Design Space of Graph Neural Networks by Rethinking Folklore Weisfeiler-Lehman
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph neural network

expressive power

Folklore Weisfeiler-Lehman test.
Abstract: Message passing neural networks (MPNNs) have emerged as the most popular framework of graph neural networks (GNNs) in recent years. However, their expressive power is limited by the 1-dimensional Weisfeiler-Lehman (1-WL) test. Some works are inspired by $k$-WL/FWL (Folklore WL) and design the corresponding neural versions. Despite the high expressive power, there are serious limitations in this line of research. In particular, (1) $k$-WL/FWL requires at least $O(n^k)$ space complexity, which is impractical for large graphs even when $k=3$; (2) The design space of $k$-WL/FWL is rigid, with the only adjustable hyper-parameter being $k$. To tackle the first limitation, we propose an extension, $(k, t)$-FWL. We theoretically prove that even if we fix the space complexity to $O(n^k)$ (for any $k \geq 2$) in $(k, t)$-FWL, we can construct an expressiveness hierarchy up to solving the graph isomorphism problem. To tackle the second problem, we propose $k$-FWL+, which considers any equivariant set as neighbors instead of all nodes, thereby greatly expanding the design space of $k$-FWL. Combining these two modifications results in a flexible and powerful framework $(k, t)$-FWL+. We demonstrate $(k, t)$-FWL+ can implement most existing models with matching expressiveness. We then introduce an instance of $(k,t)$-FWL+ called Neighborhood$^2$-FWL (N$^2$-FWL), which is practically and theoretically sound. We prove that N$^2$-FWL is no less powerful than 3-WL, and can encode many substructures while only requiring $O(n^2)$ space. Finally, we design its neural version named **N$^2$-GNN** and evaluate its performance on various tasks. N$^2$-GNN achieves record-breaking results on ZINC-Subset (**0.059**), outperforming previous SOTA results by 10.6\%. Moreover, N$^2$-GNN achieves new SOTA results on the BREC dataset (**71.8\%**) among all existing high-expressive GNN methods.

Extending the Design Space of Graph Neural Networks by Rethinking Folklore Weisfeiler-Lehman

Abstract

Message passing neural networks (MPNNs) have emerged as the most popular framework of graph neural networks (GNNs) in recent years. However, their expressive power is limited by the 1-dimensional Weisfeiler-Lehman (1-WL) test. Some works are inspired by k -WL/FWL (Folklore WL) and design the corresponding neural versions. Despite the high expressive power, there are serious limitations in this line of research. In particular, (1) k -WL/FWL requires at least O ( n k ) space complexity, which is impractical for large graphs even when k = 3 ; (2) The design space of k -WL/FWL is rigid, with the only adjustable hyper-parameter being k . To tackle the first limitation, we propose an extension, ( k, t ) -FWL. We theoretically prove that even if we fix the space complexity to O ( n k ) (for any k ≥ 2 ) in ( k, t ) -FWL, we can construct an expressiveness hierarchy up to solving the graph isomorphism problem. To tackle the second problem, we propose k -FWL+, which considers any equivariant set as neighbors instead of all nodes, thereby greatly expanding the design space of k -FWL. Combining these two modifications results in a flexible and powerful framework ( k, t ) -FWL+. We demonstrate ( k, t ) -FWL+ can implement most existing models with matching expressiveness. We then introduce an instance of ( k, t ) -FWL+ called Neighborhood 2 -FWL (N 2 -FWL), which is practically and theoretically sound. We prove that N 2 -FWL is no less powerful than 3-WL, and can encode many substructures while only requiring O ( n 2 ) space. Finally, we design its neural version named N 2 -GNN and evaluate its performance on various tasks. N 2 -GNN achieves record-breaking results on ZINC-Subset ( 0.059 ) outperforming previous SOTA results by 10.6%. Moreover, N 2 -GNN achieves new SOTA results on the BREC dataset ( 71.8% ) among all existing high-expressive GNN methods.

1 Introduction

In recent years, graph neural networks (GNNs) have become one of the most popular and powerful methods for graph representation learning, following a message passing framework [1-4]. However, the expressive power of message passing GNNs is bounded by the one-dimensional WeisfeilerLehman (1-WL) test [5, 6]. As a result, numerous efforts have been made to design GNNs with higher expressive power. We provide a more detailed discussion in Section 5.

Several works have drawn inspiration from the k -dimensional Weisfeiler-Lehman ( k -WL) or Folklore Weisfeiler-Lehman ( k -FWL) test [7] and developed corresponding neural versions [6, 8-10]. However, k -WL/FWL has two inherent limitations. First, while the expressive power increases

with higher values of k , the space and time complexity also grows exponentially, requiring O ( n k ) space complexity and O ( n k +1 ) time complexity, which makes it impractical even when k = 3 . Thus, the question arises: Can we retain high expressiveness without exploding both time and space complexities? Second, the design space of WL-based algorithms is rigid, with the only adjustable hyper-parameter being k . However, there is a significant gap in expressive power even between consecutive values of k , making it hard to fine-tune the tradeoffs. Moreover, increasing the expressive power does not necessarily translate into better real-world performance, as it may lead to overfitting [8, 11]. Although some works try to tackle this problem [8, 10], there is still limited understanding of how to expand the design space of the original k -FWL to a broader space that enables us to identify the most appropriate instance to match the complexity of real-world tasks.

To tackle the first limitation, we notice that k -FWL and k -WL have the same space complexity but k -FWL can achieve the same expressive power as ( k +1)-WL. We found the key component that allows k -FWL to have stronger power is the tuple aggregation style. Enlightened by this observation, we propose ( k, t ) -FWL, which extends the tuple aggregation style in k -FWL. Specifically, in the original k -FWL, a neighbor of a k -tuple is defined by iteratively replacing its i -th element with a node u , and u traverses all nodes in the graph. In ( k, t ) -FWL, we extend a single node u to a t -tuple of nodes and carefully design a replacement scheme to insert the t -tuple into a k -tuple to define its neighbor. We demonstrate that even with a fixed space complexity of O ( n k ) (for any k ≥ 2 ), ( k, t ) -FWL can construct an expressive hierarchy capable of solving the graph isomorphism problem. To deal with the second limitation, we revisit the definition of neighborhood in k -FWL. Inspired by previous works [8, 12] which consider only local neighbors (i.e., u must be connected to the k -tuple) instead of global neighbors in k -WL/FWL, we find that the neighborhood (i.e., which u are used to construct a k -tuple's neighbors) can actually be extended to any equivariant set related to the k -tuple, resulting in k -FWL+. Combining the two modifications leads to a novel and powerful FWL-based algorithm ( k, t ) -FWL+. ( k, t ) -FWL+ is highly flexible and can be used to design different versions to fit the complexity of real-world tasks. Based on the proposed ( k, t ) -FWL+ framework, we implement many different instances that are closely related to existing powerful GNN/WL models, further demonstrating the flexibility of ( k, t ) -FWL+.

Finally, we propose an instance of (2 , 2) -FWL+ named Neighborhood 2 -FWL that is both theoretically expressive and practically powerful. It considers the local neighbors of both two nodes in a 2-tuple. Despite having a space complexity of O ( n 2 ) , which is lower than 3-WL's O ( n 3 ) space complexity, this instance can still partially outperform 3-WL and is able to count many substructures. We implement a neural version named Neighborhood 2 -GNN (N 2 -GNN) and evaluate its performance on various synthetic and real-world datasets. Our results demonstrate that N 2 -GNN outperforms existing SOTA methods across most tasks. Particularly, it achieves 0.059 in ZINC-Subset, surpassing existing state-of-the-art models by significant margins. Meanwhile, it achieves 71.8% on the BREC dataset, the new SOTA among all existing high-expressive GNN methods.

2 Preliminaries

Notations. Let {·} denote a set, { {·} } denote a multiset (a set that allows repetition), and ( · ) denote a tuple. As usual, let [ n ] = { 1 , 2 , . . . , n } . Let G = ( V ( G ) , E ( G ) , l G ) be an undirected, colored graph, where V ( G ) = [ n ] is the node set with n nodes, E ( G ) ⊆ V ( G ) × V ( G ) is the edge set, and l G : V ( G ) → C is the graph coloring function with C = { c 1 , . . . , c d } denote a set of d distinct colors. Let N k ( v ) denote a set of nodes within k hops of node v and Q k ( v ) denote the k -th hop neighbors of node v and we have N k ( v ) = ⋃ k i =0 Q i ( v ) . Let SPD ( u, v ) denote the shortest path distance between u and v . We use x v ∈ R d n to denote attributes of node v ∈ V ( G ) and e uv ∈ R d e to denote attributes of edge ( u, v ) ∈ E ( G ) . They are usually the one-hot encoding of the node and edge color respectively. We say that two graphs G and H are isomorphic (denoted as G ≃ H ) if there exists a bijection φ : V ( G ) → V ( H ) such that ∀ u, v ∈ V ( G ) , ( u, v ) ∈ E ( G ) ⇔ ( φ ( u ) , φ ( v )) ∈ E ( H ) and ∀ v ∈ V ( G ) , l G ( v ) = l H ( φ ( v )) . Denote V ( G ) k the set of k -tuples of vertices and v = ( v 1 , . . . , v k ) ∈ V ( G ) k a k -tuple of vertices. Let S n denote the permutation group of [ n ] and g ∈ S n : [ n ] → [ n ] be a particular permutation. When a permutation g ∈ S n operates on any target X , we denote it by g · X . Particularly, a permutation operating on an edge set E ( G ) is g · E ( G ) = { ( g ( u ) , g ( v )) | ( u, v ) ∈ E ( G ) } . A permutation operating on a k -tuple v is g · v = ( g ( v 1 ) , . . . , g ( v k )) . Apermutation operating on a graph is g · G = ( g · V ( G ) , g · E ( G ) , g · l G ) .

k -dimensional Weisfeiler-Lehman test. The k -dimensional Weisfeiler-Lehman ( k -WL) test is a family of algorithms used to test graph isomorphism. There are two variants of the k -WL test: k -WL and k -FWL (Folklore WL). We first describe the procedure of 1-WL, which is also called the color refinement algorithm [7]. Let C 0 1 wl ( v ) = l G ( v ) be the initial color of node v ∈ V ( G ) . At the l -th iteration, 1-WL updates the color of each node using the following equation:

After the algorithm converges, a color histogram is constructed using the colors assigned to all nodes. If the color histogram is different for two graphs, then the two graphs are non-isomorphic. However, if the color histogram is the same for two graphs, they can still be non-isomorphic.

The k -WL and k -FWL, for k ≥ 2 , are generalizations of the 1-WL, which do not color individual nodes but node tuples v ∈ V ( G ) k . Let v w/j be a k -tuple obtained by replacing the j -th element of v with w . That is v w/j = ( v 1 , . . . , v j -1 , w, v j +1 , . . . , v k ) . The main difference between k -WL and k -FWL lies in their aggregation way. For k -WL, the set of j -th neighbors of tuple v is denoted as Q j ( v ) = { v w/j | w ∈ V ( G ) } , j ∈ [ k ] . Instead, the w -neighbor of tuple v for k -FWL is denoted as Q F w ( v ) = ( v w/ 1 , v w/ 2 , . . . , v w/k ) . Let C 0 kwl ( v ) = C 0 kfwl ( v ) be the initial color for k -WL and k -FWL, respectively. They are usually the isomorphism types of tuple v . At the l -th iteration, k -WL and k -FWL update the color of each tuple according to the following equations:

The procedure described above is repeated until convergence, resulting in a color histogram of the graph that can be compared with the histograms of other graphs. Prior research has shown that ( k +1)-WL and ( k +1)-FWL are strictly more powerful than k -WL and k -FWL, respectively, except in the case where 1-WL is equivalent to 2-WL. Additionally, it has been shown that k -FWL is equivalent to ( k +1)-WL [13, 14, 6]. We leave the additional discussion on k -WL/FWL in Appendix A.

Message passing neural networks. Message Passing Neural Networks (MPNNs) are a type of GNNs that update node representations by iteratively aggregating information from their direct neighbors. Let h l v be the output of MPNNs of node v ∈ V ( G ) after l layers and we let h 0 v = x v . At l -th layer, the representation is updated by:

where U l and M l are learnable update and message functions, usually parameterized by multilayer perceptrons (MLPs). After L layers, MPNNs output the final representation h L v for all nodes v ∈ V ( G ) . The graph-level representation is obtained by:

where R is a readout function. The expressive power of MPNNs is at most as powerful as 1-WL [5, 6].

3 Rethinking and extending the Folklore Weisfeiler-Lehman test

3.1 Rethinking and extending the aggregation style in k -FWL

Increasing k can increase the expressive power of k -FWL for distinguishing graph structures. However, increasing k brings significant memory costs, as k -FWL requires O ( n k ) memory. It becomes impractical even when k = 3 [9]. Therefore, it is natural to ask:

Can we achieve higher expressive power without exploding the memory cost?

To achieve this goal, we first notice that k -FWL has a higher expressive power than k -WL but only requires the same O ( n k ) memory cost. To understand why, let's use 2-WL and 2-FWL as examples and rewrite Equation (2) and Equation (3):

𝑣

!

𝑣

"

𝑤

"

𝑤

!

Construction of

Neighborhood tuple

𝑣

!

𝑣

"

𝑣

!

𝑤

!

𝑣

!

𝑤

"

𝑣

"

𝑤

!

𝑣

"

𝑤

"

𝑤

!

𝑤

"

Recover

𝑣

!

𝑣

"

𝑤

"

𝑤

!

𝑣

!

𝑣

"

𝑤

!

𝑤

"

Select 0 element

Select 1 element

𝑤

!

𝑤

"

Select 2 elements

𝑤

!

𝑤

"

Select 0 index

Select 1 index

Select 2 indices

𝑣

!

𝑣

"

𝑣

!

𝑣

"

×

×

×

𝑄

!

!

,!

"

#

(𝑣

$

, 𝑣

%

)

High-order

structures

𝑸

𝒘

𝟏

,𝒘

𝟐

𝑭

(𝒗

𝟏

, 𝒗

𝟐

)

where C l 2 wl ( v 1 , v 2 ) and C l 2 fwl ( v 1 , v 2 ) are the color of tuple ( v 1 , v 2 ) at iteration l for 2-WL and 2-FWL, respectively. We can see the key difference between 2-WL and 2-FWL is that: in 2-FWL, a tuple of color ( C l -1 2 fwl ( v 1 , w ) , C l -1 2 fwl ( w,v 2 )) is aggregated. While in 2-WL, the colors of nodes are considered in separate multisets. To understand why the first aggregation is more powerful, we can further rewrite the update equation of 2-FWL as follows:

It is easy to verify that the above equation is equivalent to the original one. This means that 2-FWL updates the color of tuple ( v 1 , v 2 ) by aggregating different tuples of (( v 1 , v 2 ) , ( v 1 , w ) , ( w,v 2 )) , which can be viewed as aggregating information of 3-tuples ( v 1 , v 2 , w ) . For example, HASH ( C 0 2 fwl ( v 1 , v 2 ) , C 0 2 fwl ( v 1 , w ) , C 0 2 fwl ( w,v 2 )) can fully recover the isomorphism type of tuple ( v 1 , v 2 , w ) used in 3-WL. The above observations can be easily extended to k -FWL. The key insight here is that tuple-style aggregation is the key to lifting the expressive power of k -FWL. Since aggregating k -tuple can boost the expressive power of k -FWL to be equivalent to ( k +1)-WL without the increase of space complexity, we may wonder, can we further extend the size of the tuple?

We first introduce the neighborhood tuple to extend the original w -neighbor used in k -FWL. Let Q F w ( v ) denote a neighborhood tuple related to k -tuple v and t -tuple w . The neighborhood tuple contains all possible results such that we sequentially select m ∈ [0 , 1 , . . . , min ( k, t )] elements from w to replace m elements in v . For each m , we first select all possible combinations of m -tuples (no repeated elements) from tuple w with a pre-defined order to form a new tuple, denoted as P m ( w ) . At the same time, we select all possible combinations of m indices from i = (1 , 2 , . . . , k ) with a pre-defined order to form another new tuple P m ( i ) . Finally, we iterate all ( w ′ , i ′ ) ∈ P m ( w ) × P m ( i ) to get a | P m ( w ) || P m ( i ) | -sub-tuple, where for each ( w ′ , i ′ ) we replace the elements with indices i ′ in the original v with w ′ . The final neighborhood tuple is the concatenation of all sub-tuples. Note that the pre-defined orders can be flexible as long as they are consistent for any k and t .

Here is an example of a possible construction of a neighborhood tuple when k = t = 2 . Let Q F ( w 1 ,w 2 ) ( v 1 , v 2 ) denote the neighborhood tuple. In this case, we need to enumerate m in [0 , 1 , 2] . First, when m = 0 , we select 0 elements from ( w 1 , w 2 ) to replace ( v 1 , v 2 ) , resulting in a sub-tuple with only one element (( v 1 , v 2 )) . Next, when m = 1 . we sequentially select w 1 and w 2 , resulting in P 1 ( w ) = ( w 1 , w 2 ) . Similarly, we can get P 1 ( i ) = (2 , 1) . Thus, the final sub-tuple is (( v 1 , w 1 ) , ( v 1 , w 2 ) , ( w 1 , v 2 ) , ( w 2 , v 2 )) . The case of m = 2 is simple and we have the result sub-tuple be (( w 1 , w 2 )) . By concatenating all three sub-tuples, we have Q F ( w 1 ,w 2 ) ( v 1 , v 2 ) = (( v 1 , v 2 ) , ( v 1 , w 1 ) , ( v 1 , w 2 ) , ( w 1 , v 2 ) , ( w 2 , v 2 ) , ( w 1 , w 2 )) . The construction procedure is shown in Figure 1. By default, we adopt the above order in Q F w ( v ) in the rest of the paper if k = t = 2 . We leave a more formal definition of the neighborhood tuple Q F w ( v ) in Appendix A.

With the neighborhood tuple, we are ready to introduce ( k, t ) -FWL , which extends k -FWL by extending the tuple size in the update function. Let C l ktfwl ( v ) be the color of tuple v at iteration l for ( k, t ) -FWL, the update function of ( k, t ) -FWL is:

where { {·} } t is hierarchical multiset over t -tuples. In a hierarchical multiset { { v | v ∈ V t ( G ) } } t , elements are grouped hierarchically according to the node order of the tuple. For example, to construct { { ( v 1 , v 2 , v 3 ) | ( v 1 , v 2 , v 3 ) ∈ V 3 ( G ) } } 3 from { { ( v 1 , v 2 , v 3 ) | ( v 1 , v 2 , v 3 ) ∈ V 3 ( G ) } } , we first group together all elements with the same v 2 and v 3 . That is, ∀ v 2 , v 3 ∈ V ( G ) , we denote t ( v 2 , v 3 ) = { { ( v 1 , v 2 , v 3 ) | v 1 ∈ V ( G ) } } as the grouped result. Next, use the similar procedure, we have ∀ v 3 ∈ V ( G ) , t ( v 3 ) = { { t ( v 2 , v 3 ) | v 2 ∈ V ( G ) } } . Finally, we group all possible v 3 ∈ V ( G ) to get { { ( v 1 , v 2 , v 3 ) | ( v 1 , v 2 , v 3 ) ∈ V 3 ( G ) } } 3 = { { t ( v 3 ) | v 3 ∈ V ( G ) } } .

It is easy to see ( k, 1) -FWL is equivalent to k -FWL under our definition of Q F w ( v ) . Further, as we only need to maintain representations of all k -tuples. ( k, t ) -FWL has a fixed space complexity of O ( n k ) . Here we show the expressive power of ( k, t ) -FWL.

Proposition 3.1. For k ≥ 2 and t ≥ 1 , if t ≥ n -k , ( k, t ) -FWL can solve the graph isomorphism problems with the size of the graph less than or equal to n .

Theorem 3.2. For k ≥ 2 , t ≥ 1 , ( k, t ) -FWL is at most as powerful as ( k + t ) -WL. In particular, ( k, 1) -FWL is as powerful as ( k +1) -WL.

Proposition 3.3. For k ≥ 2 , t ≥ 1 , ( k, t + 1) -FWL is strictly more powerful than ( k, t ) -FWL; ( k +1 , t ) -FWL is strictly more powerful than ( k, t ) -FWL.

We leave all formal proofs and complexity analysis in Appendix B. Briefly speaking, even if we fix the size of k , ( k, t ) -FWL can still construct an expressive hierarchy by varying t . Further, if t is large enough, ( k, t ) -FWL can actually enumerate all possible combinations of tuples with the size of the graph, and thus equivalent to the relational pooling on graph [15]. It is worth noting that the size of Q F w ( v ) will grow exponentially with an increase in the size of t . However, the key contribution of ( k, t ) -FWL is that even when k = 2 , ( k, t ) -FWL can still construct an expressive hierarchy for solving graph isomorphism problems. Therefore, high-order embedding may not be necessary for building high-expressivity WL algorithms. Note that our ( k, t ) -FWL is also different from subgraph GNNs such as k, l -WL [16] and l -OSAN [17], where l -tuples are labeled independently to enable learning k -tuple representations in all l -tuples' subgraphs, resulting in O ( n k + l ) space complexity.

3.2 Rethinking and extending the aggregation scope of k -FWL

Another problem of k -FWL is its limited design space, as the only adjustable hyperparameter is k . It is well known that there is a huge gap in expressive power even if we increase from k to k +1 . For example, 1-WL cannot count any cycle even with a length of 3, but 2-FWL can already count up to 7-cycle [18-20]. Moreover, increasing the expressive power does not always bring better performance when designing the corresponding neural version as it quickly leads to overfitting [6, 9, 11]. Therefore, we ask another question:

Can we extend the k -FWL to a more flexible and fine-grained design space?

To address this issue, we identify that the inflexibility of k -FWL's design space arises from the definition of the neighbor used in the aggregation step. Unlike 1-WL, k -FWL lacks the concept of local neighbors and instead requires the aggregation of all | V ( G ) | global neighbors to update the color of each tuple v . Recently, some works have extended k -WL by incorporating local information [8, 12]. Inspired by previous works, we find that the definition of neighbor can actually be much more flexible than just considering local neighbors or global neighbors. Specifically, for each k -tuple v in graph G , we define equivariant set ES ( v ) to be the neighbors set of tuple v and propose k -FWL+ .

Definition 3.4. An equivariant set ES ( v ) is a set of nodes related to v and equivariant given the permutation g ∈ S n . That is, ∀ w ∈ ES ( v ) in graph G implies g ( w ) ∈ ES ( g · v ) in graph g · G .

Some nature equivariant sets ES ( v ) including V ( G ) , N k ( v ) , and Q k ( v ) , etc. Let C l kfwl + ( v ) be the color of tuple v at iteration l for k -FWL+, we have:

The key of k -FWL+ is that the equivariant set ES ( v ) can be any set of nodes as long as it is equivariant to any permutation g ∈ S n . For example, if ES ( v ) = V ( G ) , the k -FWL+ will reduce to the original k -FWL. Instead, if ES ( v ) = ⋃ k i =0 N 1 ( v i ) , it becomes the localized k -FWL [12]. We can also design other more innovative equivariant sets ES ( v ) . For example, denote SP ( v ) to be a set that contains all nodes in the shortest paths between any v i , v j ∈ v . It is still a valid equivariant set of tuple v . We can see that the design space of k -FWL+ is much broader than k -FWL.

3.3 ( k, t ) -FWL+: A flexible and powerful extension of k -FWL

In this section, we propose ( k, t ) -FWL+ , which combines both ( k, t ) -FWL in Equation (6) and k -FWL+ in Equation (7). Let C l ktf + ( v ) be the color of tuple v at iteration l for ( k, t ) -FWL+, the color update equation of ( k, t ) -FWL+ is:

where ES t ( v ) = ES 1 ( v ) × ES 2 ( v ) ×··· × ES t ( v ) and ES i ( v ) can be any ES ( v ) . ( k, t ) -FWL+ integrates advantages from both ( k, t ) -FWL and k -FWL+. First, by extending V t ( G ) to ES t ( v ) , we can hugely reduce the time complexity of ( k, t ) -FWL as ES t ( v ) usually contains much less elements than V t ( G ) . Second, by extending the tuple size in aggregation, we only require a small k (even with k = 2 ) to achieve high expressive power instead of being bounded by k -FWL. Finally, the most important advantage is that the design space of ( k, t ) -FWL+ is much more flexible than k -FWL and we can design the most suitable instance for different tasks. However, as the expressive power of ( k, t ) -FWL+ highly relies on the choice of ES t ( v ) , it is hard to analyze the expressive power of ( k, t ) -FWL+ in a systematic way. Instead, in the next section, we provide some practical implementations of ( k, t ) -FWL+ that can cover many existing powerful models.

3.4 Gallery of ( k, t ) -FWL+ instances

In this section, we provide some practical designs of ES t ( v ) that are closely correlated to many existing powerful GNN/WL models. Note that any model directly follows the original k -FWL like PPGN [9] can obviously be implemented by ( k, t ) -FWL+ as ( k, t ) -FWL+ also includes the original k -FWL. We specify that k -tuple v = ( v 1 , v 2 , . . . , v k ) to avoid any confusion.

Proposition 3.5. Let t = 1 , k = 2 , and ES ( v ) = N 1 ( v 1 ) ∪N 1 ( v 2 ) , the corresponding ( k, t ) -FWL+ instance is equivalent to SLFWL(2) [12] and strictly more powerful than any existing node-based subgraph GNNs.

Proposition 3.6. Let t = 1 , k = k , and ES ( v ) = ⋃ k i =1 Q 1 ( v i ) , the corresponding ( k, t ) -FWL+ instance is more powerful than δ -k-LWL [8].

Proposition 3.7. Let t = 2 , k = 2 , and ES 2 ( v ) = ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) × ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) , the corresponding ( k, t ) -FWL+ instance is more powerful than GraphSNN [21].

Proposition 3.8. Let t = 2 , k = 2 , and ES 2 ( v ) = Q 1 ( v 2 ) × Q 1 ( v 1 ) , the corresponding ( k, t ) -FWL+ instance is more powerful than edge-based subgraph GNNs like I 2 -GNN [19].

Proposition 3.9. Let t = 1 , k = 2 , and ES ( v ) = Q SPD ( v 1 ,v 2 ) ( v 1 ) ∩ Q 1 ( v 2 ) , the corresponding ( k, t ) -FWL+ instance is at most as powerful as KP-GNN [22] with the peripheral subgraph encoder as powerful as 1-WL.

Proposition 3.10. Let t = 2 , k = 2 , and ES 2 ( v ) = Q 1 ( v 2 ) × SP ( v 1 , v 2 ) , the corresponding ( k, t ) -FWL+ is more powerful than GDGNN [23].

We leave all formal proofs and discussion on complexity in Appendix C. Significantly, Proposition 3.8 indicates that ( k, t ) -FWL+ can be more powerful than edge-based subgraph GNNs but only require O ( n 2 ) space, strictly lower than O ( nm ) space used in edge-based subgraph GNNs, where m is the number of edges in the graph. However, most existing works employ MPNNs as their basic encoder, which differs from the tuple aggregation in ( k, t ) -FWL+. Thus it is still non-trivial to find an instance that can exactly fit many existing works. Nevertheless, given the strict hierarchy between node-based subgraph GNNs and SLFWL(2) [12], we conjecture there also exists a strict hierarchy between the ( k, t ) -FWL+ instances and corresponding existing works based on MPNNs and leave the detailed proof in our future works.

4 Neighborhood 2 -GNN: a practical and powerful (2 , 2) -FWL+ implementation

Due to the extremely flexible and broad design space of ( k, t ) -FWL+, it is hard to evaluate all possible instances. Therefore, we focus on a particular instance that is both theoretically and practically sound. All detailed proof and complexity analysis related to this section can be found in Appendix D.

4.1 Neighborhood 2 -FWL

In this section, we propose a practical and powerful implementation of (2 , 2) -FWL+ named Neighborhood 2 -FWL (N 2 -FWL). Let C l n 2 fwl ( v ) denote the color of tuple v at iteration l for N 2 -FWL, the color is updated by:

where N 2 ( v ) = ( N 1 ( v 2 ) ×N 1 ( v 1 )) ∩ ( N h ( v 1 ) ∩ N h ( v 2 )) 2 and h is the number of hop. Briefly speaking, N 2 -FWL considers neighbors of both node v 1 and v 2 within the overlapping h -hop subgraph between v 1 and v 2 . In the following, we show the expressive power and counting power of N 2 -FWL.

Theorem 4.1. Given h is large enough, N 2 -FWL is more powerful than SLFWL(2) [12] and edgebased subgraph GNNs.

Corollary 4.2. N 2 -FWL is strictly more powerful than all existing node-based subgraph GNNs and no less powerful than 3-WL.

Theorem 4.3. N 2 -FWL can count up to (1) 6-cycles; (2) all connected graphlets with size 4; (3) 4-paths at node level.

4.2 Neighborhood 2 -GNN

In this section, we provide a neural version of N 2 -FWL called Neighborhood 2 -GNN (N 2 -GNN). Let h l v be the output of N 2 -GNN of tuple v at layer l and h 0 v encode the isomorphism type of tuple v . At the l -th layer, the representation of each tuple is updated by:

After L layers of message passing, the final graph representation is obtained by:

We leave the detailed implementation in Appendix D. Finally, we show the expressive power of N 2 -GNN as follows:

Proposition 4.4. If U l , M l , and R are injective functions, there exist a N 2 -GNN instance that can be as powerful as N 2 -FWL.

5 Related works

After the expressive power of MPNNs has been demonstrated to be limited by the 1-WL test [5, 6], significant research efforts have focused on breaking the limitations of MPNNs.

Subgraph-based methods . Subgraph-based methods aim to address the limitation of MPNNs by leveraging the fact that although some graphs may be difficult to distinguish, they contain distinct substructures that can be easily differentiated by MPNNs. To this end, various subgraph selection policies and aggregation strategies have been proposed to enhance the expressive power of MPNNs. These include K-ego subgraphs of nodes [24-28], distance labeling [29], graph elements deletion [30, 31], edge-based subgraph selection [19, 32, 33], etc. ESAN [34] summarizes different subgraph selection policies and proposes a general framework for subgraph-based GNNs. l -OSAN [17] further extends the subgraph-based model to l -tuples and k, l -WL [16] considers running k -WL on l -tuples subgraphs. KP-GNN [22] takes a slightly different view by considering the peripheral subgraph at each hop of nodes. Recently, Frasca et al. [27] proved that all node-based subgraph GNNs are bounded by 3-WL test. Zhang et al. [12] further show that they are even strictly less powerful than localized 2-FWL, highlighting the inherent limitations of neighbor aggregation-based methods compared to tuple-based aggregation.

WL-based methods . WL-based methods try to simulate k -WL/FWL test. In particular, Morris et al. [6] proposed k -GNNs to approximate k -WL test. To reduce the complexity of k -GNNs, Morris et al. [8] considered only local neighbors. Similarly, PPGN [9] adopted k -FWL instead of k -WL and proposed a tensor-based model with 3-WL power but requiring only O ( n 2 ) space. To further reduce complexity, KC-SetGNN [10] leveraged node sets rather than tuples and considered only

connected node sets. However, these approaches can still be impractical despite careful design. Our work hugely extends the design space of FWL-based methods and demonstrates that the resulting model can be expressive even with a lower space complexity cost.

Permutation equivariant and invariant networks . This line of research aims to leverage the permutation equivariant and invariant properties of graphs and devise corresponding architectures. For example, Maron et al. [35] proposed k -IGNs, which employ permutational equivariant linear layers with point-wise activation functions to the adjacency matrix. k -IGNs have been proven to have the same expressive power as k -WL [9, 36, 37]. Another approach utilizes the Reynold operator to consider all possible permutations on either global [15] or local [20] levels. However, they consider all possible permutations in order to achieve universality, which may lead to overfitting in real-world tasks and are more valuable from a theoretical perspective. In contrast, our framework is highly flexible and can be tailored to fit the complexity of real-world tasks.

Feature-based methods . Feature-based methods aim to incorporate graph-related features that cannot be computed by MPNNs. For instance, some methods add random features to nodes to break the symmetry [38, 39]. GSN [40] introduces substructure counting features into MPNNs. GD-WL [41] applies general distance features to enhance the expressiveness of MPNNs and shows it can solve the graph biconnectivity problem. GDGNN [23] integrates the geodesic information between node pairs and achieves a great balance between performance and efficiency. ESC-GNN [42] employs subgraph features to simulate subgraph GNNs without running them explicitly. Puny et al. [43] computes graph polynomial features and injects them into PPGN to achieve greater expressive power than 3-WL within O ( n 2 ) space. Although feature-based methods are highly efficient, they often suffer from overfitting in real-world tasks and produce suboptimal results.

6 Experiments

In this section, we conduct various experiments on synthetic and real-world tasks to verify the effectiveness of N 2 -GNN. The details of all experiments can be found in Appendix E. Additional experimental results and ablation studies are included in Appendix F. The source code is provided in https://github.com/JiaruiFeng/N2GNN .

6.1 Expressive power

Datasets . To verify the expressive power of N 2 -GNN, we select four synthetic datasets, all containing graphs that cannot be distinguished by 1-WL/MPNNs. The datasets we selected are (1) EXP [38], which consists of 600 pairs of non-isomorphic graphs that 1-WL fails to distinguish; (2) CSL [15], which contains 150 4-regular graphs divided into 10 isomorphism classes; and (3) SR25 [44], which contains 15 non-isomorphic strongly regular graphs, each has 25 nodes. Even 3-WL is unable to distinguish between these graphs. (4) BREC [45], which contains 400 non-isomorphic graph pairs that range from 1-WL-indistinguishable to 4-WL-indistinguishable.

Models . For EXP, CSL, and SR25, we only report results for N 2 -GNN. For BREC, we compare N 2 -GNN with I 2 -GNN [19] as it is the current SOTA on BREC among all GNN methods.

Results . We report results in Table 1 and Table 2. For EXP and CSL, we report the average accuracy for 10-time cross-validation; For SR25, we report single-time accuracy. We can see N 2 -GNN achieves perfect results on all three datasets, empirically verifying its expressive power. Notably, N 2 -GNN is able to distinguish strongly regular graphs. This empirically verified Corollary 4.2. Moreover, N 2 -GNN achieves the best result on the BREC dataset among all GNNs methods, surpassing the previous SOTA, I 2 -GNN (See the complete comparison and further discussion in Appendix F). Despite this improved performance, N 2 -GNN only requires at most O ( n 2 ) space complexity, which is less than 3-WL. This further demonstrates the superiority of N 2 -GNN.

6.2 Substructure counting

Datasets . To verify the substructure counting power of N 2 -GNN, we select the synthetic dataset from [25, 19]. The dataset consists of 5000 randomly generated graphs from different distributions, which are split into training, validation, and test sets with a ratio of 0.3/0.2/0.5. The task is to perform node-level counting regression. Specifically, we choose tailed-triangle, chordal cycles, 4-cliques, 4-paths, triangle-rectangle, 3-cycles, 4-cycles, 5-cycles, and 6-cycles as target substructures.

Models . We compare N 2 -GNN with the following baselines: Identity-aware GNN (ID-GNN) [28], Nested GNN (NGNN) [24], GNN-AK+ [25], PPGN [9], I 2 -GNN [19]. Results for all baselines are reported from [19].

Results . We report the results for counting substructures in Table 3. All results are the average normalized test MAE of three runs with different random seeds. We colored all results that are less than 0.01, which is an indication of successful counting, the same as in [19]. We can see that N 2 -GNN achieves MAE less than 0.01 among all different substructures/cycles, which empirically verified Theorem 4.3. Specifically, N 2 -GNN achieves the best result on 4-path counting and comparable performance to I 2 -GNN on most substructures. Moreover, N 2 -GNN can count 4-clique extremely well, which is theoretically and empirically infeasible for 3-WL [42] and PPGN [9].

6.3 Molecular properties prediction

Datasets . To evaluate the performance of N 2 -GNN on real-world tasks, we select two popular molecular graphs datasets: QM9 [46, 50] and ZINC [51]. QM9 dataset contains over 130K molecules with 12 different molecular properties as the target regression task. The dataset is split into training, validation, and test sets with a ratio of 0.8/0.1/0.1. The ZINC dataset has two variants: ZINC-subset (12k graphs) and ZINC-full (250k graphs), and the task is graph regression. The training, validation, and test splits for the ZINC datasets are provided.

Models . For QM9, we report baseline results of DTNN and MPNN from [46]. We further adopt PPGN [9], NGNNs [24], KP-GIN ′ [22], I 2 -GNN [19]. We report results of PPGN, NGNN, and KP-GIN ′ from [22] and results of I 2 -GNN from [19]. For ZINC, the baseline including CIN [47], δ -2-GNN [8], KC-SetGNN [10], PPGN [9], Graphormer-GD [41], GPS [48], Specformer [49], NGNN [24], GNN-AK-ctx [25], ESAN [34], SUN [27], KP-GIN ′ [22], I 2 -GNN [19], SSWL+ [12]. We report results of all baselines from [12, 22, 43, 10, 48, 49, 19].

Results . For QM9, we report the singletime final test MAE in Table 4. We can see that N 2 -GNN outperforms all other methods in almost all targets with a large margin. Particularly, we achieve 63 . 8% , 54 . 7% , 71 . 6% , and 63 . 2% improvement comparing to the second-best results on target U 0 , U , H , and G respectively. For ZINC-Subset and ZINCFull, we report the average test MAE and standard deviation of 10 runs with different random seeds in Table 5. We can see N 2 -GNN still surpasses all previous methods with only the smallest model parameter size among all baselines. It achieves 10 . 6% improvement over the second-best methods on ZINC-Subset and the same performance over the previous SOTA on ZINC-full. All these results demonstrate the superiority of N 2 -GNN on real-world tasks.

7 Limitations, future works, and conclusions

Limitations: For ( k, t ) -FWL, although we can control the space complexity by fixing the k , the time complexity of ( k, t ) -FWL will grow exponentially with the increase of t (as discussed in Appendix B). Therefore, the resulting model can still be impractical when t is large. We believe this result is evidence of the "no free lunch" in developing a more expressive GNN model. For ( k, t ) -FWL+, although we demonstrate its capability by implementing many instances corresponding to different existing GNN models, it is still unclear what is the whole space of ( k, t ) -FWL+, especially the space of ES ( v ) . Moreover, the quantitative expressiveness analysis of ( k, t ) -FWL+ is still unexplored. Finally, for N 2 -GNN, the practical complexity can still be unbearable, especially for dense graphs if the aggregation is implemented in a parallel way, as we need to aggregate many more neighbors for each tuple than normal MPNN. It also introduces the optimization issue on N 2 -GNN, which makes it hard to achieve its theoretical expressiveness. In Appendix B, C, and D, we provide more detailed discussion on the limitation of ( k, t ) -FWL, ( k, t ) -FWL+, and N 2 -GNN, respectively.

Future works : There are several directions that are worth further exploration. First, can we characterize the whole space of ES ( v ) and thus can theoretically and quantitatively analyze the expressive power of ( k, t ) -FWL+. Further, how to design advanced model architecture to achieve the theoretical power of ( k, t ) -FWL+. Finally, can we provide a systematic implementation code package and practical usage of different downstream tasks such that researchers can easily develop an instance of ( k, t ) -FWL+ that can best fit their downstream tasks? We leave these to our future work.

Conclusions: In this work, we propose ( k, t ) -FWL+, a flexible and powerful extension of k -FWL. First, ( k, t ) -FWL+ expands the tuple aggregation style in k -FWL. We theoretically prove that given any fixed pace complexity of O ( n k ) , ( k, t ) -FWL+ can still construct an expressiveness hierarchy up to solving the graph isomorphism problem. Second, ( k, t ) -FWL+ extends the global neighborhood definition in k -FWL to any equivariant set, which enables a finer-grained design space. We show that the ( k, t ) -FWL+ framework can implement many existing powerful GNN models with matching expressiveness. We further implement an instance named N 2 -GNN which is both practically powerful and theoretically expressive. Theoretically, it partially outperforms 3-WL but only requires O ( n 2 ) space. Empirically, it achieves new SOTA on BREC, ZINC-Subset, and ZINC-Full datasets. We envision ( k, t ) -FWL+ can serve as a promising general framework for designing highly effective GNNs tailored to real-world problems.

8 Acknowledgments

Jiarui Feng, Lecheng Kong, Hao Liu, and Yixin Chen are supported by NSF grant CBE-2225809. Muhan Zhang is partially supported by the National Natural Science Foundation of China (62276003) and Alibaba Innovative Research Program.

References

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017. URL https: //openreview.net/forum?id=SJU4ayYgl .

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems , pages 1025-1035, 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=rJXMpikCZ .

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2018.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , pages 4602-4609, 2019.

Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. NTI, Series , 2(9):12-16, 1968.

Christopher Morris, Gaurav Rattan, and Petra Mutzel. Weisfeiler and leman go sparse: Towards scalable higher-order graph embeddings. Advances in neural information processing systems , 33:21824-21840, 2020.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. Advances in neural information processing systems , 32, 2019.

Lingxiao Zhao, Neil Shah, and Leman Akoglu. A practical, progressively-expressive GNN. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id= WBv9Z6qpA8x .

Christopher Morris, Floris Geerts, Jan Tönshoff, and Martin Grohe. WL meet VC. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 25275-25302. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/morris23a.html .

Bohang Zhang, Guhao Feng, Yiheng Du, Di He, and Liwei Wang. A complete expressiveness hierarchy for subgraph GNNs via subgraph weisfeiler-lehman tests. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 41019-41077. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/zhang23k.html .

Jin-Yi Cai, Martin Fürer, and Neil Immerman. An optimal lower bound on the number of variables for graph identification. Combinatorica , 12(4):389-410, 1992.

Martin Grohe. Descriptive Complexity, Canonisation, and Definable Graph Structure Theory . Lecture Notes in Logic. Cambridge University Press, 2017. doi: 10.1017/9781139028868.

Ryan Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Relational pooling for graph representations. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 4663-4673, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http://proceedings.mlr.press/v97/murphy19a.html .

Cai Zhou, Xiyuan Wang, and Muhan Zhang. From relational pooling to subgraph GNNs: A universal framework for more expressive graph neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 42742-42768. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/zhou23n.html .

Chendi Qian, Gaurav Rattan, Floris Geerts, Mathias Niepert, and Christopher Morris. Ordered subgraph aggregation networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=w0QoqmUT9vJ .

V. Arvind, Frank Fuhlbrück, Johannes Köbler, and Oleg Verbitsky. On Weisfeiler-Leman invariance: Subgraph counts and related graph properties. Journal of Computer and System Sciences , 113:42-59, November 2020. ISSN 0022-0000. doi: 10.1016/j.jcss.2020.04.003. URL https://www.sciencedirect.com/science/article/pii/S0022000020300386 .

Yinan Huang, Xingang Peng, Jianzhu Ma, and Muhan Zhang. Boosting the cycle counting power of graph neural networks with i$^2$-GNNs. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=kDSmxOspsXQ .

Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count substructures? Advances in neural information processing systems , 33:10383-10395, 2020.

Asiri Wijesinghe and Qing Wang. A new perspective on 'how graph neural networks go beyond weisfeiler-lehman?'. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=uxgg9o7bI_3 .

Jiarui Feng, Yixin Chen, Fuhai Li, Anindya Sarkar, and Muhan Zhang. How powerful are k-hop message passing graph neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=nN3aVRQsxGd .

Lecheng Kong, Yixin Chen, and Muhan Zhang. Geodesic graph neural network for efficient graph representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=6pC5OtP7eBx .

Muhan Zhang and Pan Li. Nested graph neural networks. Advances in Neural Information Processing Systems , 34:15734-15747, 2021.

Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any GNN with local structure awareness. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=Mspk_WYKoEH .

Dylan Sandfelder, Priyesh Vijayan, and William L. Hamilton. Ego-gnns: Exploiting ego structures in graph neural networks. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 8523-8527, 2021. doi: 10.1109/ ICASSP39728.2021.9414015.

Fabrizio Frasca, Beatrice Bevilacqua, Michael M. Bronstein, and Haggai Maron. Understanding and extending subgraph GNNs by rethinking their symmetries. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=sc7bBHAmcN .

Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 10737-10745, 2021.

Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. Advances in Neural Information Processing Systems , 33, 2020.

Pál András Papp, Karolis Martinkus, Lukas Faber, and Roger Wattenhofer. DropGNN: Random dropouts increase the expressiveness of graph neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=fpQojkIV5q8 .

Leonardo Cotta, Christopher Morris, and Bruno Ribeiro. Reconstruction for powerful graph representations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/ forum?id=ZKbZ4mebI9l .

Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=Hcr9mgBG6ds .

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems , pages 5165-5175, 2018.

Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M. Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=dFbKQaRk15w .

Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference on Learning Representations , 2019. URL https: //openreview.net/forum?id=Syx72jC9tm .

Waiss Azizian and marc lelarge. Expressive power of invariant and equivariant graph neural networks. In International Conference on Learning Representations , 2021. URL https: //openreview.net/forum?id=lxHgXYN4bwl .

Floris Geerts. The expressive power of kth-order invariant graph networks. ArXiv , abs/2007.12035, 2020.

Ralph Abboud, ˙ Ismail ˙ Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. In Proceedings of the Thirtieth International Joint Conference on Artifical Intelligence (IJCAI) , 2021.

Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In Proceedings of the 2021 SIAM International Conference on Data Mining, SDM , 2021.

Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael M. Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting, 2021. URL https: //openreview.net/forum?id=LT0KSFnQDWF .

Bohang Zhang, Shengjie Luo, Liwei Wang, and Di He. Rethinking the expressive power of GNNs via graph biconnectivity. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=r9hNv76KoT3 .

Zuoyu Yan, Junru Zhou, Liangcai Gao, Zhi Tang, and Muhan Zhang. Efficiently counting substructures by subgraph gnns without running gnn on subgraphs, 2023.

Omri Puny, Derek Lim, Bobak T. Kiani, Haggai Maron, and Yaron Lipman. Equivariant polynomials for graph neural networks, 2023.

Muhammet Balcilar, Pierre Héroux, Benoit Gaüzère, Pascal Vasseur, Sébastien Adam, and Paul Honeine. Breaking the limits of message passing graph neural networks. In Proceedings of the 38th International Conference on Machine Learning (ICML) , 2021.

Yanbo Wang and Muhan Zhang. Towards better evaluation of gnn expressiveness with brec dataset, 2023.

Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science , 9(2):513-530, 2018.

Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yu Guang Wang, Pietro Liò, Guido Montufar, and Michael M. Bronstein. Weisfeiler and lehman go cellular: CW networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=uVPZCMVtsSG .

Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a General, Powerful, Scalable Graph Transformer. Advances in Neural Information Processing Systems , 35, 2022.

Deyu Bo, Chuan Shi, Lele Wang, and Renjie Liao. Specformer: Spectral graph neural networks meet transformers. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=0pdSt3oyJa1 .

Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data , 1(1):1-7, 2014.

Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982 , 2020.

Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Appendix

Table of Contents

A Additional preliminaries

In this section, we provide additional preliminaries.

A.1 Additional definitions

Definition A.1. When we say we select a t -tuple w ′ = ( w ′ 1 , w ′ 2 , · · · , w ′ t ) from a p -tuple v = ( v 1 , v 2 , · · · , v p ) ( p ≥ t ), it means that we select t unique indices ( i 1 , i 2 , · · · , i t ) from [ p ] such that i 1 < i 2 < · · · < i t . Next, to construct w ′ , we let w ′ 1 = v i 1 , . . . , w ′ t = v i t . When we say we select all possible t -tuple w ′ from a p -tuple v , we enumerate all possible t indices and construct t -tuples correspondingly. There are ( p t ) different selection results and we denote ˜ P t ( v ) as the set of all

possible selections. Finally, when we say select all possible t -tuple w ′ from a p -tuple v with order , we further assume that the selection process is taken with a predefined order such that the sequence of selected t indices will be the same if we repeat the selection. For example, we can always assume we start from indices (1 , 2 , . . . , t ) . Next, fix other indices and enumerate all possible i t in ascending order. Then, increase i t -1 by 1 and repeat the procedure. If we select all possible 2-tuple from ( v 1 , v 2 , v 3 ) , the result is (( v 1 , v 2 ) , ( v 1 , v 3 ) , ( v 2 , v 3 )) . We denote P t ( v ) as the tuple of the selection result.

Definition A.2. Neighborhood tuple Q F w ( v ) is a generalization of neighbor set Q F w ( v ) in k -FWL. Briefly speaking, the neighborhood tuple enumerates all possible combinations of elements in v and w follows a pre-defined order. Suppose the length of the tuple is k and t for v and w , respectively. To construct the neighborhood tuple Q F w ( v ) , we enumerate m from 0 to min ( k, t ) . For each m , we construct a sub-tuple, and the final neighborhood tuple Q F w ( v ) is the concatenation of all sub-tuples. Given a particular m , the sub-tuple is constructed by selecting all possible m -tuple w ′ = ( w ′ 1 , · · · , w ′ m ) from t -tuple w with order to construct a tuple P m ( w ) . At the same time, we select all possible m indices i ′ = ( i ′ 1 , . . . , i ′ m ) from i = (1 , 2 , . . . , k ) with order to construct another tuple P m ( i ) . Finally, for each ( w ′ , i ′ ) ∈ P m ( w ) × P m ( i ) , we add a tuple to the resulted sub-tuple by replacing v i ′ o with w ′ o for o = 1 , 2 , . . . , m . The neighborhood tuple is constructed by concatenating

Definition A.3. Let F 1 and F 2 be two color refinement algorithms, and denote C i ( G ) , i ∈ { 1 , 2 } as the color histogram computed by F i for graph G after convergence. We say:

F 1 is more powerful than F 2 , denoted as F 2 ⪯ F 1 , if for any pair of graph G and H , C 1 ( G ) = C 1 ( H ) also implies C 2 ( G ) = C 2 ( H ) .

F 1 is as powerful as F 2 , denoted as F 1 ≃ F 2 , if we have F 1 ⪯ F 2 and F 2 ⪯ F 1 .

̸

F 1 is strictly more powerful than F 2 , denoted as F 2 ≺ F 1 , if we have F 2 ⪯ F 1 and there exist graphs G and H such that C 1 ( G ) = C 1 ( H ) and C 2 ( G ) = C 2 ( H ) .

F 1 and F 2 are incomparable , denoted as F 1 ≁ F 2 , if neither F 1 ⪯ F 2 nor F 2 ⪯ F 1 holds.

Remark A.4 . To prove that F 1 is more powerful than F 2 , one way is to show that the color of node/tuple cannot be further refined by F 2 if the color output from F 1 is already stable (convergence). This was also indicated in Remark B.2 in [12]. Note that the initial color of F 1 should be at least as powerful as the initial color of F 2 . Namely, if we use the initial color to compute the color histogram, we have F 2 ⪯ F 1 .

A.2 More on Weisfeiler-Lehman test and Folklore Weisfeiler-Lehman test

First, we restate the color update equation of both k -WL and k -FWL.

The first step of both k -WL and k -FWL is to encoder the isomorphic type of tuple v = ( v 1 , v 2 , ..., v k ) as initial color such that for any two tuples u , v ∈ v k ( G ) , C 0 kfwl ( v ) = C 0 kfwl ( u ) and C 0 kwl ( v ) =

C 0 kwl ( u ) if and only if the ordered subgraphs induced by two tuples have the same isomorphism type. Namely, ∀ i, j ∈ [ k ] (1) l G ( v i ) = l G ( u i ) and (2) ( v i , v j ) ∈ E ( G ) ⇐⇒ ( u i , u j ) ∈ E ( G ) . Another thing is that the HSAH here is an injective function. As long as the input is different, the output of HASH is different. This will serve as a basic concept in the later proof. The algorithm converge means that ∀ v ∈ V k ( G ) , we have C l +1 kwl ( v ) = C l kwl ( v ) or C l +1 kfwl ( v ) = C l kfwl ( v ) . Denote the convergent iteration as ∞ . After the algorithm convergence, the color histogram can be computed by another injective function:

Here we use k -FWL as an example, but the procedure for other color refinement algorithms is the same.

B Additional discussion on ( k, t ) -FWL

B.1 Detailed proofs for ( k, t ) -FWL

In this section, we provide all detailed proofs related to ( k, t ) -FWL. Let p = ( v , w ) be a new tuple that is the concatenation of a k -tuple v and a t -tuple w . Let C 0 ( k + t ) wl ( p ) be the isomorphic type of tuple p . We start to prove the expressive power of ( k, t ) -FWL with the first lemma:

Lemma B.1. Suppose k ≥ 2 and t ≥ 1 . Given two graphs G , H , two tuples v 1 ∈ V k ( G ) , v 2 ∈ V k ( H ) , and another two tuples w 1 ∈ V t ( G ) , w 2 ∈ V t ( H ) . Let p 1 = ( v 1 , w 1 ) , p 2 = ( v 2 , w 2 ) . We have C 0 ( k + t ) wl ( p 1 ) = C 0 ( k + t ) wl ( p 2 ) ⇐⇒ ( C 0 ktfwl ( u 1 ) | u 1 ∈ Q F w 1 ( v 1 ) ) = ( C 0 ktfwl ( u 2 ) | u 2 ∈ Q F w 2 ( v 2 ) ) .

Proof. First, we prove the forward direction. That is, if C 0 ( k + t ) wl ( p 1 ) = C 0 ( k + t ) wl ( p 2 ) , we have ( C 0 ktfwl ( u 1 ) | u 1 ∈ Q F w 1 ( v 1 ) ) = ( C 0 ktfwl ( u 2 ) | u 2 ∈ Q F w 2 ( v 2 ) ) . Recall the definition of the initial color, if C 0 ( k + t ) wl ( p 1 ) = C 0 ( k + t ) wl ( p 2 ) , we must have the ordered subgraph induced by p 1 is isomorphic to the ordered subgraph induced by p 2 . Namely, ∀ i, j ∈ [ k + t ] , ( p 1 i , p 1 j ) ∈ E ( G ) ⇐⇒ ( p 2 i , p 2 j ) ∈ E ( H ) and l G ( p 1 i ) = l G ( p 2 i ) . Next, given Definition A.2, we know that for any neighborhood tuple, there exists a single indices mapping function O that given p = ( v , w ) , for any u ∈ Q F w ( v ) , we have O ( u , Q F w ( v )) = ( o 1 , . . . , o k ) such that ( u 1 , . . . , u k ) = ( p o 1 , . . . , p o k ) . Further, the construction of the neighborhood tuple follows the same ordering rule no matter the input. This means that, for both u 1 ∈ Q F w 1 ( v 1 ) and u 2 ∈ Q F w 2 ( v 2 ) , as long as u 1 and u 2 are in the same position of neighborhood tuple Q F w 1 ( v 1 ) and Q F w 2 ( v 2 ) , we must have O ( u 1 , Q F w 1 ( v 1 )) = O ( u 2 , Q F w 2 ( v 2 )) = ( o 1 , . . . , o k ) . Finally, as for any o i and o j , we have ( p 1 o i , p 1 o j ) ∈ E ( G ) ⇐⇒ ( p 2 o i , p 2 o j ) ∈ E ( H ) , l G ( p 1 o i ) = l G ( p 2 o i ) , and l G ( p 1 o j ) = l G ( p 2 o j ) , we conclude that C 0 ktfwl ( u 1 ) = C 0 ktfwl ( u 2 ) . As it is the same for any u 1 ∈ Q F w 1 ( v 1 ) and u 2 ∈ Q F w 2 ( v 2 ) , it must holds that ( C 0 ktfwl ( u 1 ) | u 1 ∈ Q F w 1 ( v 1 ) ) = ( C 0 ktfwl ( u 2 ) | u 2 ∈ Q F w 2 ( v 2 ) ) . The proof of the backward direction is similar to the forward direction. As for any i, j ∈ [ k + t ] , we can find p 1 i , p 1 j exist in some u 1 ∈ Q F w 1 ( v 1 ) and p 2 i , p 2 j in the corresponding u 2 ∈ Q F w 2 ( v 2 ) . Since we have C 0 ktfwl ( u 1 ) = C 0 ktfwl ( u 2 ) , we can easily conclude that ( p 1 i , p 1 j ) ∈ E ( G ) ⇐⇒ ( p 2 i , p 2 j ) ∈ E ( G ) , l G ( p 1 i ) = l G ( p 2 i ) , and l G ( p 1 j ) = l G ( p 2 j ) . This concludes the proof.

Lemma B.1 indicates that ( k, t ) -FWL can recover the isomorphism type of high-order tuple, which is impossible in the original k -FWL. Therefore, as long as t is large enough, we can recover the isomorphism type of the whole graph. Further, Lemma B.1 also implies that we can resort to highorder tuples to prove the expressive power of ( k, t ) -FWL. Define the H t ( v ) = { ( v , w ) | w ∈ V t ( G ) } , we slightly rewrite Equation (6):

It is easy to verify the rewritten will not affect the expressive power of ( k, t ) -FWL. Now, we are ready to prove the Proposition 3.1 in the main paper. We restate it here:

Proposition B.2. For k ≥ 2 and t ≥ 1 , if t ≥ n -k , ( k, t ) -FWL can solve the graph isomorphism problems with the size of the graph less than or equal to n .

Proof. Here we show that with only 1 iteration, ( k, t ) -FWL can solve the graph isomorphism problem with the size of graph n ≤ t + k . Given Lemma B.1, for any v ∈ V k ( G ) , we have C 1 ktfwl ( p 1 ) = C 1 ktfwl ( p 2 ) ⇐⇒ C 0 ( k + t ) wl ( p 1 ) = C 0 ( k + t ) wl ( p 2 ) for any p 1 , p 2 ∈ H t ( v ) . Therefore { {C 1 ktfwl ( p ) | p ∈ H t ( v ) } } is equivalent to { {C 0 ( k + t ) wl ( p ) | p ∈ H t ( v ) } } , which is essentially the multiset of isomorphism type of p ∈ H t ( v ) . According to Equation (14), C 1 ktfwl ( v ) can injectively encode this information. Next, the color histogram of a graph is computed by C ktfwl ( G ) = HASH ( { {C 1 ktfwl ( v ) | v ∈ V k ( G ) } } ) . It is easy to verify that C ktfwl ( G ) can encode the isomorphism type of all ( k + t ) -tuple p ∈ V k + t ( G ) . Then, if the graph has size n ≤ k + t , C ktfwl ( G ) can give different color histograms for any non-isomorphic graph pairs. This concludes the proof.

Before we prove the relationship between ( k, t ) -FWL and ( k + t ) -WL, we first show some properties of ( k + t ) -WL. Let C ∞ ( k + t ) wl ( p ) be the stable color of tuple p ∈ V k + t ( G ) output from ( k + t ) -WL. We first show that:

Lemma B.3. Suppose k ≥ 2 , t ≥ 1 . Given a graph G , for any v 1 , v 2 ∈ V k ( G ) , w 1 , w 2 ∈ V t ( G ) , p 1 = ( v 1 , w 1 ) , p 2 = ( v 2 , w 2 ) , C ∞ ( k + t ) wl ( p 1 ) = C ∞ ( k + t ) wl ( p 2 ) implies that { {C ∞ ( k + t ) wl ( q 1 ) | q 1 ∈ H t ( v 1 ) } } t = { {C ∞ ( k + t ) wl ( q 2 ) | q 2 ∈ H t ( v 2 ) } } t .

̸

Proof. Let p 1 = ( p 11 , . . . , p 1( k + t ) ) , p 2 = ( p 21 , . . . , p 2( k + t ) ) . As we have C ∞ ( k + t ) wl ( p 1 ) = C ∞ ( k + t ) wl ( p 2 ) , given it is the stable color, we must have { {C ∞ ( k + t ) wl (( p 11 , . . . , p 1 k , w, p 1( k +2) , . . . , p 1( k + t ) )) | w ∈ V ( G ) } } = { {C ∞ ( k + t ) wl (( p 21 , . . . , p 2 k , w, p 2( k +2) , . . . , p 2( k + t ) )) | w ∈ V ( G ) } } . Otherwise, by doing one more iteration, we will have C ∞ +1 ( k + t ) wl ( p 1 ) = C ∞ +1 ( k + t ) wl ( p 2 ) , which is a contradiction. Further, we can do another unrolling to have { {{ {C ∞ ( k + t ) wl (( p 11 , . . . , p 1 k , w 1 , w 2 , . . . , p 1( k + t ) )) | w 2 ∈ V ( G ) } }| w 1 ∈ V ( G ) } } = { {{ {C ∞ ( k + t ) wl (( p 21 , . . . , p 2 k , w 1 , w 2 , . . . , p 2( k + t ) )) | w 2 ∈ V ( G ) } }| w 1 ∈ V ( G ) } } . This implies that { {C ∞ ( k + t ) wl (( p 11 , . . . , p 1 k , w 1 , w 2 , . . . , p 1( k + t ) )) | ( w 1 , w 2 ) ∈ V 2 ( G ) } } 2 = { {C ∞ ( k + t ) wl (( p 21 , . . . , p 2 k , w 1 , w 2 , . . . , p 2( k + t ) )) | ( w 1 , w 2 ) ∈ V 2 ( G ) } } 2 Therefore, by iteratively doing the unrolling, we must have { {C ∞ ( k + t ) wl (( p 11 , . . . , p 1 k , w 1 , . . . , w t )) | ( w 1 , . . . , w t ) ∈ V t ( G ) } } t = { {C ∞ ( k + t ) wl (( p 21 , . . . , p 2 k , w 1 , . . . , w t )) | ( w 1 , . . . , w t ) ∈ V t ( G ) } } t , which is exactly { {C ∞ ( k + t ) wl ( q 1 ) | q 1 ∈ H t ( v 1 ) } } t = { {C ∞ ( k + t ) wl ( q 2 ) | q 2 ∈ H t ( v 2 ) } } t . This concludes the proof.

Lemma B.3 indicates that if the ( k, t ) -WL converges, the hierarchical set will not increase the expressive power of ( k + t ) -WL. Based on this observation, we can slightly rewrite the equation for computing the graph color histogram of ( k + t ) -WL:

where C ∞ ( k + t ) wl ( v ) is the stable color of k -tuple v . Here we change the computation of graph color histogram by first hierarchically pooling all tuples in H t ( v ) to get a color for v and then pooling all tuples v ∈ V k ( G ) . It is straightforward that the rewritten will not change the expressive power of ( k + t ) -WL given Lemma B.3. Equation (15) and Equation (16) will be used in the latter proofs. Not surprisingly, we have similar result for ( k, t ) -FWL:

Lemma B.4. Suppose k ≥ 2 , t ≥ 1 . Given a graph G , for any v 1 , v 2 ∈ V k ( G ) , w 1 , w 2 ∈ V t ( G ) , p 1 = ( v 1 , w 1 ) , p 2 = ( v 2 , w 2 ) , C ∞ ktfwl ( p 1 ) = C ∞ ktfwl ( p 2 ) implies that { {C ∞ ktfwl ( q 1 ) | q 1 ∈ H t ( v 1 ) } } t = { {C ∞ ktfwl ( q 2 ) | q 2 ∈ H t ( v 2 ) } } t .

Proof. Given the definition of the neighborhood tuple, we must have v 1 ∈ Q F w 1 ( v 1 ) and v 2 ∈ Q F w 1 ( v 2 ) . This means we also have C ∞ ktfwl ( v 1 ) = C ∞ ktfwl ( v 2 ) . Otherwise, by doing one more iteration of ( k, t ) -FWL, we will get different colors for p 1 and p 2 , which is a contradiction. C ∞ ktfwl ( v 1 ) = C ∞ ktfwl ( v 2 ) directly implies that { {C ∞ ktfwl ( q 1 ) | q 1 ∈ H t ( v 1 ) } } t = { {C ∞ ktfwl ( q 2 ) | q 2 ∈ H t ( v 2 ) } } t given Equation (14) and thus conclude the proof.

Next, we show another property of ( k + t ) -WL. Denote I k + t as a permutation group operates on the indices of a k + t -tuple. For σ ∈ I k + t , we denote it operates on a k + t -tuple p = ( p 1 , . . . , p k + t ) as σ · p = ( p σ (1) , . . . , p σ ( k + t ) ) . For example, for a σ ∈ I 3 with σ (1) = 2 , σ (2) = 1 , σ (3) = 3 , we have σ · ( p 1 , p 2 , p 3 ) = ( p 2 , p 1 , p 3 ) .

Lemma B.5. Given two graphs G and H , for any p 1 = ( p 11 , . . . , p 1( k + t ) ) ∈ V k + t ( G ) , p 2 = ( p 21 , . . . , p 2( k + t ) ) ∈ V k + t ( H ) and a permutation σ ∈ I k + t operates on the indices of tuples, C l ( k + t ) wl ( p 1 ) = C l ( k + t ) wl ( p 2 ) implies that C l ( k + t ) wl ( σ · p 1 ) = C l ( k + t ) wl ( σ · p 2 ) .

Proof. We prove it by induction on l . At iteration 0, C 0 ( k + t ) wl ( p 1 ) = C 0 ( k + t ) wl ( p 2 ) means the ordered subgraph induced by p 1 is isomorphic to the ordered subgraph induced by p 2 . As the permutation is operating on the indices, it will not change the nodes in the tuple but only change the order of the tuple. As we use the same permutation σ for both p 1 and p 2 , it is easy to verify C 0 ( k + t ) wl ( σ · p 1 ) = C 0 ( k + t ) wl ( σ · p 2 ) . Suppose it follows for 1 , . . . , l -1 . At iteration l , if C l ( k + t ) wl ( p 1 ) = C l ( k + t ) wl ( p 2 ) , we have:

Then, since the statement follows at iteration l -1 , we must have

The above equation directly implies that C l ( k + t ) wl ( σ · p 1 ) = C l ( k + t ) wl ( σ · p 2 ) . This concludes the proof.

With all conclusions at hand, we can start to prove the main proposition in this section.

Proposition B.6. Suppose k ≥ 2 , t ≥ 1 . Given any two graphs G and H . If C ∞ ( k + t ) wl ( G ) = C ∞ ( k + t ) wl ( H ) , then for any v 1 ∈ V k ( G ) , v 2 ∈ V k ( H ) , w 1 ∈ V t ( G ) , w 2 ∈ V t ( H ) , p 1 = ( v 1 , w 1 ) , p 2 = ( v 2 , w 2 ) with C ∞ ( k + t ) wl ( p 1 ) = C ∞ ( k + t ) wl ( p 2 ) , we also have C ∞ ktfwl ( p 1 ) = C ∞ ktfwl ( p 2 ) .

Proof. As indicated in Remark A.4, this can be proved by doing one iteration of ( k, t ) -FWL using the stable color computed by the k + t -WL as the initial color. If the color is not refined anymore, we can conclude it. However, since there is no stable color of low dimensional tuples in original ( k + t ) -WL, we resort to Equation (15) for help:

Using Lemma B.3, we must have:

Further, given the definition of the neighborhood tuple, any u 1 ∈ Q F w 1 ( v 1 ) is a sub-tuple of p 1 , which means that we can find a permutation σ ∈ I [ k + t ] , such that the first k position of σ · p 1 is exactly u 1 . As the order of the neighborhood tuple is predefined and the same for both Q F w 1 ( v 1 ) and Q F w 2 ( v 2 ) , the first k position of σ · p 2 will be u 2 , as long as u 1 and u 2 is at the same position in the neighborhood tuple. Therefore, for any u 1 ∈ Q F w 1 ( v 1 ) and u 2 ∈ Q F w 2 ( v 2 ) at the same position o in the neighborhood tuple, we can find a permutation σ o ∈ I [ k + t ] to achieve it. Then, given Lemma B.5, we have C ∞ ( k + t ) wl ( σ o · p 1 ) = C ∞ ( k + t ) wl ( σ o · p 2 ) , which indicates C ∞ ( k + t ) wl ( u 1 ) = C ∞ ( k + t ) wl ( u 2 ) by using Lemma B.3. This means:

Now, suppose we use the stable color generated from ( k + t ) -wl as the initial color to perform one more iteration of ( k, t ) -FWL. First, Equation (19)indicates that we have C ∞ +1 ktfwl ( p 1 ) = C ∞ +1 ktfwl ( p 2 ) by doing one iteration of Equation (13). Next, Equation (18) further indicates we have C ∞ ktfwl ( v 1 ) = C ∞ ktfwl ( v 2 ) by doing one iteration of Equation (14). This means the color of p 1 and p 2 cannot be further refined by ( k, t ) -FWL. This concludes the proof.

Now, we can prove Theorem 3.2 in the main paper. We restate it here:

Theorem B.7. For k ≥ 2 , t ≥ 1 , ( k, t ) -FWL is at most as powerful as ( k + t ) -WL. In particular, ( k, 1) -FWL is as powerful as ( k +1) -WL.

Proof. Based on Proposition B.6, given two graphs G , H with C ∞ ( k + t ) wl ( G ) = C ∞ ( k + t ) wl ( H ) . For any v 1 ∈ V k ( G ) , v 2 ∈ V k ( H ) , w 1 ∈ V t ( G ) , w 2 ∈ V t ( H ) , p 1 = ( v 1 , w 1 ) , p 2 = ( v 2 , w 2 ) , we have C ∞ ktfwl ( p 1 ) = C ∞ ktfwl ( p 2 ) if C ∞ ( k + t ) wl ( p 1 ) = C ∞ ( k + t ) wl ( p 2 ) . This further indicate that C ∞ ktfwl ( v 1 ) = C ∞ ktfwl ( v 2 ) if C ∞ ( k + t ) wl ( v 1 ) = C ∞ ( k + t ) wl ( v 2 ) . As we have C ∞ ( k + t ) wl ( G ) = C ∞ ( k + t ) wl ( H ) , for any v 1 ∈ V k ( G ) , we can find a corresponding v 2 ∈ V k ( H ) such that C ∞ ( k + t ) wl ( v 1 ) = C ∞ ( k + t ) wl ( v 2 ) . This means C ∞ ktfwl ( G ) = HASH ( { {C ∞ ktfwl ( v 1 ) | v 1 ∈ V k ( G ) } } ) = HASH ( { {C ∞ ktfwl ( v 2 ) | v 2 ∈ V k ( H ) } } ) = C ∞ ktfwl ( H ) . Based on Definition A.3, we complete the proof of the first part. As ( k, 1) -FWL is exactly k -FWL, we have ( k, 1) -FWL is as powerful as ( k +1) -WL.

Finally, we prove the strict hierarchy of ( k, t ) -FWL. First, we provide the following lemma.

Proof. First, we prove ( k, t ) -FWL ⪯ ( k, t +1) -FWL. Actually, we can prove a strong result: given two graphs G and H , v 1 ∈ V k ( G ) and v 2 ∈ V k ( H ) , C l k ( t +1) fwl ( v 1 ) = C l k ( t +1) fwl ( v 2 ) also implies C l ktfwl ( v 1 ) = C l ktfwl ( v 2 ) for any l ≥ 0 . We prove it by induction. For base case, it is easy to verify that if C 0 k ( t +1) fwl ( v 1 ) = C 0 k ( t +1) fwl ( v 2 ) , we have C 0 ktfwl ( v 1 ) = C 0 ktfwl ( v 2 ) as both ( k, t ) -FWL and ( k, t +1) -FWL use the isomorphism type of k -tuples. Suppose it follows for l = 1 , . . . , l -1 . At iteration l , if C l k ( t +1) fwl ( v 1 ) = C l k ( t +1) fwl ( v 2 ) , according to Equation (14), we have:

This means, for any p 1 = ( v 1 , w 1 ) ∈ H t +1 ( v 1 ) , we can find a p 2 = ( v 2 , w 2 ) ∈ H t +1 ( v 2 ) such that C l k ( t +1) fwl ( p 1 ) = C l k ( t +1) fwl ( p 2 ) . According to Equation (13), we further obtain:

Therefore, we have C l -1 k ( t +1) fwl ( u 1 ) = C l -1 k ( t +1) fwl ( u 2 ) for u 1 and u 2 at the same position. As the statement follows for l = l -1 , we have C l -1 ktfwl ( u 1 ) = C l -1 ktfwl ( u 2 ) for u 1 and u 2 at the same position. Now, suppose we select a t -tuple w ′ 1 from w 1 and let the selection indices be ( i 1 , i 2 , ..., i t ) . That is, w ′ 1 = ( w 1 i 1 , . . . , w 1 i t ) . Based on the definition A.2, for any u ′ 1 ∈ Q F w ′ 1 ( v 1 ) , it must exists in Q F w 1 ( v 1 ) . Further, let w ′ 2 = ( w 2 i 1 , . . . , w 2 i t ) , for any u ′ 1 ∈ Q F w ′ 1 ( v 1 ) , we can find a corresponding u ′ 2 ∈ Q F w ′ 2 ( v 2 ) at the same position and they must be the same position in Q F w 1 ( v 1 ) and Q F w 2 ( v 2 ) , respectively. This means, let p ′ 1 = ( v 1 , w ′ 1 ) and p ′ 2 = ( v 2 , w ′ 2 ) , we have C l ktfwl ( p ′ 1 ) = C l ktfwl ( p ′ 2 ) . Let selection indices be (1 , 2 , . . . , t ) , which is the first t elements in the tuple, for any p ′ 1 = ( v 1 , w ′ 1 ) ∈ H t ( v 1 ) , we can find at least one p 1 = ( v 1 , w 1 ) ∈ H t +1 ( v 1 ) that the first t elements of w 1 is w ′ 1 . For the corresponding p 2 = ( v 2 , w 2 ) ∈ H t +1 ( v 2 ) with C l k ( t +1) fwl ( p 1 ) = C l k ( t +1) fwl ( p 2 ) , we have C l ktfwl ( p ′ 1 ) = C l ktfwl ( p ′ 2 ) , where p ′ 2 = ( v 2 , w ′ 2 ) and w ′ 2 is the first t elements of w 2 . Based on Equation (21), we have:

Finally, given Equation (20), we also have C l -1 ktfwl ( v 1 ) = C l -1 ktfwl ( v 2 ) , which implies C l ktfwl ( v 1 ) = C l ktfwl ( v 2 ) . This concludes the proof.

For the second part, the procedure is similar, as for any v ∈ V k ( G ) , we can find at least one v ′ ∈ V k +1 ( G ) that contains all nodes and persevere the order of v . Further, given a w ∈ V t ( G ) , for any u ∈ Q F w ( v ) , we can find at least one u ′ ∈ Q F w ( v ′ ) that contains all nodes and persevere the order of u . Then, an induction on l can be performed and we omit the details.

Next, we show a useful property of ( k, t ) -FWL.

Lemma B.9. For k ≥ 2 , t ≥ 1 , given a graph G , for any v 1 , v 2 ∈ V k ( G ) , C ∞ ktfwl ( v 1 ) = C ∞ ktfwl ( v 2 ) only if ∀ i, j ∈ [ k ] , SPD ( v 1 i , v 1 j ) = SPD ( v 2 i , v 2 j ) .

̸

Proof. Based on Lemma B.8, we only need to show it is true for ( k, 1) -FWL, which is the original k -FWL. At the iteration 0, C 0 k 1 fwl ( v 1 ) = C 0 k 1 fwl ( v 2 ) if and only if the induced subgraph of v 1 and v 2 are isomorphic. Therefore, ∀ i, j ∈ [ k ] , if SPD ( v 1 i , v 1 j ) = 0 or SPD ( v 1 i , v 1 j ) = 1 , we must have SPD ( v 2 i , v 2 j ) = 0 or SPD ( v 2 i , v 2 j ) = 1 , respectively. After the first iteration, ∀ i, j ∈ [ k ] , if there exist at least one w 1 ∈ V ( G ) such that ( v 1 i , w 1 ) , ( w 1 , v 1 j ) ∈ E ( G ) but no such w 2 ∈ V ( G ) exist with ( v 2 i , w 2 ) , ( w 2 , v 2 j ) ∈ E ( G ) , ( k, 1) -FWL will output different color for v 1 and v 2 , given the color update equation of ( k, 1) -FWL. This means C 1 k 1 fwl ( v 1 ) = C 1 k 1 fwl ( v 2 ) implies that for ∀ i, j ∈ [ k ] , if SPD ( v 1 i , v 1 j ) = 2 , we must have SPD ( v 2 i , v 2 j ) = 2 . Similarly, after the second iteration, ( k, 1) -FWL will output different color for v 1 and v 2 if there exist at least one w 1 ∈ V ( G ) with SPD ( v 1 i , w 1 ) = 2 and SPD ( w 1 , v 1 j ) = 1 but no such w 2 ∈ V ( G ) exists for v 2 . Therefore, we can conclude that l iteration of ( k, 1) -FWL can generate different color for v 1 and v 2 if there exist some i, j ∈ [ k ] with SPD ( v 1 i , v 1 j ) = d but SPD ( v 2 i , v 2 j ) = d ( d ≤ l + 1 ). Therefore, by performing enough iterations, ( k, 1) -FWL can compute the shortest path distance between any pair of nodes in a k -tuple and it is easy to verify C ∞ ktfwl ( v 1 ) = C ∞ ktfwl ( v 2 ) only if ∀ i, j ∈ [ k ] , SPD ( v 1 i , v 1 j ) = SPD ( v 2 i , v 2 j ) . This concludes the proof.

Next, we introduce Cai-Furer-Immermman (CFI) graphs [13]. CFI graphs are a family of graphs that k +1 -WL can differentiate while k -WL cannot. Here we first state the construction of CFI graphs [8]. Construction. Let K k +1 denote a complete graph on k +1 nodes (without self-loop). Nodes in K k +1 are labeled from 0 to k . Let E ( v ) denote the set of edges incident to v in K k +1 . We have | E ( v ) | = k for all v ∈ V ( K k +1 ) . Then, we define the graph G k as follows:

For the node set V ( G k ) , we add

(a) ( v, S ) for each v in V ( K k +1 ) and for each even subset S ⊆ E ( v ) ;

(b) two nodes e 1 and e 0 for each edge e ∈ E ( K k +1 ) .

For the edge set E ( G k ) , we add

(a) an edge ( e 0 , e 1 ) for each e ∈ E ( K k +1 ) ;

(b) an edge between ( v, S ) and e 1 if v ∈ e and e ∈ S ;

(c) an edge between ( v, S ) and e 0 if v ∈ e and e / ∈ S .

Further, we construct a companion graph H k in a similar way to G k , except that: in Step 1(a), for the vertex 0 ∈ V ( K k +1 ) , we choose all odd subsets of E (0) . It is easy to verify that both G k and H k

We say a set S of nodes forms a distance-two-clique if the distance between any two vertices in S is exactly two.

Lemma B.10. The following holds for the graphs G k and H k defined above.

There exists a distance-two-clique of size ( k +1) in G k .

There does not exist a distance-two-clique of size ( k +1) inside H k .

Proof. Please see the Lemma 8 in [8] for detailed proof.

Next, we show that ( k, t ) -FWL cannot distinguish G k + t and H k + t but ( k +1 , t ) -FWL and ( k, t +1) -FWL can.

Lemma B.11. For k ≥ 2 , t ≥ 1 , ( k, t ) -FWL cannot distinguish G k + t and H k + t .

Proof. Given Theorem 3.2, ( k, t ) -FWL is at most as powerful as ( k + t ) -WL. As we know ( k + t ) -WL cannot distinguish G k + t and H k + t , we conclude that ( k, t ) -FWL cannot distinguish between them neither.

Lemma B.12. For k ≥ 2 , t ≥ 1 , ( k + 1 , t ) -FWL and ( k, t + 1) -FWL can distinguish G k + t and H k + t .

Proof. The proof idea is to show both ( k + 1 , t ) -FWL and ( k, t + 1) -FWL are powerful enough to detect distance-two-cliques of size ( k +1) , which is sufficient to show that ( k +1 , t ) -FWL and ( k, t +1) -FWL can distinguish between G k + t and H k + t . Given Lemma B.9, we know that both ( k +1 , t ) -FWL and ( k, t +1) -FWL can compute the distance between any two nodes in a k +1 -tuple and k -tuple, respectively. We first discuss ( k + 1 , t ) -FWL. In G k , there exist a k + t + 1 -tuple p = ( v , w ) with v ∈ V k +1 ( G k ) and w ∈ V t ( G k ) that exactly form a distance-two-clique. Based on Equation 13, there exists one p ∈ V k + t +1 ( G k ) such that C l ( k +1) tfwl ( p ) encodes the distance between any pair of nodes in p is 2. However, there is no such p in H k . This is sufficient to show that ( k +1 , t ) -FWL can distinguish G k and H k . The proof for ( k, t +1) -FWL is similar, as ( k, t +1) -FWL can also encode pairwise distance information for any k + t +1 -tuple, thus detecting distance-two-clique with the size of k + t +1 . This concludes the proof.

Now, we are ready to prove Proposition 3.3 in the main paper. We restate it here:

Proposition B.13. For k ≥ 2 , t ≥ 1 , ( k, t + 1) -FWL is strictly more powerful than ( k, t ) -FWL; ( k +1 , t ) -FWL is strictly more powerful than ( k, t ) -FWL.

Proof. It is clear the proposition holds given Lemma B.8, Lemma B.12, and Definition A.3.

B.2 Complexity analysis

For ( k, t ) -FWL, the space complexity is O ( n k ) as we only need to store representations of all k -tuples. For time complexity, if k and t are relatively small, to update the color of each k -tuple, ( k, t ) -FWL needs to aggregate all V t ( G ) possible neighborhood tuples, resulting in O ( n k + t ) time complexity. If both k and t are large enough (related to n ), the time complexity further increases to O ( n k + t · m · ( q ! m ! ) 2 ) , where m = min ( k, t ) and q = max ( k, t ) .

B.3 Limitations

As discussed in the above section, although ( k, t ) -FWL can have a fixed space complexity of O ( n k ) , the time complexity will grow exponentially if we increase the t . Further, the increase of the t will also make the length of the neighborhood tuple increase. This makes the practical usage of ( k, t ) -FWL limited (This will be hugely alleviated by ( k, t ) -FWL+). This is evidence of the "no free lunch" issue in developing an expressive GNN model. However, we do believe there are many repeat computations in the ( k, t ) -FWL that can be safely removed without the hurt of the expressive power. We leave it to our future work.

C Additional discussion on ( k, t ) -FWL+

C.1 Detailed proofs for ( k, t ) -FWL+

In this section, we provide all detailed proofs for ( k, t ) -FWL+. Specifically, we will use ( k, t ) -FWL+ to implement many existing powerful GNN/WL models with the best matching expressive power. To simplify notations and make it more clear, let ˆ C denote the color output by ( k, t ) -FWL+, ˜ C denote the

color output by any existing model we want to compare. Further, denote ˆ C l ( v 1 , v 2 ) as the color of tuple ( v 1 , v 2 ) at iteration l if k = 2 .

SLFWL(2) [12]: We prove Proposition 3.5 in the main paper. We restate it here:

Proposition C.1. Let t = 1 , k = 2 , and ES ( v ) = N 1 ( v 1 ) ∪N 1 ( v 2 ) , the corresponding ( k, t ) -FWL+ instance is equivalent to SLFWL(2) [12] and strictly more powerful than any existing node-based subgraph GNNs.

Proof. Given t = 1 , k = 2 , and ES ( v ) = N 1 ( v 1 ) ∪ N 1 ( v 2 ) , the color update equation of ( k, t ) -FWL+ can be written as:

We can see Equation (24) exactly matches SLFWL(2) stated in [12] and trivially this instance is as powerful as SLFWL(2). Further, given Theorem 7.1 in [12], we conclude that this instance is strictly more powerful than all existing node-based subgraph GNNs.

δ -k-LWL : The aggregation scheme of δ -k-LWL can be written as:

where Q j ( v ) = { v w/j | w ∈ Q 1 ( v j ) } . Next, we prove Proposition 3.6 in the main paper. We restate it here:

Proposition C.2. Let t = 1 , k = k , and ES ( v ) = ⋃ k i =1 Q 1 ( v i ) , the corresponding ( k, t ) -FWL+ instance is more powerful than δ -k-LWL [8].

Proof. Given t = 1 , k = k , and ES ( v ) = ⋃ k i =1 Q 1 ( v i ) , the color update equation of ( k, t ) -FWL+ can be written as:

̸

Given two k -tuple v 1 , v 2 ∈ V k ( G ) , it is easy to see that for any node w ∈ V ( G ) , ˆ C 0 ( v 1 ,w/j ) will be different to ˆ C 0 ( v 2 ,w/j ) as long as there exist any v 1 i and v 2 i such that v 1 i is a neighbor of node w but v 2 i is not for any i ∈ [ k ] , i = j . This means that for any u ∈ Q F w ( v ) , ˆ C 0 ( u ) can injectively encode whether w is a neighbor to all other nodes in tuples. Further, as ⋃ k i =1 Q 1 ( v i ) includes neighbors of all nodes in tuple v , we can injectively encode ( { { ˆ C l ( v w/j ) | w ∈ Q 1 ( v j ) } }| j ∈ [ k ]) for any l by { { ( ˆ C l -1 ( u ) | u ∈ Q F w ( v )) | w ∈ ⋃ k i =1 Q 1 ( v i ) } } . Now, given any two graphs G , H with ˆ C ∞ ( G ) = ˆ C ∞ ( H ) . For any v 1 ∈ V k ( G ) and v 2 ∈ V k ( H ) with ˆ C ∞ ( v 1 ) = ˆ C ∞ ( v 2 ) , we must have { {C ∞ ( v 1 ,w/j ) | w ∈ Q 1 ( v 1 j ) } } = { {C ∞ ( v 2 ,w/j ) | w ∈ Q 1 ( v 2 j ) } } for any j ∈ [ k ] given the above statement. Then, it is easy to verify that we cannot further refine the color of tuple v 1 and v 2 if we use the stable color of ( k, t ) -FWL+ as initial color to perform another iteration of δ -k-LWL, so as the final graph color histogram. Therefore, we conclude that ( k, t ) -FWL+ is more powerful than δ -k-LWL.

GraphSNN [21]: The GNN aggregation scheme of GraphSNN can be written as:

where | E v 1 v 2 | , | V v 1 v 2 | are the number of edges and nodes in the overlapping 1-hop subgraph between node v 1 and v 2 , λ is a constant. Next, we prove Proposition 3.7 in the main paper. We restate it here:

Proposition C.3. Let t = 2 , k = 2 , and ES 2 ( v ) = ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) × ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) , the corresponding ( k, t ) -FWL+ instance is more powerful than GraphSNN [21]..

Proof. Here we prove a slightly stronger one: even without a hierarchical multiset but only a plain multiset, the resulted ( k, t ) -FWL+ is already powerful enough to match GraphSNN. Given t = 2 , k = 2 , and ES 2 ( v ) = ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) × ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) , the color update equation of ( k, t ) -FWL+ can be written as:

First, we show that ˆ C 1 ( v 1 , v 2 ) is already powerful to recover all the information in Equation (28). For any ( w 1 , w 2 ) ∈ ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) × ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) , w 1 and w 2 must all be common neighbors of node v 1 and v 2 . Therefore ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) × ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) will contain all node pairs in the 1-hop overlapping subgraph of ( v 1 , v 2 ) . Thus we have the size of ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) × ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) equal to | V v 1 v 2 | 2 and this information will be definitely encoded in ˆ C 1 ( v 1 , v 2 ) by aggregating all possible tuple ( ˆ C 0 ( u 1 , u 2 ) | ( u 1 , u 2 ) ∈ Q F ( w 1 ,w 2 ) ( v 1 , v 2 ) ) . Further, we have ( w 1 , w 2 ) ∈ Q F ( w 1 ,w 2 ) ( v 1 , v 2 ) , which means ˆ C 1 ( v 1 , v 2 ) will also encode { { ˆ C 0 ( w 1 , w 2 ) | ( w 1 , w 2 ) ∈ ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) × ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) } } . As ˆ C 0 ( w 1 , w 2 ) encode whether there is an edge between w 1 and w 2 , it is easy to see { { ˆ C 0 ( w 1 , w 2 ) | ( w 1 , w 2 ) ∈ ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) × ( Q 1 ( v 1 ) ∩ Q 1 ( v 2 )) } } can encode | E v 1 v 2 | . Thus we can conclude that ˆ C 1 ( v 1 , v 2 ) can encode Equation (28). Namely, for any two tuples ( v 11 , v 12 ) , ( v 21 , v 22 ) ∈ V 2 ( G ) , we have: ˆ C l ( v 11 , v 12 ) = ˆ C l ( v 21 , v 22 ) ⇒ ˜ C l ( v 11 , v 12 ) = ˜ C l ( v 21 , v 22 ) , for any l ≥ 1 .

̸

Second, for any tuple ( v 1 , v 1 ) ∈ V 2 ( G ) , the neighbor in ( k, t ) -FWL+ is Q 2 1 ( v 1 ) . For any ( w 1 , w 2 ) ∈ Q 2 1 ( v 1 ) , it is easy to see that ˆ C 0 ( w 1 , w 2 ) will be different between tuples with w 1 = w 2 and w 1 = w 2 . This implies that ( k, t ) -FWL+ can injectively encode { { ˆ C l ( w,w ) | w ∈ Q 1 ( v 1 ) } } for any l as ( w 1 , w 2 ) ∈ Q F ( w 1 ,w 2 ) ( v 1 , v 1 ) . Namely, for any two nodes v 1 , v 2 ∈ V ( G ) , we have: ˆ C l ( v 1 , v 1 ) = ˆ C l ( v 2 , v 2 ) ⇒{ { ˆ C l -1 ( w 1 , w 1 ) | w 1 ∈ Q 1 ( v 1 ) } } = { { ˆ C l -1 ( w 2 , w 2 ) | w 2 ∈ Q 1 ( v 2 ) } } .

Third, since ( v 1 , w 1 ) ∈ Q F ( w 1 ,w 2 ) ( v 1 , v 1 ) , it is easy to verify that ( k, t ) -FWL+ can injectively encode { { ˆ C l ( v 1 , w 1 ) | w 1 ∈ Q 1 ( v 1 ) } } for any l . Namely, for any two nodes v 1 , v 2 ∈ V ( G ) , we have: ˆ C l ( v 1 , v 1 ) = ˆ C l ( v 2 , v 2 ) ⇒{ { ˆ C l -1 ( v 1 , w 1 ) | w 1 ∈ Q 1 ( v 1 ) } } = { { ˆ C l -1 ( v 2 , w 2 ) | w 2 ∈ Q 1 ( v 2 ) } } .

̸

Now, given any two graphs G , H with ˆ C ∞ ( G ) = ˆ C ∞ ( H ) . For any ( v 11 , v 12 ) ∈ V 2 ( G ) and ( v 21 , v 22 ) ∈ V 2 ( H ) with ˆ C ∞ ( v 11 , v 12 ) = ˆ C ∞ ( v 21 , v 22 ) , if v 11 = v 12 , we must have v 21 = v 22 and we have ˜ C ∞ ( v 11 , v 12 ) = ˜ C ∞ ( v 21 , v 22 ) if ˆ C ∞ ( v 11 , v 12 ) = ˆ C ∞ ( v 21 , v 22 ) given the first statement. If v 11 = v 12 and v 21 = v 22 , we further denote it by ( v 11 , v 11 ) and ( v 22 , v 22 ) . We have { { ˆ C ∞ ( w 11 , w 11 ) | w 11 ∈ Q 1 ( v 11 ) } } = { { ˆ C ∞ ( w 21 , w 21 ) | w 21 ∈ Q 1 ( v 21 ) } } given the second statement and { { ˆ C ∞ ( v 11 , w 11 ) | w 11 ∈ Q 1 ( v 11 ) } } = { { ˆ C ∞ ( v 21 , w 21 ) | w 21 ∈ Q 1 ( v 21 ) } } given the third statemtn. Now, given the stable color output by ( k, t ) -FWL+, we can verify that performing another iteration using Equation (27) and Equation (28) will not further refine the color of ( v 11 , v 11 ) and ( v 22 , v 22 ) , resulting in the same final graph color histogram. This concludes the proof.

̸

Edge-based subgraph GNNs The GNN aggregation scheme of edge-based subgraph GNNs like I 2 -GNN [19] can be written as:

̸

where we assume that the initial color of ˜ C 0 ( v 1 , v 2 , w 2 ) is l G ( v 2 ) if v 2 = [ v 1 , w 2 ] or HASH ( l G ( v 2 ) , 1) if v 2 = v 1 or HASH ( l G ( v 2 ) , 2) if v 2 = w 2 . Equation (30) can be described as in the subgraph rooted at edge ( v 1 , w 2 ) , we do the 1-WL color update on all node v 2 . Meanwhile, the initial color is the node marking of v 1 and w 2 , which is the most powerful subgraph selection policy [19]. First, we show some useful lemma for edge-based subgraph GNNs.

Lemma C.4. Given two graphs G , H , for any two tuples ( v 11 , v 12 ) ∈ V 2 ( G ) , ( v 21 , v 22 ) ∈ V 2 ( H ) , w 12 ∈ Q 1 ( v 11 ) , and w 22 ∈ Q 1 ( v 21 ) , we have ˜ C ∞ ( v 11 , v 11 , w 12 ) = ˜ C ∞ ( v 21 , v 21 , w 22 ) and ˜ C ∞ ( v 11 , w 21 , w 21 ) = ˜ C ∞ ( v 21 , w 22 , w 22 ) if ˜ C ∞ ( v 11 , v 12 , w 12 ) = ˜ C ∞ ( v 21 , v 22 , w 22 ) .

̸

̸

Proof. We prove it by contradiction. First, if v 12 = v 11 or v 12 = w 12 we must have v 22 = v 21 or v 22 = w 22 , respectively, as the initial color of ( v 1 , v 1 , w 2 ) and ( v 1 , w 2 , w 2 ) are different from other nodes in the subgraph. Further it is easy to show that ˜ C ∞ ( v 11 , v 11 , w 12 ) = ˜ C ∞ ( v 21 , v 21 , w 22 ) ⇐⇒ ˜ C ∞ ( v 11 , w 12 , w 12 ) = ˜ C ∞ ( v 21 , w 22 , w 22 ) as they are uniquely labeled. Second, if v 12 = v 11 , w 12 and v 22 = v 21 , w 22 , we can leverage the fact that the information of tuple ( v 1 , v 1 , w 2 ) and ( v 1 , w 2 , w 2 ) can be passed to other nodes injectively through message passing. Suppose we have ˜ C ∞ ( v 11 , v 12 , w 12 ) = ˜ C ∞ ( v 21 , v 22 , w 22 ) , if ˜ C ∞ ( v 11 , v 11 , w 12 ) = ˜ C ∞ ( v 21 , v 21 , w 22 ) , by doing another m iterations ( m is large enough), this discrepancy must be reflected by ˜ C ∞ ( v 11 , v 12 , w 12 ) and ˜ C ∞ ( v 21 , v 22 , w 22 ) , respectively. Therefore, we must have ˜ C ∞ + m ( v 11 , v 12 , w 12 ) = ˜ C ∞ + m ( v 21 , v 22 , w 22 ) , which is a contradiction to the stable color. The case of ( v 1 , w 2 , w 2 ) is similar to the first one. This concludes the proof.

Lemma C.5. Given two graphs G , H , for any two tuples ( v 11 , v 12 ) ∈ V 2 ( G ) , ( v 21 , v 22 ) ∈ V 2 ( H ) , w 12 ∈ Q 1 ( v 11 ) , and w 22 ∈ Q 1 ( v 21 ) , we have { { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 11 ∈ V ( G )) } } = { { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 21 ∈ V ( H ) } } if ˜ C ∞ ( v 11 , v 11 , w 12 ) = ˜ C ∞ ( v 21 , v 21 , w 22 ) and ˜ C ∞ ( v 11 , w 12 , w 12 ) = ˜ C ∞ ( v 21 , w 22 , w 22 ) .

Proof. This lemma is an extension of Lemma B.6 in [12]. First, iven Lemma B.6 in [12], it is easy to conclude that we have { { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 11 ∈ Q k 1 ( v 11 ) ∩ Q k 2 ( w 12 ) } } = { { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 21 ∈ Q k 1 ( v 21 ) ∩ Q k 2 ( w 22 ) } } for any k 1 , k 2 ∈ N . This directly implies { { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 11 ∈ V ( G )) } } = { { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 21 ∈ V ( H ) } } . This concludes the proof.

Lemma C.6. Given two graphs G , H , for any two tuples ( v 11 , v 12 ) ∈ V 2 ( G ) , ( v 21 , v 22 ) ∈ V 2 ( H ) , w 12 ∈ Q 1 ( v 11 ) , and w 22 ∈ Q 1 ( v 21 ) , we have { { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 11 ∈ V ( G )) } } = { { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 21 ∈ V ( H ) } } if ˜ C ∞ ( v 11 , v 12 , w 12 ) = ˜ C ∞ ( v 21 , v 22 , w 22 ) .

Proof. Given Lemma C.4, if ˜ C ∞ ( v 11 , v 12 , w 12 ) = ˜ C ∞ ( v 21 , v 22 , w 22 ) , we have ˜ C ∞ ( v 11 , v 11 , w 12 ) = ˜ C ∞ ( v 21 , v 21 , w 22 ) and ˜ C ∞ ( v 11 , w 21 , w 21 ) = ˜ C ∞ ( v 21 , w 22 , w 22 ) . Next, given Lemma C.5, we have { { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 11 ∈ V ( G )) } } = { { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 21 ∈ V ( H ) } } . This concludes the proof.

Lemma C.7. For any two graphs G , H and two nodes v 11 ∈ V ( G ) and v 21 ∈ V ( H ) , we have { {{ { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 11 ∈ V ( G ) } }| w 12 ∈ Q 1 ( v 11 ) } } = { {{ { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 21 ∈ V ( H ) } }| w 22 ∈ Q 1 ( v 21 ) } } if { {{ { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 12 ∈ Q 1 ( v 11 ) } }| w 11 ∈ V ( G ) } } = { {{ { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 22 ∈ Q 1 ( v 21 ) } }| w 21 ∈ V ( H ) } } .

Proof. If we have { {{ { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 12 ∈ Q 1 ( v 11 ) } }| w 11 ∈ V ( G ) } } = { {{ { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 22 ∈ Q 1 ( v 21 ) } }| w 21 ∈ V ( H ) } } , for any w 11 ∈ V ( G ) , we can find a w 21 ∈ V ( H ) such that { { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 12 ∈ Q 1 ( v 11 ) } } = { { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 22 ∈ Q 1 ( v 21 ) } } . Next, as the color is stable, based on Lemma C.6, we can conclude that { {{ { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 11 ∈ V ( G ) } }| w 12 ∈ Q 1 ( v 11 ) } } = { {{ { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 21 ∈ V ( H ) } }| w 22 ∈ Q 1 ( v 21 ) } } .

Now, given t = 2 , k = 2 , and ES 2 ( v ) = Q 1 ( v 2 ) × Q 1 ( v 1 ) , the corresponding ( k, t ) -FWL+ instance can be written as:

̸

̸

However, edge-based subgraph GNNs use 3-tuple representations instead of 2-tuples in the ( k, t ) -FWL+, which makes the comparison not trivial. Therefore, we slightly rewrite the above equation:

where we let ˆ C 0 ( v 1 , v 2 , w 2 ) and ˆ C 0 ( v 1 , v 2 , w 1 , w 2 ) be the same for any ( v 1 , v 2 ) ∈ V 2 ( G ) , w 1 ∈ Q 1 ( v 2 ) , and w 2 ∈ Q 1 ( v 1 ) . It is easy to verify that the above equations the equivalent to the original one. Now, we prove a strong lemma:

Lemma C.8. For any two graphs G , H , given two tuples ( v 11 , v 12 ) ∈ V 2 ( G ) and ( v 21 , v 22 ) ∈ V 2 ( H ) , for any ( w 11 , w 12 ) ∈ Q 1 ( v 12 ) × Q 1 ( v 11 ) and ( w 21 , w 22 ) ∈ Q 1 ( v 22 ) × Q 1 ( v 21 ) , we have ˜ C l -1 ( v 11 , w 11 , w 12 ) = ˜ C l -1 ( v 21 , w 21 , w 22 ) if ˆ C l ( v 11 , v 12 , w 11 , w 12 ) = ˆ C l ( v 21 , v 22 , w 21 , w 22 ) for any l ≥ 1 .

Proof. We prove it by induction on l . The base case is easy to verify given Lemma B.1 and the definition of initial color in edge-based subgraph GNNs.

Now suppose it is true for l = 2 , . . . , l -1 . At iteration l , if ˆ C l ( v 11 , v 12 , w 11 , w 12 ) = ˆ C l ( v 21 , v 22 , w 21 , w 22 ) , given Equation 33, we must have:

for any ( u 11 , u 12 ) ∈ Q F ( w 11 ,w 12 ) ( v 11 , v 12 ) and ( u 21 , u 22 ) ∈ Q F ( w 21 ,w 22 ) ( v 21 , v 22 ) at the same position. Particularly, we have ( v 11 , w 11 ) ∈ Q F ( w 11 ,w 12 ) ( v 11 , v 12 ) and ( v 21 , w 21 ) ∈ Q F ( w 21 ,w 22 ) ( v 21 , v 22 ) and they are in the same position. Given Equation (35), we have:

Equation (38) further indicates that for any y 12 ∈ Q 1 ( v 11 ) , we can find a y 22 ∈ Q 1 ( v 21 ) such that

Next, given Equation (34), we can conclude that:

Since the statement follows at iteration l -1 , we have { { ˜ C l -2 ( v 11 , x 11 , y 12 ) | x 11 ∈ Q 1 ( w 11 ) } } = { { ˜ C l -2 ( v 21 , x 21 , y 12 ) | x 21 ∈ Q 1 ( w 21 ) } } . We also have ˜ C l -2 ( v 11 , w 11 , y 12 ) = ˜ C l -2 ( v 21 , w 21 , y 22 ) given Equation (39). This means that we must have ˜ C l -1 ( v 11 , w 11 , y 12 ) = ˜ C l -1 ( v 21 , w 21 , y 22 ) based on Equation (30). Finally, as w 12 is one of y 12 and w 22 is one of y 22 and it is one such pair that holds the statement at iteration l -1 . It must be the case that they are also the pair at iteration l . This means we have ˜ C l -1 ( v 11 , w 11 , w 12 ) = ˜ C l -1 ( v 21 , w 21 , w 22 ) , which concludes the proof.

Now we prove Proposition 3.8 in the main paper. We restate it here:

Proposition C.9. Let t = 2 , k = 2 , and ES 2 ( v ) = Q 1 ( v 2 ) × Q 1 ( v 1 ) , the corresponding ( k, t ) -FWL+ instance is more powerful than edge-based subgraph GNNs like I 2 -GNN [19].

Proof. Note that the initial color of edge-based subgraph GNNs is weaker than ( k, t ) -FWL+ given Lemma C.8, but it is still sufficient to show that the stable color from ( k, t ) -FWL+ cannot be further refined by edge-based subgraph GNNs. For any two graphs G and H with ˆ C ∞ ( G ) = ˆ C ∞ ( H ) , it is easy to see that for any ( v 11 , v 12 ) ∈ V 2 ( G ) and w 12 ∈ Q 1 ( v 11 ) , we can find a ( v 21 , v 22 ) ∈ V 2 ( H ) and w 22 ∈ Q 1 ( v 21 ) with ˆ C ∞ ( v 11 , v 12 , w 12 ) = ˆ C ∞ ( v 21 , v 22 , w 22 ) . Then, given Equation (34), we have { { ˆ C ∞ ( v 11 , v 12 , w 11 , w 12 ) | w 11 ∈ Q 1 ( v 12 ) } } = { { ˆ C ∞ ( v 21 , v 22 , w 21 , w 22 ) | w 21 ∈ Q 1 ( v 22 ) } } . Further, given Lemma C.8, we have { { ˜ C ∞ ( v 11 , w 11 , w 12 ) | w 11 ∈ Q 1 ( v 12 ) } } = { { ˜ C ∞ ( v 21 , w 21 , w 22 ) | w 21 ∈

Q 1 ( v 22 ) } } Now, if we use the stable color from ( k, t ) -FWL+ as the initial color to do one more iteration of edge-based subgraph GNNs, it is easy to see that the color of ( v 11 , w 11 , w 12 ) and ( v 21 , w 21 , w 22 ) cannot get further refined and we have ˜ C ∞ ( v 11 , v 12 , w 12 ) = ˜ C ∞ ( v 21 , v 22 , w 22 ) . Note that the graph color histogram of edge-based subgraph GNNs is slightly different than ( k, t ) -FWL+ as in edge-based subgraph GNNs, the order of multiset is v 2 → w 2 → v 1 . In ( k, t ) -FWL+, the order of multiset is w 2 → v 2 → v 1 . However, based on Lemma C.7, the first order is less powerful than the second order. Therefore we can conclude that the final graph color histogram of edge-based subgraph GNNs will also be the same. This concludes the proof.

KP-GNN [22]: The GNN aggregation scheme of KP-GNN with the shortest path distance kernel and peripheral subgraph encoder as 1-WL can be written as:

where the initial color of ˜ C 0 ( v 1 , v 2 ) is the isomorphism type of ( v 1 , v 2 ) and SPD ( v 1 , v 2 ) . Before we prove Proposition 3.9, we first show that a slightly stronger version of ( k, t ) -FWL+ is still bounded by KP-GNN.

Lemma C.10. Let t = 1 , k = 2 , and ES t ( v ) = Q SPD ( v 1 ,v 2 ) ( v 1 ) ∩ Q 1 ( v 2 ) . Further let the initial color ˆ C 0 ( v 1 , v 2 ) encode SPD ( v 1 , v 2 ) , the corresponding ( k, t ) -FWL+ instance is at most as powerful as KP-GNN [22] with the peripheral subgraph encoder as powerful as 1-WL.

Proof. Given t = 1 , k = 2 , and ES t ( v ) = Q SPD ( v 1 ,v 2 ) ( v 1 ) ∩ Q 1 ( v 2 ) , the color update equation of ( k, t ) -FWL+ can be written as:

We can see Equation (43) exactly match Equation (42). However, the update of ˆ C l ( v 1 , v 1 ) is not equal to Equation (41) as the neighbor of ( v 1 , v 1 ) in ( k, t ) -FWL+ only contain itself. Therefore it is easy to see that for any l given two nodes v 1 , v 2 ∈ V ( G ) , must have ˜ C l ( v 1 , v 1 ) = ˜ C l ( v 2 , v 2 ) if ˆ C l ( v 1 , v 1 ) = ˆ C l ( v 2 , v 2 ) . Given two graphs G and H such that ˜ C ∞ ( G ) = ˜ C ∞ ( H ) , for any ( v 11 , v 12 ) ∈ V 2 ( G ) and ( v 21 , v 22 ) ∈ V 2 ( H ) with ˜ C ∞ ( v 11 , v 12 ) = ˜ C ∞ ( v 21 , v 22 ) , we must have { { ˜ C ∞ ( v 11 , w ) | w ∈ Q SPD ( v 11 ,v 12 ) ( v 11 ) ∩ Q 1 ( v 12 ) } } = { { ˜ C ∞ ( v 21 , w ) | w ∈ Q SPD ( v 21 ,v 22 ) ( v 21 ) ∩ Q 1 ( v 22 ) } } . Thus, it is easy to verify that if we use the stable color output by KP-GNN as initial color to perform 1 iteration of ( k, t ) -FWL+, we have ˆ C ∞ ( v 11 , v 12 ) = ˆ C ∞ ( v 21 , v 22 ) . Further, given Equation (41), for any ( v 11 , v 11 ) ∈ V 2 ( G ) , we can find a ( v 22 , v 22 ) ∈ V 2 ( H ) such that ˜ C ∞ ( v 11 , v 11 ) = ˜ C ∞ ( v 22 , v 22 ) , which means { { ˜ C ∞ ( v 11 , w 12 ) | w 12 ∈ V ( G ) } } = { { ˜ C ∞ ( v 22 , w 22 ) , | w 22 ∈ V ( H ) } } , and thus { { ˜ C ∞ ( w 11 , w 12 ) | ( w 11 , w 12 ) ∈ V 2 ( G ) } } = { { ˜ C ∞ ( w 21 , w 22 ) | ( w 21 , w 22 ) ∈ V 2 ( H ) } } . This indicates that even if ( k, t ) -FWL+ use the shortest path distance as the initial color, we still have { { ˆ C ∞ ( w 11 , w 12 ) | ( w 11 , w 12 ) ∈ V 2 ( G ) } } = { { ˆ C ∞ ( w 21 , w 22 ) | ( w 21 , w 22 ) ∈ V 2 ( H ) } } and thus the final graph color histogram cannot be refined. This concludes the proof.

Now, it is easy to prove Proposition 3.9 in the main paper. We restate it here:

Proposition C.11. Let t = 1 , k = 2 , and ES ( v ) = Q SPD ( v 1 ,v 2 ) ( v 1 ) ∩ Q 1 ( v 2 ) , the corresponding ( k, t ) -FWL+ instance is at most as powerful as KP-GNN [22] with the peripheral subgraph encoder as powerful as 1-WL.

Proof. It is easy to see that the corresponding ( k, t ) -FWL+ is weaker than Equation (43) with the initial color as the shortest path distance, we omit the detailed proof. Then, given Lemma C.10, we can directly conclude the proof.

We conjecture that if we further equip ( k, t ) -FWL+ with Equation (41), ( k, t ) -FWL+ can be as powerful as KP-GNN. We also conjecture that if we let t = p and k = 2 , ES ( v ) = ( Q SPD ( v 1 , v 2 )( v 1 ) ∩ Q 1 ( v 2 )) p with initial color as the shortest path distance, the resulting ( k, t ) -FWL+ has a close relationship with KP-GNN with the peripheral subgraph encoder as powerful as LFWL( p ) [12] and leave the detailed proof in the future.

̸

GDGNN [23]. Before we give the formal update equation of GDGNN, we first formally define SP ( v 1 , v 2 ) as a set of nodes that are in the shortest distance path between node v 1 and v 2 . Namely, SP ( v 1 , v 2 ) = { w | SPD ( v 1 , w ) + SPD ( w,v 2 ) = SPD ( v 1 , v 2 ) , ∀ w ∈ V ( G ) } if v 1 = v 2 and SP ( v 1 , v 1 ) = { v 1 } . Then, the color update equation of GDGNN can be written as:

Note that Equation (44) is slightly different from the original version of GDGNN [23] stated in their paper. In the original paper, the author use MPNNs in O ( n ) space. Meanwhile, they only consider the direct neighbor of node v 1 and v 2 in SP ( v 1 , v 2 ) , and they only add it in the last layer. However, it is easy to verify that Equation (44) is at least as powerful as the original version and we show that the ( k, t ) -FWL+ can be more powerful than this version. Now we prove Proposition 3.10. We restate it here:

Proposition C.12. Let t = 2 , k = 2 , and ES 2 ( v ) = Q 1 ( v 2 ) × SP ( v 1 , v 2 ) , the corresponding ( k, t ) -FWL+ is more powerful than GDGNN [23].

Proof. Given t = 2 , k = 2 , and ES 2 ( v ) = Q 1 ( v 2 ) × SP ( v 1 , v 2 ) , the color update equation of ( k, t ) -FWL+ can be written as:

First, as ( v 1 , w 1 ) ∈ Q F ( w 1 ,w 2 ) ( v 1 , v 2 ) , we can directly conclude that ( k, t ) -FWL+ can injectively encode { { ˆ C l ( v 1 , w 1 ) | w 1 ∈ Q 1 ( v 2 ) } } for any l .

Second, we also have ( v 1 , w 2 ) ∈ Q F ( w 1 ,w 2 ) ( v 1 , v 2 ) . Further, as in the inner multiset, each element has the same w 2 . This means that the result of the inner multiset will not affect the information in the outer multiset and it is easy to verify ( k, t ) -FWL+ can injectively encode { { ˆ C l ( v 1 , w 2 ) | w 2 ∈ SP ( v 1 , v 2 ) } } .

Now, given any two graphs G , H with ˆ C ∞ ( G ) = ˆ C ∞ ( H ) . For any ( v 11 , v 12 ) ∈ V 2 ( G ) and ( v 21 , v 22 ) ∈ V 2 ( H ) with ˆ C ∞ ( v 11 , v 12 ) = ˆ C ∞ ( v 21 , v 22 ) we must have { { ˆ C ∞ ( v 11 , w 11 ) | w 11 ∈ Q 1 ( v 12 ) } } = { { ˆ C ∞ ( v 21 , w 21 ) | w 21 ∈ Q 1 ( v 22 ) } } and { { ˆ C ∞ ( v 11 , w 12 ) | w 12 ∈ SP ( v 11 , v 12 ) } } = { { ˆ C ∞ ( v 21 , w 22 ) | w 22 ∈ SP ( v 21 , v 22 ) } } based on two statements. Then, it is easy to verify that if we use the stable color from ( k, t ) -FWL+ as the initial color to do one more iteration of GDGNN, the color of ( v 11 , v 12 ) and ( v 21 , v 22 ) cannot be further refined so as the final graph color histogram. This concludes the proof.

C.2 Discussion on the complexity

The space complexity of ( k, t ) -FWL+ is still O ( n k ) . However, the time complexity of ( k, t ) -FWL+ is dependent on the choice of ES ( v ) and it is hard to give a formal analysis.

C.3 Limitations

In the paper, we implement several different instances of ( k, t ) -FWL+ which expressive power is closely related to existing expressive GNNs. However, the whole space of ( k, t ) -FWL+ is less explored. Especially, how many different ES ( v ) exists? Moreover, how do theoretically and quantitatively analyze the expressive power of ( k, t ) -FWL+? Finally, although ( k, t ) -FWL+ can be arbitrarily powerful even within O ( n 2 ) space from the theoretical side, we observe it is sometimes hard to optimize the model to achieve its theoretical power by using fewer embedding. Specifically, how to ensure the injectiveness of the multiset function? For example, if we use large t and small k for ( k, t ) -FWL+ to conduct a graph isomorphism test, for each k -tuple, the hierarchical multiset { {} } t contains exponentially increased elements. So far, we only use the summation as the function to encode this hierarchical multiset. Although it is an injective function from a theoretical view, we find it works badly with information loss when the number of elements is too large. It is worth investigating how to design model architecture that can unleash the full power of ( k, t ) -FWL+. We leave all these to our future work.

D Additional discussion on N 2 -FWL and N 2 -GNN

D.1 Detailed proofs for N 2 -FWL

In this section, we provide all detailed proofs for N 2 -FWL. To make it clear, we denote ˆ C l ( v 1 , v 2 ) as the color of tuple ( v 1 , v 2 ) at iteration l for N 2 -FWL. We rewrite Equation 9 as:

We start with the first lemma:

Lemma D.1. Given h large enough, N 2 -FWL is more powerful than SLFWL(2) [12].

Proof. Given h large enough, it is clearly that the neighbor of N 2 -FWL becomes N 1 ( v 2 ) ×N 1 ( v 1 ) . For any ( w 1 , w 2 ) ∈ N 1 ( v 2 ) ×N 1 ( v 1 ) , it is easy to verify that Q F ( w 1 ,w 2 ) ( v 1 , v 2 ) contains all tuples in the aggregation of SLFWL(2). Further, the elements in N 1 ( v 2 ) ×N 1 ( v 1 ) includes all elements in Q 1 ( v 2 ) ∪ Q 1 ( v 1 ) . Therefore, it is easy to prove that N 2 -FWL is more powerful than SLFWL(2) by doing induction on l . Here we omit the detailed proof.

Lemma D.2. Given h large enough, N 2 -FWL is more powerful than edge-based subgraph GNNs.

Proof. Given h large enough, it is clear that the neighbor of N 2 -FWL becomes N 1 ( v 2 ) ×N 1 ( v 1 ) . The N 2 -FWL exactly matches the instance in Proposition 3.8 with additional root nodes. As the initial feature of root tuples( ( v 1 , v 1 ) , ( v 2 , v 2 ) ) are different from other tuples, its information will not affect other neighbors. This concludes the proof.

It is intuitive to conjecture that N 2 -FWL with the number of h is more powerful than edge-based subgraph GNNs with h -hop subgraph and we leave the detailed proof in the future. Now, we can prove Theorem 4.1 in the main paper. We restate it here:

Theorem D.3. Given h is large enough, N 2 -FWL is more powerful than SLFWL(2) [12] and edgebased subgraph GNNs.

Proof. Lemma D.1 and Lemma D.2 directly implies the theorem. This concludes the proof.

Next, we prove Theorem 4.3 in the main paper. We restate it here:

Theorem D.4. N 2 -FWL can count up to (1) 6-cycles; (2) all connected graphlets with size 4; (3) 4-paths at node level.

Proof. Given Theorem 4.1, N 2 -FWL is more powerful than I 2 -GNN [19]. Given Theorem 4.1, 4.2, 4.3 in [19], I 2 -GNN [19] is able to count up to (1) 6-cycles; (2) all connected graphlets with size 4; (3) 4-paths at node level. This concludes the proof.

Then, we prove Corollary 4.2 in the main paper. We restate it here:

Corollary D.5. N 2 -FWL is strictly more powerful than all existing node-based subgraph GNNs and no less powerful than 3-WL.

Proof. Given Theorem 4.1, this corollary can be directly concluded given Theorem 7.1 in [12] and Proposition 4.1 in [19]. But we further give an example to show that even with h = 1 , N 2 -FWL is already powerful enough to distinguish some non-isomorphic strongly regular graph pairs that 3-WL cannot distinguish. When h = 1 , the neighbors of N 2 -FWL is ( N 1 ( v 2 ) × N 1 ( v 1 )) ∩ ( N 1 ( v 1 ) ∩ N 1 ( v 2 )) 2 . Consider the Shrikhande graph and 4 × 4 Rook's graph in Figure 2. It is well known this pair of graphs cannot be distinguished by 3-WL. First, look at node v 1 and v 2 in Shrikhande graph, N 1 ( v 1 ) ∩ N 1 ( v 2 ) includes two green nodes, v 1 , and v 2 . This means that ( w 1 , w 2 ) ∈ ( N 1 ( v 2 ) ×N 1 ( v 1 )) ∩ ( N 1 ( v 1 ) ∩ N 1 ( v 2 )) 2 contains a pair of ( w 1 , w 2 ) such that w 1 is one green node and w 2 is the other. Given Definition A.2, ( w 1 , w 2 ) ∈ Q F ( w 1 ,w 2 ) ( v 1 , v 2 ) . This means that the information of ˆ C 0 ( w 1 , w 2 ) will be encoded into ˆ C 1 ( v 1 , v 2 ) and we have ( w 1 , w 2 ) is not an

Shrikhande Graph

4×4 Rook's Graph

𝑣

!

𝑣

"

𝑣

!

𝑣

"

edge. Similarly, in 4 × 4 Rook's graph, N 1 ( v 1 ) ∩ N 1 ( v 2 ) includes two green nodes, v 1 , and v 2 . Thus, ˆ C 0 ( w 1 , w 2 ) will be encoded into ˆ C 1 ( v 1 , v 2 ) . However, ( w 1 , w 2 ) here is an edge, which will have a different isomorphism type than ( w 1 , w 2 ) in the Shrikhande graph, thus resulting in different ˆ C 1 ( v 1 , v 2 ) and further graph color histogram. This concludes the proof.

Finally, we prove Proposition 4.4 in the main paper. We restate it here:

Proposition D.6. If U l , M l , and R are injective functions, there exist a N 2 -GNN instance that can be as powerful as N 2 -FWL.

Proof. If U l , M l , and R are injective, they are exactly the same as HASH used in N 2 -FWL. Therefore, it is easy to prove it by doing induction on l to show that if N 2 -GNN outputs the same color for two tuples, N 2 -FWL also does for any l ≥ 0 and vice versa. We omit the detailed proof here.

D.2 Complexity analysis

In this section, we provide a complexity analysis of N 2 -GNN. Let m denote the number of edges in the graph and d denote the average degree of the graph. First, we consider h is large enough, where the neighbor of N 2 -GNN becomes N 1 ( v 2 ) ×N 1 ( v 1 ) . As N 2 -GNN is an instance of (2 , 2) -FWL, it is straightforward that the space complexity of N 2 -GNN is upper-bounded by O ( n 2 ) . Further, each tuple needs to aggregate ( d +1) 2 different neighbors, which results in a total time complexity of O ( n 2 d 2 ) = O ( m 2 ) . Notes that the time complexity of N 2 -GNN is the same as edge-based subgraph GNNs. However, the space complexity of N 2 -GNN is O ( n 2 ) , which is far less than edge-based subgraph GNNs with a O ( nm ) space complexity.

If we consider a pratical h , the neighbors of N 2 -GNN is ( N 1 ( v 2 ) ×N 1 ( v 1 )) ∩ ( N h ( v 1 ) ∩N h ( v 2 )) 2 . This means, only ( w 1 , w 2 ) that within the overlapping h -hop subgraph of v 1 and v 2 can send message to v 1 , v 2 . Therefore, the time complexity is bounded by O ( nd h +2 ) , which is still the same as edgebased subgraph GNNs with h -hop subgraph selection policy. Further, it is easy to verify that only tuple ( v 1 , v 2 ) with SPD ( v 1 , v 2 ) ≤ h can update its representation. Thus, in memory-sensitive tasks, we can only keep representation for tuples that can be updated, resulting in a space complexity of O ( nd h ) .

D.3 Model implementation details

Here we provide the details about model implementation. As N 2 -GNN is an instance of (2 , 2) -FWL+, we are dealing with a tuple of size 6 in the aggregation process. Note that for any tuple ( v 1 , v 2 ) , Q F ( w 1 ,w 2 ) ( v 1 , v 2 ) will also include itself. However, it does not increase the expressive power as

we already encode its information. Therefore, in the implementation, we remove it from the tuple. Thus, we implement the neighbor tuple as follows: for any 2-tuple ( v 1 , v 2 ) , the message from each neighbor ( w 1 , w 2 ) ∈ N 2 ( v 1 , v 2 ) is defined as (( v 1 , w 1 ) , ( v 1 , w 2 ) , ( w 1 , v 2 ) , ( w 2 , v 2 ) , ( w 1 , w 2 )) . Note the order of tuples is flexible as long as it is uniform across different neighbors, according to Definition A.2. We implement N 2 -GNN in a memory-saving way. To make it more clear, we slightly redefine some notations. Let h l ( v 1 , v 2 ) be the representation of tuple ( v 1 , v 2 ) , m l ( v 1 ,v 2 ) ( w 1 , w 2 ) be the message from the neighbor ( w 1 , w 2 ) , N 2 ( v 1 , v 2 ) be the set of all neighborhoods of ( v 1 , v 2 ) in N 2 -GNN with the output embedding hidden size of hs . We let the initial feature of tuple h 0 ( v 1 , v 2 ) = MLP ( l G ( v 2 ) + SPD ( v 1 , v 2 )) . The message function is implemented as:

where LIN 1 is a linear layer project the embedding from hidden size h to a lower value d , named as inner size, W i ∈ R d , i ∈ [5] are learnable weight vector with size d and ∗ is element-wise product, CON is the concatenation, and LIN 2 is another linear layer project the concatenated embedding from size 5 ∗ d back to hidden size h . It is easy to validate that this implementation is injective to tuple input. Next, the update function U l is implemented as:

where ϵ l is a learnable scalar and MLP l is a one-hidden-layer MLP with hidden size and output size equal to h and activation function be ReLu. Note that in the implementation we do not consider the hierarchical multiset but directly pooling all neighbors together. We observe it already achieves superior results in all real-world tasks and is more parameter-saving.

As discussed in [12, 27], for each h l -1 ( v 1 , v 2 ) , add the representation of h l -1 ( v 2 , v 2 ) can boost the performance in real-world tasks. Therefore, we adopt a similar way for all real-world experiments:

If add this additional aggregation, we will use the ˆ h l ( v 1 , v 2 ) instead of h l ( v 1 , v 2 ) as the output representation. We conjecture this works as virtual nodes in real-work tasks, which allows a better mixture of node features and structure information. As indicated by [12], adding this term will not increase the expressive power from the theoretical side. Finally, the readout function is implemented in a hierarchical way:

where POOL 1 and POOL 2 are two different pooling function.

In addition, we adopted two implementation strategies for N 2 -GNN. The first strategy is to keep all O ( n 2 ) tuples even if it has a distance greater than h . Notes that it will not aggregate information from other tuples but is only updated in Equation (50) and Equation (51). We observe it has great performance in real-world tasks combined with Equation (52) and Equation (53). The second implementation is used for memory-sensitive tasks. We remove all tuples that have a distance greater than h . We refer to the first strategy as the dense version and the second strategy as the sparse version. We will specify which version we used in our experiments later on. The code is implemented using PyG [52]. We provide open-sourced code in https://github.com/JiaruiFeng/N2GNN for reproducing all results reported in the main paper.

D.4 Limitations

Although N 2 -GNN has efficient theoretical space complexity, the empirical space complexity can still be high as we need to keep neighbors index for all tuple ( u 1 , u 2 ) ∈ Q F ( w 1 ,w 2 ) ( v 1 , v 2 ) . This introduces

a constant factor (5 in the implementation) on the saving edge index and corresponding gradients during training (see Table 9 and Table 10 for the practical time and memory comparison). It is still worth studying how to improve the empirical space complexity of N 2 -GNN. Further, in the current implementation, we didn't consider the hierarchical set but directly pooled all elements together to save the parameters and computational cost. Additionally, each tuple needs to aggregate many more neighbors than normal MPNNs, which makes the sum aggregation hard to achieve injectiveness during the aggregation, introducing the optimization issue. These hinder the model from achieving its theoretical power. Finally, although N 2 -GNN has O ( nd h +2 ) time complexity, which is practical for most real-world tasks, the complexity still goes exponentially if the graph is dense (like a strongly regular graph). But as the design space of ( k, t ) -FWL+ is extremely flexible, it is easy to design more practical and expressive instances suitable for dense graphs.

E Experimental details

This section presents additional information on the experimental settings.

E.1 Dataset statistics

E.2 Experimental details

In this section, we provide all the experimental details. First, we discuss some general experimental settings that, if not specified further, are consistent across all experiments.

Model . We adopt POOL 1 as summation and POOL 2 as mean pooling in Equation (52) and Equation (53). We set the ϵ l in Equation (49), ϵ l a in Equation (50) to be 0.0 and fix it during training and inference. At each MLP, we add batch normalization.

Training . For all experiments, we use Adam as the optimizer and ReduceLRonPlateau as the learning rate scheduler. We set l2 weight decay to 0.0 across all experiments. All experiments are conducted on a single NVIDIA A100 80GB GPU.

E.2.1 Graph structures counting

Model . For the substructure counting dataset, we adopt a similar setting as I 2 -GNN [19]. Specifically, we set the number of hop h to be 1, 2, 2, 3, 2, 2, 1, 4, 2 for 3-cycles, 4-cycles, 5-cycles, 6-cycles, tailed triangles, chordal cycles, 4-cliques, 4-paths, and triangle-rectangle respectively. For all substructures, we set the number of layers to 5, the hidden size h to 96, and the inner size d to 32. In the substructure count dataset, we remove all normalization layers. Finally, we use the sparse version of data preprocessing.

Training . The training/validation/test splitting ratio is 0.3/0.2/0.5. The initial learning rate is 0.001 and the minimum learning rate is 1e-5. The patience and factor of the scheduler are 10 and 0.9 respectively. The batch size is set to 256 and the number of epochs is 2000. For all substructures, we run the experiments 3 times and report the mean results on the test dataset.

E.2.2 ZINC

Model . The setting is the same for both ZINC-Subset and ZINC-Full except for the inner size. We set the number of hop h to be 3. We set the number of layers to 6, and the hidden size h to 96. The inner size d is 20 for the ZINC-subset and 48 for the ZINC-full.

Training . The initial learning rate is 0.001 and the minimum learning rate is 1e-6. The patience and factor of the scheduler are 20 and 0.5 respectively. The batch size is set to 128 and the number of epochs is 500. For both ZINC-Subset and ZINC-Full, we run experiments 10 times and report the mean results and standard deviation on the test dataset.

E.2.3 QM9

Model . For all targets, we adopt the same setting. We set the number of hop h to be 3. We set the number of layers to 6, the hidden size h to 120, and the inner size d to 32. We further add the resistance distance feature into the model, similar to the implementation in Feng et al. [22], Huang et al. [19].

Training . The initial learning rate is 0.001. The patience and factor of the scheduler are 5 and 0.9 respectively. The batch size is set to 128 and the number of epochs is 350. we run experiments 1 time and report the test result with the model of the best validation result.

E.2.4 CSL

Model : We set the number of hop h to be 4, the number of layers to be 4, the hidden size h to 48, and the inner size d to 16. We use the sparse version of data preprocessing and we do not add root aggregation (Equation (50)-(51)).

Training . The initial learning rate is 0.001 and the minimum learning rate is 1e-6. The patience and factor of the scheduler are 20 and 0.5 respectively. The batch size is set to 32 and the number of epochs is 80. We use 10-fold cross-validation and report the average results.

E.2.5 SR25

Model : We set the number of hop h to be 1, the number of layers to be 6, the hidden size h to 64, and the inner size d to 16. We use the sparse version of data preprocessing and we do not add root aggregation (Equation (50)-(51)). Further, the normalization layer in SR25 is set to Layer normalization to avoid test/training mismatch.

Training . The initial learning rate is 0.001 and the minimum learning rate is 1e-6. The patience and factor of the scheduler are 200 and 0.5 respectively. The batch size is set to 15 and the number of epochs is 800. We report the single-time results.

E.2.6 EXP

Model : We set the number of hop h to be 3, the number of layers to be 4, the hidden size to 48, and the inner size d to 24. We use the sparse version of data preprocessing and we do not add root aggregation (Equation (50)-(51)).

Training . The initial learning rate is 0.001 and the minimum learning rate is 1e-6. The patience and factor of the scheduler are 20 and 0.5 respectively. The batch size is set to 32 and the number of epochs is 200. We use 10-fold cross-validation and report the average results.

E.2.7 BREC

Model : We set the number of hop h to be 8, the number of layers to be 4, the hidden size to 64, and the inner size d to 32. Meanwhile, the output channel is set to 16 for computing the similarity. We use the sparse version of data preprocessing and we do not add root aggregation (Equation (50)-(51)).

Training . We follow the same training and evaluation procedure as described by BREC [45]. The initial learning rate is 0.001 and the weight decay is set to 0.0001. The batch size is set to 4 and the number of epochs is 20. Due to the resource limitation, we only ran the experiment on 340 graph pairs (removed 20 4-vertex-condition graphs and 40 CFI graphs), which follow the same protocol as the reported result for I 2 -GNN.

F Additional experimental results

F.1 Ablation studies for ( k, t ) -FWL+

In this section, we provide an additional ablation study on (2 , t ) -FWL+ by varying different t and ES . We select two important ES mentioned in our paperQ 1 ( v ) and SP ( v 1 , v 2 ) . We conduct experiments on both Expressiveness verification datasets and counting datasets. The experiment setup is the same as what is discussed in the above section. The results can be seen in Table 7 and Table 8. Notes that we find the performance of SP or SP × SP on counting dataset is similar to MPNN and thus omit it in Table 8.

We can see that as long as t = 2 , we can have a perfect performance on SR25 no matter if we use SP or Q 1 ( v 1 ) , this aligns with our theoretical results. Further, SP ( v 1 , v 2 ) × Q 1 ( v 1 ) achieve great results on all counting tasks and expressiveness verification. This indicates the SP can also be a great choice in the practical scenario.

F.2 Practical complexity of N 2 -GNN

In this section, we provide practical complexity analysis for N 2 -GNN. Here we use the cycle counting datasets with a similar parameter size for all models and a batch size of 256. To ensure a fair comparison, in N 2 -GNN, we use the sparse version for data preprocessing (which is true for all compared models), and set the number of workers as 0 and random seed as 1 for all models. The experiments are run on a single TeslaV100 32GB GPU. We report the training time, maximum training memory, inference time, and maximum inference memory usage during inference for both h = 1 and h = 2 for an average of 20 epochs. The results is shown in Table 9 and Table 10.

We can see that the empirical memory usage of N 2 -GNN is much less than I 2 -GNN when h = 1 and comparable when h = 2 . We conjecture the reason why N 2 -GNN require more training memory than I 2 -GNN when h = 2 is that when h increase, each tuple will need to aggregate much more neighbors than each node in I 2 -GNN and each aggregation require a constant size (5 in N 2 -GNN) of more memory for saving the gradients. Meanwhile, As we mentioned in limitations (Appendix D), to enjoy some extent of parallelism, the current implementation of N 2 -GNN needs to save all neighbor indices, which introduces a constant factor of more memory on saving. But the inference memory of N 2 -GNN is less than I 2 -GNN in both h = 1 and h = 2 . The training time and inference time of N 2 -GNN are also more efficient than I 2 -GNN. However, currently, the time and memory cost by N 2 -GNN are higher than node-based subgraph GNNs. We do want to highlight several points: (1) By changing the way of implementation, N 2 -GNN can be implemented strictly within O ( n 2 ) space. Therefore, the merit of N 2 -GNN still holds. (2) The experiment is conducted on the same parameter budget level. However, in real-world tasks, N 2 -GNN needs much fewer parameters to achieve SOTA results (Table 5). (3) In this work, we not only aim to propose a new model but also mean to introduce a new flexible framework, ( k, t ) -FWL+. The empirical success of N 2 -GNN demonstrates the great potential of this framework for designing new expressive GNN variants.

F.3 Additional discussion on BREC experiment

In Table 11, we provide a complete comparison of all baseline methods on the BREC dataset. Please see the detailed description of all baseline methods in BREC [45]. We can see that N 2 -GNN achieve the best result among all GNN baselines and the second best result among all baselines, even if we only test on 340 pair of graphs. Particularly, we surpass I 2 -GNN on all parts. This result empirically verified Theorem 4.1 that the theoretical power of N 2 -GNN is more powerful than I 2 -GNN. Meanwhile, we notice that SSWL_P can distinguish more CFI graph pairs than ours, which may contradict Theorem 4.1. This may be due to the fact that we remove 40 CFI graph pairs in the experiments given the resource limitation and SSWL_P uses 8 layers in the experiment and we only use 4 layers. Meanwhile, we don't implement the hierarchical multiset in the current version of N 2 -GNN, which slightly compromises the expressive power.

F.4 Ablation studies for N 2 -GNN

In this subsection, we perform ablation studies to investigate each component in N 2 -GNN. All ablation studies are performed on ZINC-Subset and target C v in QM9. Except hyper-parameter mentioned later, all other setting is exactly the same as what is described in Section E.2.

In the first ablation study, we investigate different components of the model. Specifically, we analyze two components. The first component is aggregation h l ( v 1 , v 2 ) described in Equation (50)-(51). The second component is the dense/sparse embedding described in Section D.3. In the ablation study, we verify all different combinations of two components. For the model without Equation (50)-(51), to ensure that the size of the model is comparable, we increase the model hidden size to 120 and 136 for

ZINC-Subset and QM9, respectively. The results are shown in Table 12. First, we can see adding Equation (50)-(51) indeed improve the performance across different tasks. Second, by explicitly keeping O ( n 2 ) embedding the performance, of N 2 -GNN on ZINC-Subset improved from 0.079 to 0.059. This may indicate that even though the two models have the same expressive power from the theoretical side, adding additional feature embedding can still boost the performance significantly in real-world tasks as it allows a better mixture of both feature and structure information. Note that these two components were used in many previous models like ESAN [43], SUN [27], and SSWL+ [12], etc. However, our model still outperforms them by a large margin. This indicates that the expressive power of the model is still the most critical ingredient, adding these additional components is just a way to unleash all the potential of the model for real-world tasks.

In the second ablation study, we evaluate the effect of the number of hop h . We vary h from 1 to 4 and results are shown in Table 13. We can see an increase of h can boost the performance of the model, which has already been verified in previous subgraph-based methods. Particularly, for ZINC-Subset, we find that the third hop is critical to the performance. However, if we continue to increase the number of hops, the performance drops, which is a sign of overfitting.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper builds an extension to the $k$-FWL isomorphism test algorithms called $(k,t)$-FWL+. The proposed extended family subsumes the existing $k$-FWL and adds finer variants to the expressivity landscape of isomorphism testing algorithms. The $(k,t)$-FWL+ algorithm is a combination of two modified schemes to the original $k$-FWL algorithm. \n\nThe first one, called $(k,t)$-FWL, which also colors $k$ tuples, and draws insparation from the tuple based aggregation of the $k$-FWL algorithm vs. $k$-WL. The proposed algorithm changes the construction of neighborhood multi-stets used in the encoding step of the $k$-FWL algorithm. Briefly explained, instead of consisting of $n$ tuples of size $k$ , $(k,t)$-FWL aggregates a multi-sets consisting of $n^t$ tuples of size $\\sum_{i=0}^{\\min (k,t)}{ k \\choose i} {t \\choose i}$. Where each tuple in the new proposed algorithm holds more information than in the original $k$-FWL algorithm. The authors prove that for a fixed $k$, increasing $t$ induces a strict hierarchy where there exist a large enough $t$ for which the algorithm solves the isomorphism problem. \n\nThe second modification, called $k$-FWL+, unifies many previously introduced approaches, and suggests modifying the neighborhoods of tuples to be equivariant sets of nodes instead of the entire node set of the graph. Which can reduce time complexity of the algorithm. \n\nThe main contributions of the paper are (i) introducing a novel finer hierarchy of isomorphism testing algorithms; (ii) showing that it is possible to construct a strict hierarchy at fixed memory complexity that can achieve solving the isomorphism problem; (iii) coming up with a practical neural instantiation; and (iv) achieving state of the art results on common benchmarks.\n\nSTRENGTHS:\n- The paper is well written and successfully introduces the limitations of the $k$-WL hierarchy for the evaluation and design of more expressive GNNs. It is part of a body of works that refine the WL hierarchy aiming for more efficient algorithms with higher expressive power. \n\n- The paper introduces a general framework that is also shown to unify several recent approaches and can provide a canonical design space for previous and future GNNs. \n- The construction of the $(k,t)$-FWL hierarchy naturally arises from the original $k$-FWL algorithms and elegantly shifts memory complexity to runtime complexity in the fixed $k$ scenario. \n- The experimental evaluation is extensive and the proposed method seems to achieve SOTA performance.\n\nWEAKNESSES:\n- The paper does not properly discuss the \"no free lunch\" that hides in the proposed method. Indeed, the authors came up with a fixed space complexity hierarchy which, for large enough parameter $t$, can solve the isomorphism problem. But, this comes at the cost of time complexity - having to aggregate an exponentially growing number of tuples ($O(n^t)$). The authors deferred most of this discussion to the appendix, while I believe it should be placed in the main body of the paper.\n- A valuable ablation study lacking in the paper is an experiment showing the performance of several variants of the new proposed hierarchy on the substructure counting benchmark. Since it is unclear what each variant gains in expressiveness, I believe such an evaluation will contribute to the paper's intuition and wholeness.\n- Questionable results in the QM9 dataset. See the questions section.\n\nQUESTIONS:\n- A neural counterpart is suggested only for a single instance in the hierarchy. An interesting realm to explore would be to find a neural analog to the proposed hierarchy (similar to [1])\n- There seems to be a mismatch in the units for the QM9 experiment between your code and the code in the $I^2$-GNN repository [2]. They seem to convert back to previously used units to be consistent with previous works while your code uses the new conversion. Can the authors please clarify this issue? To my understanding, for example, the value that should be reported for $\\tilde{U}_0=U_0/0.04336414=0.5744$, where $U_0$ is the value reported in this paper. \n- Can the authors report runtimes of their GNN training and how it compares to other approaches? Since their approach trades space complexity for runtime complexity, I think it is necessary to show that the method remains scalable where the authors hinted that runtimes might increase when graphs are dense.\n\n- In the appendix, the authors say (l1094): \"...we observe it some time hard to optimize\n1095 the model to achieve its theoretical power by using fewer embedding\". Can the authors clarify this statement? \n\n\n[1] Provably Powerful Graph Neural Networks. Maron et. al. (NuerIPS 2019)\n\n[2] [$I^2$-GNN GitHub](https://github.com/GraphPKU/I2GNN/blob/master/run_qm9.py#L37)\n\nLIMITATIONS:\n- The authors describe the limitations of the method mainly in the appendix, I think some of the discussion should be moved to the main body (see questions section)\n\n**General comment:** I lean towards raising my score upon clarification of the experimental results. I think the proposed framework is a natural extension of the $k$-FWL family that elegantly formulates the trade-off between time and space complexity.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe Weisfeiler-Leman algorithm has been established as the formal tool of choice for measuring the expressivity of GNNs. The paper tackles the problem of designing \"higher-order\" GNNs, which has been previously considered by several papers. \n\nThe authors propose and combine two new ideas to supplement k-WL: (1) add in further subgraph information in a space-efficient manner via a hyperparameter $t$, and (2) allow neighbor selection in folklore-k-WL from any equivariant fashion, instead of the standard sets such as the set of all vertices or the set of all neighbors.  \n\nCombining these two ideas yields the (k,t)-FWL+ algorithm. The version for k=2 and t=2, i.e. (2,2)-FWL+, is thoroughly evaluated on several datasets, and it yields very good experimental results, esp. on the ZINC dataset.\n\nSTRENGTHS:\n1) The (2,2)-FWL+ performs very well on several datasets. On the ZINC datasets, they report 10.6% and 40.9% improvement over the second-best methods. \n\n2) The authors identify the problem of space limitations for higher-order GNNs and they seek to address them in their model.\n\nWEAKNESSES:\n1) The proposed algorithm lacks novelty: the claimed theoretical contributions are superficial. Graph Isomorphism can be obviously solved in quadratic space, if we were to allow iteration over all possible mappings between the two vertex sets. Such an algorithm might be space-efficient on paper but does not indicate any tractability since it brute-force tries all possible mappings. The proofs outlining the power and limitations of (k,t)-FWL are mild generalizations of known results, and lack any novel insights. \n\n2) Allowing equivariant sets for vertex choice in k-FWL is a reasonable idea: however, I would expect the authors to compare several \"equivariant set strategies\" and demonstrate the relative gains over non-classic equivariant sets such as V(G) or N(v,G), keeping other things equal (an ablation study). \n\nOverall, I feel that the paper is not much different from a multitude of papers on designing higher order GNNs using subgraph enhancement. Given the work that had already been done on this problem, the proposed model is incremental.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose generalizations of k-WL and k-FWL to (k, t)-FWL, and k-FWL+, which when combined, is notated as (k, t)-FWL+. They argue that (k, t)-FWL+ allow a more \"flexible and fine-grained\" space for exploring the graph expressiveness hierarchy, which is helpful in designing new GNN architectures. As a practical application, the authors propose Neighborhood^2-GNN which is based off a version of (2, 2)-FWL+, and outperforms state-of-the-art models on standard and synthetic tasks alike.\n\nSTRENGTHS:\nThe paper's strengths lie in the authors' extensions of FWL, Neighborhood^2-FWL (N^2-FWL), and its strong empirical performance with regards to baselines.\n\nThe new formulation of (k, t)-FWL+ allows researchers to easily derive new extensions of FWL algorithms (and lead to new GNN models as a result). N^2-FWL is an empirical demonstration of the power of (k, t)-FWL+, showing that even with O(n^2) space complexity, these algorithms can be highly expressive, which is supported with multiple experiments.\n\nWEAKNESSES:\nThe paper's weaknesses lie in its lack of clarity. Especially as a rather technical paper, there seems to be a lack of intuition-based explanations and motivations for a large portion of the paper, which makes it hard to follow at points. More figures may help as well.\n\nIn addition, the authors' do not mention any details about time in the main paper (only in the Appendix), with a time complexity of $O(nd^{h+2})$, which they claim to be practical for real-world tasks. It would be nice to see empirical run-time, as well as memory consumption charts/tables which would help solidify the empirical contribution. \n\nTypo on line 181.\n\nQUESTIONS:\n1. How was N^2-FWL motivated? Why was it chosen over other possible (k, t)-WFL+ configurations? It seems as if it is not the only configuration that gives such properties.\n2. How does the runtime and memory consumption compare to node-based GNN models? And to traditional edge-based subgraph models?\n\nLIMITATIONS:\nSome limitations are addressed, but only in the Appendix. Most points are addressed in the section in the appendix, but would prefer if it were in the main paper.\n\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with supervised learning on graphs. Specifically, it proposes a more fine-grained version of the (folklore) k-WL (k-FWL) hierarchy. In turn, these variants are neuralized via standard techniques. \n\nSpecifically, the authors propose the $(k,t)$-FWL, which extends the $k$-FWL by the parameter $t$, which controls which tuples are used for refining the color of a tuple. That is, following Eq. (6), the authors consider $t$-tuples to control the cardinality of the neighborhood of a given $k$-tuple. Subsequently, building on **standard constructions** from the $k$-WL literature, they investigate how, for a fixed $k$, varying $t$ controls the expressivity in terms of distinguishing non-isomorphic graphs (Prop. 3.1 to 3.3).\n\nThey further propose a simple extension of the above, the $k$-FWL, which uses a relaxed version of the neighborhood between a given $k$-tuple and a vertex in the underlying graph. Furthermore, they show that some recent GNN enhancements are subsumed by their framework.\n\nThe theory is complemented by an experimental study showing that the neuralized variants exhibit SOTA predictive performance on two common benchmark datasets.\n\nSTRENGTHS:\n- The authors report good experimental results on established benchmark datasets. The experimental protocol looks meaningful.\n\nWEAKNESSES:\n- The paper lacks clarity. For example, the exact definition of $Q^F_w(v)$ (line 136ff) is unclear.  It is a central concept introduced in the paper, and it should be properly formalized. Moreover, the definitions in Section 3.3 are not entirely clear and make it hard to verify the formal proofs. \n\n- The authors seem to **oversell their contribution**, e.g.,\n\n  - The discussion on the benefits of the k-FWL over the k-WL (lines 119 to 135) is folklore. There is no new insight. This also **holds for the \n  quadratic complexity of their algorithm** (e.g., lines 60 to 67). Note that big O notation is wrongly used here. Big Omega should be appropriate here.\n  - Subsection 3.1 just contains a simple extension of established ideas, e.g., from [8,12].\n  - Results in Section 4.2 are obvious by standard results from the GNN literature\n  - ...\n\n- The running time and memory complexity of the algorithms are **not formally proved**. This seems crucial for the present work. Also the authors seem to confuse big O with big Omega notation. \n- Relevant related work such as https://arxiv.org/abs/2203.13913 seems missing. \n\nComments:\n- It is not correct saying that the $k$-WL requires $\\mathcal{O}(n^k)$ space, it is a **lower** bound not an upper bound.\n\nQUESTIONS:\n- What is the definition of \"hierarchical multiset\" (line 145)?\n- Can you quantify the h in Theorem 4.1?\n- What is required h for the statement of Theorem 4.3\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors start by considering the k-dim Weisfeiler-Lehman test (WL). To solve the well-known problems of k-WL/FWL (high space complexity $O(n^k)$ and rigid design space), the authors propose two extensions of the k-FWL, named (k, t)-FWL and k-FWL+.\nThe first extension is based on the simple observation that k-WL aggregates each color separately while k-FWL aggregates the tuple of colors. Expressive results of (k,t)-FWL are provided (3.1 - 3.3). The second extension is based on the simple modification of k-FWL; instead of considering the whole vertex set, consider only a particular neighbor. These two extensions are then combined to (k, t)-FWL+, of which expressiveness results are provided compared to the previous gallery of GNN/WL models (3.5 - 3.9). Lastly, the authors then propose N2-GNN, an implementation of (2, 2)-FWL+ as well as empirical results showing the efficacy of N2-GNN.\n\nSTRENGTHS:\n- Despite the complex nature of the concepts, the exposition of their main algorithm and the conceptual design was very well done (especially Figure 1)\n\n- I very much appreciate the simplicity of the idea that leads to space-efficient variants of (F)WL tests which are still quite powerful.\n\n- Although I did not check the proofs in detail, the authors provide an impressive list of theoretical results regarding their proposed (k, t)-FWL+, all of which provide solid theoretical superiority over previous works.\n\n- The experimental results seem promising.\n\nWEAKNESSES:\n- Although the concepts are well delivered, all technical details, especially the proofs, are entirely relegated to the Appendix. Are there any notable technical novelties in the proofs of the myriad of theoretical results, or are they somewhat standard?\n\n- It is expected that the expressive power of (k, t)-FWL+ depends heavily on the definition of ES, which the authors addressed via a myriad of propositions (3.4 - 3.9) for specific instances. However, if possible, it would be great to have some intuitions on the interplay of the choices of (k, t) and ES. Maybe the authors could choose a single proposition and try to explain the intuition behind it further (e.g., Proposition 3.4) via additional figures or discussions.\n\n- Some of the references are in poor condition (e.g., [11], [12], [16]).\n\nQUESTIONS:\n1. It states on pg. 4 lines 159-160, that \"size of Q_wF(w) will grow exponentially with an increase in the size of t\". Then does that mean that requiring O(n^2) is equivalent to confining t=2 (or at least small t)? Also, throughout the paper, the only time t is not 2 is in the theories when the authors prove a hierarchy w.r.t. t. Are there situations in which t should be set to 3 or higher?\n\n2. Is there any way of comparing the expressiveness of (k, t)-FWL and (k', t')-FWL with k+t = k'+t'? For instance, can one compare the expressiveness of (2, 3)-FWL and (3, 2)-FWL?\n\n3. Before combining (k,t)-FWL and FWL+, is it possible to also obtain expressiveness hierarchy results on FLW+ depending on the choice of ES? For instance, with fixed t=1 and k=2, can you say anything about the expressiveness across the choices of ES(v) as considered in Proposition 3.4 - 3.9?\n\n4. It is mentioned in the conclusions that the practical complexity of N2-GNN can still be unbearable, especially for dense graphs. Thus I'm curious about the running times of all the algorithms in comparison. Is there a trade-off if one sets the training budget approximately the same for all the algorithms?\n\n5. There was a mention of Puny et al. [43] that performs O(n^2) space 3-WL via graph polynomial features. Although the authors mentioned that they suffer from overfitting, I'm still curious. Are there any theoretical results (e.g., expressiveness hierarchy) or empirical results that compare [43] to (k, t)-FWL+?\n\n(minor suggestion: it would be nice to have a table of contents, maybe at the start of the appendix, to make it easier to navigate)\n\nLIMITATIONS:\nAlthough this work needs not address any societal impact, the work has no mention of its limitations and possible future works. I would like to ask the authors to include some of the possible limitations of their work (possibly with possible solutions, but I won't be negative if there are none for now, which by the way is also quite natural in my eyes.)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}