Title: Distance-Restricted Folklore Weisfeiler-Leman GNNs with Provable Cycle Counting Power
Venue: NeurIPS 2023
Accepted: yes
Keywords: Cycle counting

graph neural networks
Abstract: The ability of graph neural networks (GNNs) to count certain graph substructures, especially cycles, is important for the success of GNNs on a wide range of tasks. It has been recently used as a popular metric for evaluating the expressive power of GNNs. Many of the proposed GNN models with provable cycle counting power are based on subgraph GNNs, i.e., extracting a bag of subgraphs from the input graph, generating representations for each subgraph, and using them to augment the representation of the input graph. However, those methods require heavy preprocessing, and suffer from high time and memory costs. In this paper, we overcome the aforementioned limitations of subgraph GNNs by proposing a novel class of GNNs---$d$-Distance-Restricted FWL(2) GNNs, or $d$-DRFWL(2) GNNs, based on the well-known FWL(2) algorithm. As a heuristic method for graph isomorphism testing, FWL(2) colors all node pairs in a graph and performs message passing among those node pairs. In order to balance the expressive power and complexity, $d$-DRFWL(2) GNNs simplify FWL(2) by restricting the range of message passing to node pairs whose mutual distances are at most $d$. This way, $d$-DRFWL(2) GNNs exploit graph sparsity while avoiding the expensive subgraph extraction operations in subgraph GNNs, making both the time and space complexity lower. We theoretically investigate both the discriminative power and the cycle counting power of $d$-DRFWL(2) GNNs. Our most important finding is that $d$-DRFWL(2) GNNs have provably strong cycle counting power even with $d=2$: they can count all 3, 4, 5, 6-cycles. Since 6-cycles (e.g., benzene rings) are ubiquitous in organic molecules, being able to detect and count them is crucial for achieving robust and generalizable performance on molecular tasks. Experiments on both synthetic datasets and molecular datasets verify our theory. To the best of our knowledge, 2-DRFWL(2) GNN is the most efficient GNN model to date (both theoretically and empirically) that can count up to 6-cycles.

Distance-Restricted Folklore Weisfeiler-Leman GNNs with Provable Cycle Counting Power

Abstract

The ability of graph neural networks (GNNs) to count certain graph substructures, especially cycles, is important for the success of GNNs on a wide range of tasks. It has been recently used as a popular metric for evaluating the expressive power of GNNs. Many of the proposed GNN models with provable cycle counting power are based on subgraph GNNs, i.e., extracting a bag of subgraphs from the input graph, generating representations for each subgraph, and using them to augment the representation of the input graph. However, those methods require heavy preprocessing, and suffer from high time and memory costs. In this paper, we overcome the aforementioned limitations of subgraph GNNs by proposing a novel class of GNNsd -Distance-Restricted FWL(2) GNNs, or d -DRFWL(2) GNNs, based on the well-known FWL(2) algorithm. As a heuristic method for graph isomorphism testing, FWL(2) colors all node pairs in a graph and performs message passing among those node pairs. In order to balance the expressive power and complexity, d -DRFWL(2) GNNs simplify FWL(2) by restricting the range of message passing to node pairs whose mutual distances are at most d . This way, d -DRFWL(2) GNNs exploit graph sparsity while avoiding the expensive subgraph extraction operations in subgraph GNNs, making both the time and space complexity lower. We theoretically investigate both the discriminative power and the cycle counting power of d -DRFWL(2) GNNs. Our most important finding is that d -DRFWL(2) GNNs have provably strong cycle counting power even with d = 2 : they can count all 3, 4, 5, 6-cycles. Since 6-cycles (e.g., benzene rings) are ubiquitous in organic molecules, being able to detect and count them is crucial for achieving robust and generalizable performance on molecular tasks. Experiments on both synthetic datasets and molecular datasets verify our theory. To the best of our knowledge, 2-DRFWL(2) GNN is the most efficient GNN model to date (both theoretically and empirically) that can count up to 6-cycles.

1 Introduction

Graphs are important data structures suitable for representing relational or structural data. As a powerful tool to learn node-level, link-level or graph-level representations for graph-structured data, graph neural networks (GNNs) have achieved remarkable successes on a wide range of tasks [54, 61]. Among the various GNN models, Message Passing Neural Networks (MPNNs) [27, 36, 50, 55] are a widely adopted class of GNNs. However, the expressive power of MPNNs has been shown to be upper-bounded by the Weisfeiler-Leman test [55]. More importantly, MPNNs even fail to detect some simple substructures (e.g., 3, 4-cycles) [33], thus losing great structural information in graphs.

The weak expressive power of MPNNs has aroused a search for more powerful GNN models. To evaluate the expressive power of these GNN models, there are usually two perspectives. One is to characterize their discriminative power, i.e., the ability to distinguish between non-isomorphic graphs. It is shown in [14] that an equivalence exists between distinguishing all non-isomorphic graph pairs and approximating any permutation-invariant functions on graphs. Although the discriminative power partially reveals the function approximation ability of a GNN model, it fails to tell what specific functions a model is able to approximate. Another perspective is to directly characterize what specific function classes a model can approximate. In particular, we can study the expressive power of a GNN model by asking what graph substructures it can count . This perspective is often more practical since the task of counting substructures (especially cycles) is closely related to a variety of domains such as chemistry [18, 35], biology [37], and social network analysis [34].

Despite the importance of counting cycles, the task gets increasingly difficult as the length of cycle increases [2].To have an intuitive understanding of this difficulty, we first discuss why MPNNs fail to count any cycles. The reason is clear: MPNNs only keep track of the rooted subtree around each node [55], and without node identity, any neighboring node v of u has no idea which of its neighbors are also neighbors of u . Therefore, MPNNs cannot count even the simplest 3-cycles. In contrast, the 2-dimensional Folklore Weisfeiler-Leman, or FWL(2) test [12], uses 2-tuples of nodes instead of nodes as the basic units for message passing, and thus natually encodes closed walks . For example, when a 2-tuple ( u, v ) receives messages from 2-tuples ( w,v ) and ( u, w ) in FWL(2), ( u, v ) gets aware of the walk u → w → v → u , whose length is d ( u, w ) + d ( w,v ) + d ( u, v ) =: l . As long as the intermediate nodes do not overlap, we immediately get an l -cycle passing both u and v . In fact, FWL(2) can provably count up to 7-cycles [5, 25].

However, FWL(2) has a space and time complexity of O ( n 2 ) and O ( n 3 ) , which makes it impractical for large graphs and limits its real-world applications. Inspired by the above discussion, we naturally raise a question: to what extent can we retain the cycle counting power of FWL(2) while reducing the time and space complexity? Answering this question requires another observation: cycle counting tasks are intrinsically local . For example, the number of 6-cycles that pass a given node u has nothing to do with the nodes that have a shortest-path distance ⩾ 4 to u . Therefore, if we only record the embeddings of node pairs with mutual distance ⩽ d in FWL(2), where d is a fixed positive integer, then the ability of FWL(2) to count substructures with diameter ⩽ d should be retained. Moreover, given that most real-world graphs are sufficiently sparse, this simplified algorithm runs with complexities much lower than FWL(2), since we only need to store and update a small portion of the n 2 embeddings for all 2-tuples. We call this simplified version d -Distance-Restricted FWL(2) , or d -DRFWL(2) , reflecting that only 2-tuples with restricted distances get updated.

Main contributions. Our main contributions are summarized as follows:

We propose d -DRFWL(2) tests as well as their neural versions, d -DRFWL(2) GNNs. We study how the hyperparameter d affects the discriminative power of d -DRFWL(2), and show that a strict expressiveness hierarchy exists for d -DRFWL(2) GNNs with increasing d , both theoretically and experimentally.

We study the cycle counting power of d -DRFWL(2) GNNs. Our major results include:

2-DRFWL(2) GNNs can already count up to 6-cycles, covering common structures like benzene rings in organic chemistry. The time and space complexities of 2-DRFWL(2) are O ( n deg 4 ) and O ( n deg 2 ) respectively, making it the most efficient one to date among other models with 6-cycle counting power such as I 2 -GNN [33].

With d ⩾ 3 , d -DRFWL(2) GNNs can count up to 7-cycles, fully retaining the cycle counting power of FWL(2) but with complexities strictly lower than FWL(2). This finding also confirms the existence of GNNs with the same cycle counting power as FWL(2), but strictly weaker discriminative power.

We compare the performance and empirical efficiency of d -DRFWL(2) GNNs (especially for d = 2 ) with other state-of-the-art GNNs on both synthetic and real-world datasets. The results verify our theory on the counting power of d -DRFWL(2) GNNs. Additionally, for the case of d = 2 , the amount of GPU memory required for training 2-DRFWL(2) GNNs is greatly reduced compared with subgraph GNNs like ID-GNN [57], NGNN [59] and I 2 -GNN [33]; the preprocessing time and training time of 2-DRFWL(2) GNNs are also much less than I 2 -GNN.

2 Preliminaries

2.1 Notations

For a simple, undirected graph G , we use V G and E G to denote its node set and edge set respectively. For every node v ∈ V G , we define its k -th hop neighbors as N k ( v ) = { u ∈ V G : d ( u, v ) = k } , where d ( u, v ) is the shortest-path distance between nodes u and v . We further introduce the symbols N ⩽ k ( v ) = ⋃ k N i ( v ) and N ( v ) = N 1 ( v ) . For any n ∈ N , we denote [ n ] = { 1 , 2 , . . . , n } .

̸

̸

An ℓ -cycle ( ℓ ⩾ 3) in the (simple, undirected) graph G is a sequence of ℓ edges { v 1 , v 2 } , { v 2 , v 3 } , . . . , { v ℓ , v 1 } ∈ E G with v i = v j for any i = j and i, j ∈ [ ℓ ] . The ℓ -cycle is said to pass a node v if v is among the nodes { v i : i ∈ [ ℓ ] } . An ℓ -path is a sequence of ℓ edges { v 1 , v 2 } , { v 2 , v 3 } , . . . , { v ℓ , v ℓ +1 } ∈ E G with v i = v j for any i = j and i, j ∈ [ ℓ + 1] , and it is said to start at node v 1 and end at node v ℓ +1 . An ℓ -walk from v 1 to v ℓ +1 is a sequence of ℓ edges { v 1 , v 2 } , { v 2 , v 3 } , . . . , { v ℓ , v ℓ +1 } ∈ E G but the nodes v 1 , v 2 , . . . , v ℓ +1 can coincide. An ℓ -clique is ℓ nodes v 1 , . . . , v ℓ such that { v i , v j } ∈ E G for all i = j and i, j ∈ [ ℓ ] .

̸

Let G be the set of all simple, undirected graphs. If S is a graph substructure on G , and G ∈ G is a graph, we use C ( S, G ) to denote the number of inequivalent substructures S that occur as subgraphs of G . Similarly, if u is a node of G , we use C ( S, u, G ) to denote the number of inequivalent substructures S that pass node u and occur as subgraphs of G .

Following [15, 33], we give the definition of whether a function class F on graphs can count a certain substructure S .

̸

Definition 2.1 (Graph-level count) . Let F graph be a function class on G , i.e., f graph : G → R for all f graph ∈ F graph . F graph is said to be able to graph-level count a substructure S on G if for ∀ G 1 , G 2 ∈ G such that C ( S, G 1 ) = C ( S, G 2 ) , there exists f graph ∈ F graph such that f graph ( G 1 ) = f graph ( G 2 ) .

̸

̸

Definition 2.2 (Node-level count) . Let G × V = { ( G,u ) : G ∈ G , u ∈ V G } . Let F node be a function class on G × V , i.e., f node : G × V → R for all f node ∈ F node . F node is said to be able to node-level count a substructure S on G if for ∀ ( G 1 , u 1 ) , ( G 2 , u 2 ) ∈ G × V such that C ( S, u 1 , G 1 ) = C ( S, u 2 , G 2 ) , there exists f node ∈ F node such that f node ( G 1 , u 1 ) = f node ( G 2 , u 2 ) .

̸

Notice that C ( S, G ) can be calculated by ∑ u ∈V G C ( S, u, G ) divided by a factor only depending on S , for any given substructure S . (For example, for triangles the factor is 3.) Therefore, counting a substructure at node level is harder than counting it at graph level.

2.2 FWL( k ) graph isomorphism tests

WL(1) test. The 1-dimensional Weisfeiler-Leman test, or WL(1) test, is a heuristic algorithm for the graph isomorphism problem [52]. For a graph G , the WL(1) test iteratively assigns a color W ( v ) to every node v ∈ V G . At the 0-th iteration, the color W (0) ( v ) is identical for every node v . At the t -th iteration with t ⩾ 1 ,

where HASH ( t ) and POOL ( t ) are injective hashing functions, and { {·} } means a multiset (set with potentially identical elements). The algorithm stops when the node colors become stable, i.e., ∀ v, u ∈ V G , W ( t +1) ( v ) = W ( t +1) ( v ) ⇔ W ( t ) ( v ) = W ( t ) ( u ) . We denote the stable coloring of node v as W ( ∞ ) ( v ) ; then the representation for graph G is

where READOUT is an arbitrary injective multiset function.

FWL( k ) tests. For k ⩾ 2 , the k -dimensional Folklore Weisfeiler-Leman tests, or FWL( k ) tests, define a hierarchy of algorithms for graph isomorphism testing, as described in [12, 32, 42]. For a graph G , the FWL( k ) test assigns a color W ( v ) for every k -tuple v = ( v 1 , . . . , v k ) ∈ V k G . At the 0-th iteration, the color W (0) ( v ) is the atomic type of v , denoted as atp( v ) . If we denote the

̸

̸

subgraphs of G induced by v and v ′ as G ( v ) and G ( v ′ ) respectively, then the function atp( · ) can be any function on V k G that satisfies the following condition: atp( v ) = atp( v ′ ) iff the mapping v i ↦→ v ′ i , i ∈ [ k ] (i.e., the mapping that maps each v i to its corresponding v ′ i , for i ∈ [ k ] ) induces an isomorphism from G ( v ) to G ( v ′ ) .

At the t -th iteration with t ⩾ 1 , FWL( k ) updates the color of v ∈ V k G as

where sift( f, v , w ) is defined as

Here we use v [ j → w ] to denote ( v 1 , . . . , v j -1 , w, v j +1 , . . . , v k ) for j ∈ [ k ] and w ∈ V G .

HASH ( t ) and POOL ( t ) are again injective hashing functions. The algorithm stops when all k -tuples receive stable colorings. The stable coloring of v ∈ V k G is denoted as W ( ∞ ) ( v ) . We then calculate the representation for graph G as

where READOUT is an arbitrary injective multiset function.

3 d -Distance-Restricted FWL(2) GNNs

In this section, we propose the d -Distance Restricted FWL(2) tests/GNNs. They use 2-tuples like FWL(2), but restrict the distance between nodes in each 2-tuple to be ⩽ d , which effectively reduces the number of 2-tuples to store and aggregate while still retaining great cycle counting power.

3.1 d -DRFWL(2) tests

We call a 2-tuple ( u, v ) ∈ V 2 G a distancek tuple if the shortest-path distance between u and v is k in the following. d -Distance Restricted FWL(2) tests , or d -DRFWL(2) tests , assign a color W ( u, v ) for every distancek tuple ( u, v ) with 0 ⩽ k ⩽ d . Initially, the color W (0) ( u, v ) only depends on d ( u, v ) . For the t -th iteration with t ⩾ 1 , d -DRFWL(2) updates the colors using the following rule,

where HASH ( t ) k is an injective hashing function for distance k and iteration t , and M k ( t ) ij ( u, v ) is defined as

The symbol ( M k ( t ) ij ( u, v ) ) 0 ⩽ i,j ⩽ d stands for ( M k ( t ) 00 ( u, v ) , M k ( t ) 01 ( u, v ) , . . . , M k ( t ) 0 d ( u, v ) , . . . , M k ( t ) dd ( u, v ) ) . Each of the POOL k ( t ) ij with 0 ⩽ i, j, k ⩽ d is an injective multiset hashing function. Briefly speaking, the rules (6) and (7) update the color of a distancek tuple ( u, v ) using colors of distancei and distancej tuples .

When every distancek tuple ( u, v ) with 0 ⩽ k ⩽ d receives its stable coloring, denoted as W ( ∞ ) ( u, v ) , the representation of G is calculated as

Figure 1 illustrates how 2-DRFWL(2) updates the color of a distance-2 tuple ( u, v ) . Since only distancek tuples with 0 ⩽ k ⩽ 2 are colored in 2-DRFWL(2) tests, there are 7 terms of the form (( W ( w,v ) , W ( u, w )) (with w ∈ { u, v, x, y, z, t, r } ) contributing to the update of W ( u, v ) . The 7 nodes u, v, x, y, z, t, r are filled with different colors, according to their distances to u and to v . For example, the violet node u has distance 0 to u and distance 2 to v , thus contributing to M 2 02 ( u, v ) ; the green nodes x and y have distance 1 to either u or v , thus contributing to M 2 11 ( u, v ) . Analogously, nodes with red, blue, orange and pink colors contribute to M 2 20 ( u, v ) , M 2 12 ( u, v ) , M 2 21 ( u, v ) and M 2 22 ( u, v ) , respectively. Finally, the uncolored nodes q and s do not contribute to the update of W ( u, v ) , since

t

v

s

q

r

z

x

y

u

1

1

1

1

1

1

0

2

2

0

2

2

2

2

they have distance 3 (which is greater than 2) to u . From the figure, we can observe that by redefining neighbors and sparsifying 2-tuples of FWL(2), d -DRFWL(2) significantly reduces the complexity and focuses only on local structures, especially on sparse graphs.

Now we study the expressive power of d -DRFWL(2) tests by comparing them with the WL hierarchy. First, we can prove that d -DRFWL(2) tests are strictly more powerful than WL(1), for every d .

Theorem 3.1. In terms of the ability to distinguish between non-isomorphic graphs, the d -DRFWL(2) test is strictly more powerful than WL(1), for any d ⩾ 1 .

Next, we compare d -DRFWL(2) tests with FWL(2). Actually, since it is easy for the FWL(2) test to compute distance between every pair of nodes (we can initialize W ( u, v ) as 0 , 1 and ∞ for u = v , ( u, v ) ∈ E G and all other cases, and iteratively update W ( u, v ) with min { W ( u, v ) , min w { W ( u, w )+ W ( w,v ) }} ), the FWL(2) test can use its update rule to simulate (6) and (7) by applying different HASH ( t ) k and POOL k ( t ) ij functions to different ( i, j, k ) values. This implies that d -DRFWL(2) tests are at most as powerful as the FWL(2) test . Actually, this hierarchy in expressiveness is strict, due to the following theorem.

Theorem 3.2. In terms of the ability to distinguish between non-isomorphic graphs, FWL(2) is strictly more powerful than d -DRFWL(2), for any d ⩾ 1 . Moreover, ( d +1) -DRFWL(2) is strictly more powerful than d -DRFWL(2).

The proofs of all theorems within this section are included in Appendix B.

3.2 d -DRFWL(2) GNNs

Based on the d -DRFWL(2) tests, we now propose d -DRFWL(2) GNNs. Let G be a graph which can have node features f v ∈ R d f , v ∈ V G and (or) edge features e uv ∈ R d e , { u, v } ∈ E G . A d -DRFWL(2) GNN is defined as a function of the form

The input of f is the initial labeling h (0) uv , 0 ⩽ d ( u, v ) ⩽ d . Each L t with t = 1 , 2 , . . . , T in (9) is called a d -DRFWL(2) GNN layer , which transforms h ( t -1) uv into h ( t ) uv using rules

For each k = 0 , 1 , . . . , d,

For each ( u, v ) ∈ V 2 G with d ( u, v ) = k,

where m ( t ) ijk and f ( t ) k are arbitrary learnable functions; ⊕ denotes a permutation-invariant aggregation operator (e.g., sum, mean or max). (10) and (11) are simply counterparts of (7) and (6) with POOL k ( t ) ij and HASH ( t ) k replaced with continuous functions. σ t , t = 1 , . . . , T -1 are entry-wise activation functions. R is a permutation-invariant readout function, whose input is the multiset

{ { h ( T ) uv : ( u, v ) ∈ V 2 G and 0 ⩽ d ( u, v ) ⩽ d } } . Finally, M is an MLP that acts on the graph representation output by R .

We can prove that (i) the representation power of any d -DRFWL(2) GNN is upper-bounded by the d -DRFWL(2) test, and (ii) there exists a d -DRFWL(2) GNN instance that has equal representation power to the d -DRFWL(2) test. We leave the formal statement and proof to Appendix B.

4 The cycle counting power of d -DRFWL(2) GNNs

By Theorem 3.2, the expressive power of d -DRFWL(2) tests (or d -DRFWL(2) GNNs) strictly increases with d . However, the space and time complexities of d -DRFWL(2) GNNs also increase with d . On one hand, since there are O ( n deg k ) distancek tuples in a graph G , at least O ( n deg k ) space is necessary to store the representations for all distancek tuples. For d -DRFWL(2) GNNs, this results in a space complexity of O ( n deg d ) . On the other hand, since there are at most O (deg min { i,j } ) nodes in N i ( u ) ∩N j ( v ) , ∀ 0 ⩽ i, j ⩽ d , there are at most O (deg d ) terms at the RHS of (10). Therefore, it takes O ( d 2 deg d ) time to update a single representation vector h ( t ) uv using (10) and (11). This implies the time complexity of d -DRFWL(2) GNNs is O ( nd 2 deg 2 d ) .

For scalability, d -DRFWL(2) GNNs with a relatively small value of d are used in practice. But how to find the d value that best strikes a balance between expressive power and efficiency? To answer this question, we need a practical, quantitative metric of expressive power. In the following, we characterize the cycle counting power of d -DRFWL(2) GNNs. We find that 2-DRFWL(2) GNNs are powerful enough to node-level count up to 6-cycles , as well as many other useful graph substructures . Since for d = 2 , the time and space complexities of d -DRFWL(2) GNNs are O ( n deg 4 ) and O ( n deg 2 ) respectively, our model is much more efficient than I 2 -GNN which requires O ( n deg 5 ) time and O ( n deg 4 ) space to count up to 6-cycles [33]. Moreover, with d ⩾ 3 , d -DRFWL(2) GNNs are able to node-level count up to 7-cycles , already matching the cycle counting power of FWL(2).

Our main results are stated in Theorems 4.3-4.8. Before we present the theorems, we need to give revised definitions of C ( S, u, G ) for some certain substructures S . This is because in those substructures not all nodes are structurally equal.

Definition 4.1. If S is an ℓ -path with ℓ ⩾ 2 , C ( S, u, G ) is defined to be the number of ℓ -paths in G starting from node u .

Figure 2a illustrates Definition 4.1 for the ℓ = 4 case.

Definition 4.2. The substructures in Figures 2b, 2c and 2d are called tailed triangles , chordal cycles and triangle-rectangles , respectively. If S is a tailed triangle (or chordal cycle or triangle-rectangle), C ( S, u, G ) is defined to be the number of tailed triangles (or chordal cycles or triangle-rectangles) that occur as subgraphs of G and include node u at a position shown in the figures.

u

(a) A 4-path that starts at node u

u

(b) A tailed triangle that passes node u

u

(c) A chordal cycle that passes node

u

u

Now we state our main theorems. In the following, the definition for 'node-level counting' is the same as Definition 2.2, but one should treat C ( S, u, G ) differently (following Definitions 4.1 and 4.2) when S is a path, a tailed triangle, a chordal cycle, or a triangle-rectangle. To define the output of a d -DRFWL(2) GNN f on domain G × V , we denote

i.e., we treat the embedding of ( u, u ) as the representation of node u . For 1-DRFWL(2) GNNs, we have

Theorem 4.3. 1-DRFWL(2) GNNs can node-level count 3-cycles, but cannot graph-level count any longer cycles.

For 2-DRFWL(2) GNNs, we investigate not only their cycle counting power, but also their ability to count many other graph substructures. Our results include

Theorem 4.4. 2-DRFWL(2) GNNs can node-level count 2, 3, 4-paths.

Theorem 4.5. 2-DRFWL(2) GNNs can node-level count 3, 4, 5, 6-cycles.

Theorem 4.6. 2-DRFWL(2) GNNs can node-level count tailed triangles, chordal cycles and trianglerectangles.

Theorem 4.7. 2-DRFWL(2) GNNs cannot graph-level count k -cycles with k ⩾ 7 or k -cliques with k ⩾ 4 .

For d -DRFWL(2) GNNs with d ⩾ 3 , we have

Theorem 4.8. For any d ⩾ 3 , d -DRFWL(2) GNNs can node-level count 3, 4, 5, 6, 7-cycles, but cannot graph-level count any longer cycles.

The proofs of all theorems within this section are included in Appendix C. To give an intuitive explanation for the cycle counting power of d -DRFWL(2) GNNs, let us consider, e.g., why 2DRFWL(2) GNNs can count up to 6-cycles. The key reason is that they allow a distance-2 tuple ( u, v ) to receive messages from other two distance-2 tuples ( u, w ) and ( w,v ) , and are thus aware of closed 6-walks (since 6 = 2+2+2 ). Indeed, if we forbid such kind of message passing, the modified 2-DRFWL(2) GNNs can no longer count 6-cycles, as experimentally verified in Appendix F.2.

5 Related works

The cycle counting power of GNNs. It is proposed in [33] to use GNNs' ability to count given-length cycles as a metric for their expressiveness. Prior to this work, Arvind et al. [5] and Fürer [25] have discussed the cycle counting power of the FWL(2) test: FWL(2) can and only can graph-level count up to 7-cycles. Huang et al. [33] also characterizes the node-level cycle counting power of subgraph MPNNs and I 2 -GNN. Apart from counting cycles, there are also some works analyzing the general substructure counting power of GNNs. Chen et al. [15] discusses the ability of WL( k ) tests to count general subgraphs or induced subgraphs, but the result is loose. Tahmasebi et al. [48] analyzes the substructure counting power of Recursive Neighborhood Pooling, which can be seen as a subgraph GNN with recursive subgraph extraction procedures.

The trade-off between expressive power and efficiency of GNNs. Numerous methods have been proposed to boost the expressive power of MPNNs. Many of the provably powerful GNN models have direct correspondence to the Weisfeiler-Leman hierarchy [12, 28], such as higher-order GNNs [44] and IGNs [41-43]. Despite their simplicity in theory, those models require O ( n k +1 ) time and O ( n k ) space in order to achieve equal expressive power to FWL( k ) tests, and thus do not scale to large graphs even for k = 2 .

Recent works try to strike a balance between the expressive power of GNNs and their efficiency. Among the state-of-the-art GNN models with subO ( n 3 ) time complexity, subgraph GNNs have gained much research interest [8, 17, 24, 33, 39, 46, 57-60]. Subgraph GNNs process a graph G by 1) extracting a bag of subgraphs { G i : i = 1 , 2 , . . . , p } from G , 2) generating representations for every subgraph G i , i = 1 , 2 , . . . , p (often using a weak GNN such as MPNN), and 3) combining the representations of all subgraphs into a representation of G . Most commonly, the number p of subgraphs extracted is equal to the number of nodes n (called a node-based subgraph extraction policy [24]). In this case, the time complexity of subgraph GNNs is upper-bounded by O ( nm ) , where m is the number of edges. If we further adopt the K -hop ego-network policy , i.e., extracting a K -hop subgraph G u around each node u , the time complexity becomes O ( n deg K +1 ) . Frasca et al. [24] and Zhang et al. [58] theoretically characterize the expressive power of subgraph GNNs, and prove that subgraph GNNs with node-based subgraph extraction policies lie strictly between WL(1) and FWL(2) in the Weisfeiler-Leman hierarchy. Despite the lower complexity, in practice subgraph GNNs still suffer from heavy preprocessing and a high GPU memory usage.

Apart from subgraph GNNs, there are also attempts to add sparsity to higher-order GNNs. Morris et al. [45] proposes δ -k -LWL, a variant of the WL( k ) test that updates a k -tuple u only from the k -tuples with a component connected to the corresponding component in u . Zhang et al. [58] proposes LFWL(2) and SLFWL(2), which are variants of FWL(2) that update a 2-tuple ( u, v ) from nodes in either N ( v ) or N ( u ) ∪ N ( v ) . Our model can also be classified into this type of approaches, yet we not only sparsify neighbors of a 2-tuple, but also 2-tuples used in message passing, which results in much lower space complexity. A detailed comparison between our method and LFWL(2)/SLFWL(2) is included in Appendix D.

6 Experiments

In this section, we empirically evaluate the performance of d -DRFWL(2) GNNs (especially for the case of d = 2 ) and verify our theoretical results. To be specific, we focus on the following questions:

Q1: Can d -DRFWL(2) GNNs reach their theoretical counting power as stated in Theorems 4.3-4.8?

Q2: How do d -DRFWL(2) GNNs perform compared with other state-of-the-art GNN models on open benchmarks for graphs?

Q3: What are the time and memory costs of d -DRFWL(2) GNNs on various datasets?

Q4: Do d -DRFWL(2) GNNs with increasing d values construct a hierarchy in discriminative power (as shown in Theorem 3.2)? Further, does this hierarchy lie between WL(1) and FWL(2) empirically?

Weanswer Q1 -Q3 in 6.1-6.3, as well as in Appendix F. The answer to Q4 is included in Appendix F.1. The details of our implementation of d -DRFWL(2) GNNs, along with the experimental settings, are included in Appendix E. Our code for all experiments, including those in Section 6 of the main paper and in Appendix F, is available at https://github.com/zml72062/DR-FWL-2 .

6.1 Substructure counting

Datasets. To answer Q1 , we perform node-level substructure counting tasks on the synthetic dataset in [33, 60]. The synthetic dataset contains 5,000 random graphs, and the training/validation/test splitting is 0.3/0.2/0.5. The task is to perform regression on the node-level counts of certain substructures. Normalized MAE is used as the evaluation metric.

Tasks and baselines. To verify Theorems 4.4-4.6, we use 2-DRFWL(2) GNN to perform nodelevel counting task on 9 different substructures: 3-cycles, 4-cycles, 5-cycles, 6-cycles, tailed triangles, chordal cycles, 4-cliques, 4-paths and triangle-rectangles. We choose MPNN, node-based subgraph GNNs (ID-GNN, NGNN, GNNAK+), PPGN, and I 2 -GNN as our baselines. Results for all baselines are from [33].

To verify Theorem 4.8, we compare the performances of 2-DRFWL(2) GNN and 3-DRFWL(2) GNN on node-level cycle counting tasks, with the cycle length ranging from 3 to 7.

We also conduct ablation studies to investigate what kinds of message passing are essential (among those depicted in Figure 1) for 2-DRFWL(2) GNNs to successfully count up to 6-cycles and other substructures. The experimental details and results are given in Appendix F.2. The studies also serve as a verification of Theorem 4.3.

Results. From Table 1, we see that 2-DRFWL(2) GNN achieves less-than-0.01 normalized MAE on all 3, 4, 5 and 6-cycles, verifying Theorem 4.5; 2-DRFWL(2) GNN also achieves less-than0.01 normalized MAE on tailed triangles, chordal cycles, 4-paths and triangle-rectangles, verifying Theorems 4.4 and 4.6.

It is interesting that 2-DRFWL(2) GNN has a very good performance on the task of node-level counting 4-cliques, which by Theorem 4.7 it cannot count in theory. A similar phenomenon happens for subgraph GNNs. This may be because 2-DRFWL(2) GNN and subgraph GNNs still learn some local structural biases that have strong correlation with the number of 4-cliques.

From Table 2, we see that 3-DRFWL(2) GNN achieves a comparable performance to 2-DRFWL(2) GNN on the tasks of counting 3, 4, 5 and 6-cycles. Yet when it comes to counting 7-cycles, 3-DRFWL(2) GNN greatly outperforms 2-DRFWL(2) GNN, verifying Theorems 4.7 and 4.8.

Finally, from the last row of Table 7 we see that 1-DRFWL(2) GNN achieves less-than-0.01 normalized MAE on 3-cycles, but performs badly on 4, 5 and 6-cycles. This result verifies Theorem 4.3.

6.2 Molecular property prediction

Datasets. To answer Q2 , we evaluate the performance of d -DRFWL(2) GNNs on four popular molecular graph datasets-QM9, ZINC, ogbg-molhiv and ogbg-molpcba. QM9 contains 130k small molecules, and the task is regression on 12 targets. One can refer to the page for the meaning of those 12 targets. ZINC [20], including a smaller version (ZINC-12K) and a full version (ZINC-250K), is a dataset of chemical compounds and the task is graph regression. The ogbg-molhiv (containing 41k molecules) and ogbg-molpcba (containing 438k molecules) datasets belong to the Open Graph Benchmark (OGB) [31]; the task on both datasets is binary classification. Details of the four datasets are given in Appendix E.2.2.

Baselines. For QM9, the baselines are chosen as 1-GNN, 1-2-3-GNN [44], DTNN [53], Deep LRP [15], PPGN, NGNN [59] and I 2 -GNN [33]. Methods [3, 26, 40, 47] utilizing geometric features or quantum mechanic theory are omitted to fairly compare the graph representation power of the models. For ZINC and ogbg-molhiv, we adopt GIN, PNA [16], DGN [7], HIMP [23], GSN [10], Deep LRP [15], CIN [9], NGNN, GNNAK+, SUN [24] and I 2 -GNN as our baselines. For ogbg-molpcba, the baselines are GIN, PNA, DGN, NGNN and GNNAK+. The experimental details are given in Appendix E.2.2. We leave the results on ZINC, ogbg-molhiv and ogbg-molpcba to Appendix F.3.

Results. On QM9, Table 3 shows that 2-DRFWL(2) GNN attains top two results on most (11 out of 12) targets. Moreover, 2-DRFWL(2) GNN shows a good performance on targets U 0 , U, H and G , where subgraph GNNs like NGNN or I 2 -GNN have a poor performance. The latter fact actually reveals that our method has a stronger ability to capture long-range interactions on graphs than subgraph GNNs, since the targets U 0 , U, H and G are macroscopic thermodynamic properties of

molecules and heavily depend on such long-range interactions. We leave the detailed analysis to Appendix F.4. Apart from QM9, Table 8 of Appendix F.3 shows that d -DRFWL(2) GNN outperforms CIN [9] on ZINC-12K; the performance on ogbg-molhiv is also comparable to baseline methods. These results show that although designed for cycle counting, d -DRFWL(2) GNN is also highly competitive on general molecular tasks.

It is interesting to notice that d -DRFWL(2) GNN shows an inferior performance on ogbg-molpcba, compared with baseline methods. We conjecture that the results on ogbg-molpcba might be insensitive to the gain in cycle counting power, and might prefer simple model architectures rather than highly expressive ones.

6.3 Empirical efficiency

To answer Q3 , we compare the time and memory costs of 2-DRFWL(2) GNN with MPNN, NGNN and I 2 -GNN on two datasets-QM9 and ogbg-molhiv. We use three metrics to evaluate the empirical efficiency of 2-DRFWL(2) GNNs: the maximal GPU memory usage during training, the preprocessing time, and the training time per epoch.

To make a fair comparison, we fix the number of 2-DRFWL(2) GNN layers and the number of message passing layers in all baseline methods to 5; we also fix the size of hidden dimension to 64 for QM9 and 300 for ogbg-molhiv. The subgraph heights for NGNN and I 2 -GNN are both 3. The batch size is always 64.

Results. From Table 4, we see that the preprocessing time of 2-DRFWL(2) GNN is much shorter than subgraph GNNs like NGNN or I 2 -GNN. Moreover, 2-DRFWL(2) GNN requires much less GPU memory while training, compared with subgraph GNNs. The training time of 2-DRFWL(2) GNN is comparable to NGNN (which can only count up to 4-cycles), and much shorter than I 2 -GNN.

We also evaluate the empirical efficiency of 2-DRFWL(2) GNN on graphs of larger sizes. The results, along with details of the datasets we use, are given in Appendix F.5. From Table 14 in Appendix F.5, we see that 2-DRFWL(2) GNN easily scales to graphs with ∼ 500 nodes, as long as the average degree is small.

7 Conclusion and limitations

Motivated by the analysis of why FWL(2) has a stronger cycle counting power than WL(1), we propose d -DRFWL(2) tests and d -DRFWL(2) GNNs. It is then proved that with d = 2 , d -DRFWL(2) GNNs can already count up to 6-cycles, retaining most of the cycle counting power of FWL(2). Because d -DRFWL(2) GNNs explicitly leverage the local nature of cycle counting, they are much more efficient than other existing GNN models that have comparable cycle counting power. Besides, d -DRFWL(2) GNNs also have an outstanding performance on various real-world tasks. Finally, we have to point out that our current implementation of d -DRFWL(2) GNNs, though being efficient most of the time, still has difficulty scaling to datasets with a large average degree such as ogbg-ppa, since the preprocessing time is too long ( ∼ 40 seconds per graph on ogbg-ppa). This also makes our method unsuitable for node classification tasks, since these tasks typically involve graphs with large average degrees. We leave the exploration of more efficient d -DRFWL(2) implementation to future work.

References

Ralph Abboud, Ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. arXiv preprint arXiv:2010.01179 , 2020.

Noga Alon, Raphael Yuster, and Uri Zwick. Finding and counting given length cycles. Algorithmica , 17(3):209-223, 1997.

Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems , 32, 2019.

V. Arvind, Frank Fuhlbrück, Johannes Köbler, and Oleg Verbitsky. On weisfeiler-leman invariance: Subgraph counts and related graph properties. Journal of Computer and System Sciences , 113:42-59, 2020. ISSN 0022-0000. doi: https://doi.org/10.1016/j.jcss.2020.04.003. URL https://www.sciencedirect.com/science/article/pii/S0022000020300386 .

Vikraman Arvind, Frank Fuhlbrück, Johannes Köbler, and Oleg Verbitsky. On weisfeiler-leman invariance: Subgraph counts and related graph properties. Journal of Computer and System Sciences , 113:42-59, 2020.

Muhammet Balcilar, Pierre Héroux, Benoit Gauzere, Pascal Vasseur, Sébastien Adam, and Paul Honeine. Breaking the limits of message passing graph neural networks. In International Conference on Machine Learning , pages 599-608. PMLR, 2021.

Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Directional graph networks. In International Conference on Machine Learning , pages 748-758. PMLR, 2021.

Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. In International Conference on Learning Representations , 2022.

Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Lio, Guido F Montufar, and Michael Bronstein. Weisfeiler and lehman go cellular: Cw networks. Advances in Neural Information Processing Systems , 34:2625-2640, 2021.

Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael M Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(1):657-668, 2022.

Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiv:1711.07553 , 2017.

Jin-Yi Cai, Martin Fürer, and Neil Immerman. An optimal lower bound on the number of variables for graph identification. Combinatorica , 12(4):389-410, 1992.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning , pages 1725-1735. PMLR, 2020.

Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. Advances in neural information processing systems , 32, 2019.

Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count substructures? Advances in neural information processing systems , 33:10383-10395, 2020.

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veliˇ ckovi´ c. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems , 33:13260-13271, 2020.

Leonardo Cotta, Christopher Morris, and Bruno Ribeiro. Reconstruction for powerful graph representations. Advances in Neural Information Processing Systems , 34:1713-1726, 2021.

Mukund Deshpande, Michihiro Kuramochi, and George Karypis. Automated approaches for classifying structures. Technical report, MINNESOTA UNIV MINNEAPOLIS DEPT OF COMPUTER SCIENCE, 2002.

Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699 , 2020.

Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. 2020.

Vijay Prakash Dwivedi, Ladislav Rampášek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. Advances in Neural Information Processing Systems , 35:22326-22340, 2022.

Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428 , 2019.

Matthias Fey, Jan-Gin Yuen, and Frank Weichert. Hierarchical inter-message passing for learning on molecular graphs. arXiv preprint arXiv:2006.12179 , 2020.

Fabrizio Frasca, Beatrice Bevilacqua, Michael M Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. In Advances in Neural Information Processing Systems , 2022.

Martin Fürer. On the combinatorial power of the weisfeiler-lehman algorithm. In International Conference on Algorithms and Complexity , pages 260-271. Springer, 2017.

Johannes Gasteiger, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123 , 2020.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Martin Grohe and Martin Otto. Pebble games and linear equations. The Journal of Symbolic Logic , 80(3):797-844, 2015.

Pedro Hermosilla, Marco Schäfer, Matˇ ej Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozlíková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. arXiv preprint arXiv:2007.06252 , 2020.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265 , 2019.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Ningyuan Teresa Huang and Soledad Villar. A short tutorial on the weisfeiler-lehman test and its variants. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 8533-8537, 2021. doi: 10.1109/ICASSP39728.2021.9413523.

Yinan Huang, Xingang Peng, Jianzhu Ma, and Muhan Zhang. Boosting the cycle counting power of graph neural networks with i 2 -gnns. 2023.

Chuntao Jiang, Frans Coenen, and Michele Zito. Finding frequent subgraphs in longitudinal social network data using a weighted graph mining approach. In Advanced Data Mining and Applications: 6th International Conference, ADMA 2010, Chongqing, China, November 19-21, 2010, Proceedings, Part I 6 , pages 405-416. Springer, 2010.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning , pages 2323-2332. PMLR, 2018.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Mehmet Koyutürk, Ananth Grama, and Wojciech Szpankowski. An efficient algorithm for detecting frequent subgraphs in biological networks. Bioinformatics , 20(suppl_1):i200-i207, 2004.

Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems , 34:21618-21629, 2021.

Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. Advances in Neural Information Processing Systems , 33:4465-4478, 2020.

Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.05013 , 2021.

Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. arXiv preprint arXiv:1812.09902 , 2018.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. Advances in neural information processing systems , 32, 2019.

Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. In International conference on machine learning , pages 4363-4371. PMLR, 2019.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , pages 4602-4609, 2019.

Christopher Morris, Gaurav Rattan, and Petra Mutzel. Weisfeiler and leman go sparse: Towards scalable higher-order graph embeddings. Advances in Neural Information Processing Systems , 33:21824-21840, 2020.

Chendi Qian, Gaurav Rattan, Floris Geerts, Mathias Niepert, and Christopher Morris. Ordered subgraph aggregation networks. In Advances in Neural Information Processing Systems , 2022.

Zhuoran Qiao, Matthew Welborn, Animashree Anandkumar, Frederick R Manby, and Thomas F Miller III. Orbnet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features. The Journal of chemical physics , 153(12):124111, 2020.

Behrooz Tahmasebi, Derek Lim, and Stefanie Jegelka. Counting substructures with higher-order graph neural networks: Possibility and impossibility results. arXiv preprint arXiv:2012.03174 , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Yanbo Wang and Muhan Zhang. Towards better evaluation of gnn expressiveness with brec dataset. arXiv preprint arXiv:2304.07702 , 2023.

Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. NTI, Series , 2(9):12-16, 1968.

Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science , 9(2):513-530, 2018.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems , 32(1):4-24, 2020.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2018.

Zuoyu Yan, Junru Zhou, Liangcai Gao, Zhi Tang, and Muhan Zhang. Efficiently counting substructures by subgraph gnns without running gnn on subgraphs. arXiv preprint arXiv:2303.10576 , 2023.

Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 10737-10745, 2021.

Bohang Zhang, Guhao Feng, Yiheng Du, Di He, and Liwei Wang. A complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests. arXiv preprint arXiv:2302.07090 , 2023.

Muhan Zhang and Pan Li. Nested graph neural networks. Advances in Neural Information Processing Systems , 34:15734-15747, 2021.

Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. In International Conference on Learning Representations , 2022.

Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open , 1:57-81, 2020.

A Message passing neural networks

Message passing neural networks (MPNNs) [27, 36, 50, 55] are a class of GNNs that iteratively updates node representations h u , u ∈ V G by aggregating information from neighbors. At the t -th ( t ⩾ 1) iteration, the update rule for MPNNs is

where h ( t ) u is the representation of node u at the t -th iteration and e uv is the edge feature of { u, v } ∈ E G . f ( t ) and m ( t ) are learnable functions, and ⊕ is a permutation-invariant aggregation operator, such as sum, mean or max. The representation of graph G is obtained by

where R is a permutation-invariant function of multisets.

It is known [55] that the ability of MPNNs to discriminate between non-isomorphic graphs is upperbounded by the WL(1) test. When it comes to counting cycles, MPNNs cannot graph-level count any cycles , as stated in [33].

B Proofs of theorems in Section 3

B.1 Proof of Theorem 3.1

We restate Theorem 3.1 as following,

Theorem B.1. In terms of the ability to distinguish between non-isomorphic graphs, the d -DRFWL(2) test is strictly more powerful than WL(1), for any d ⩾ 1 .

Proof. In Theorem 3.2 we will prove that ( d + 1) -DRFWL(2) is strictly more powerful than d -DRFWL(2), for any d ⩾ 1 . Therefore, we only need to show that 1-DRFWL(2) is strictly more powerful than WL(1). We will first prove that 1-DRFWL(2) can give an implementation of the WL(1) test. Actually, let

be the initial 1-DRFWL(2) colors of ( u, v ) ∈ V 2 G with 0 ⩽ d ( u, v ) ⩽ 1 . Here W (0) WL(1) ( u ) is the initial WL(1) color of node u (which is identical for all u ∈ V G ). It is obvious that W (0) ( u, v ) does only depend on d ( u, v ) . As for the update rule, at the (2 t -1) -th iteration with t ⩾ 1 , we ask

and

for those ( u, v ) with distance 1. Here, both N 0 ( u ) ∩ N 1 ( v ) and N 1 ( u ) ∩ N 0 ( v ) have only one element, so POOL 1(2 t -1) 01 and POOL 1(2 t -1) 10 simply select the second and the first component from the unique 2-tuple in the corresponding multisets, respectively. The HASH (2 t -1) k functions for k = 0 or 1 are chosen as

What we did is to ask the (2 t -1) -th iteration of 1-DRFWL(2) to record the WL(1) colors ( W ( t -1) WL(1) ( u ) , W ( t -1) WL(1) ( v ) ) in the 1-DRFWL(2) color of node pair ( u, v ) . In the 2 t -th iteration, 1-DRFWL(2) then uses this record to update the colors of ( u, u ) and ( v, v ) :

And we ask

Here W (2 t -2) ( u, u ) and W (2 t -2) ( v, v ) are stored in W (2 t -1) ( u, v ) in the last iteration (in the first and the second components respectively). HASH ( t ) WL(1) and POOL ( t ) WL(1) are the hashing functions used by WL(1) test, as in (1). It is easy to see the above implementation uses 2 iterations of 1DRFWL(2) update to simulate 1 iteration of WL(1) update. Therefore, 1-DRFWL(2) is at least as powerful as WL(1) in terms of the ability to distinguish between non-isomorphic graphs.

To see why 1-DRFWL(2) is strictly more powerful than WL(1), we only need to find out a pair of graphs G and H such that WL(1) cannot distinguish between them while 1-DRFWL(2) can. Let G be two 3-cycles and H be one 6-cycle. Of course WL(1) cannot distinguish between G and H . However, 1-DRFWL(2) can distinguish between them because there are no triangles in H but two in G , as is made clear in the following. In the first iteration of 1-DRFWL(2), the M 1(1) 11 term collects common neighbors of every node pair ( u, v ) with distance 1. In G this results in non-empty common neighbor multisets for all distance-1 tuples ( u, v ) , while in H all such multisets are empty. 1-DRFWL(2) then makes use of this discrepancy by properly choosing the HASH (1) 1 function that assigns different colors for distance-1 tuples in G and in H . Therefore, 1-DRFWL(2) can distinguish between G and H .

B.2 Proof of Theorem 3.2

We restate Theorem 3.2 as following,

Theorem B.2. In terms of the ability to distinguish between non-isomorphic graphs, FWL(2) is strictly more powerful than d -DRFWL(2), for any d ⩾ 1 . Moreover, ( d +1) -DRFWL(2) is strictly more powerful than d -DRFWL(2).

Proof. First we show that FWL(2) can give an implementation of d -DRFWL(2) for any d ⩾ 1 . The implementation is in three steps:

Distance calculation. For any 2-tuple ( u, v ) ∈ V 2 G , let its FWL(2) color W (0) ( u, v ) be 0 if u = v , 1 if { u, v } ∈ E G , and ∞ otherwise. In the first ( n -1) iterations, FWL(2) can use the following update rule to calculate distance between any pair of nodes,

When t = n -1 , the FWL(2) color for any ( u, v ) ∈ V 2 G gets stable and becomes d ( u, v ) .

Initial color generation. At the n -th iteration, FWL(2) transforms W ( n -1) ( u, v ) = d ( u, v ) into

Here W (0) d -DRFWL(2) ( u, v ) is the initial d -DRFWL(2) color of 2-tuple ( u, v ) . Because W (0) d -DRFWL(2) ( u, v ) only depends on d ( u, v ) , the RHS in (24) is a function of W ( n -1) ( u, v ) = d ( u, v ) , and thus complies with the general form of FWL(2) as in (3).

u

v

u

v

Figure 3: Illustration on the counterexample used in the proof of the separation result in Theorem 3.2. Here we take d = 3 and a (3 d +1) -cycle, or 10-cycle, is shown. It is clear that for a distance-( d +1) tuple there are nodes on both the inferior arc and the superior arc that contribute to its update. Contrarily, for a distanced tuple only nodes on the inferior arc contribute.

d -DRFWL(2) Update. For the ( n + t ) -th iteration with t ⩾ 1 , the FWL(2) test uses the following POOL ( n + t ) and HASH ( n + t ) functions to simulate the t -th iteration of d -DRFWL(2):

HASH ( n + t ) reads the first field of W ( n + t -1) ( u, v ) to get d ( u, v ) = k , and decides whether to update the color for ( u, v ) (if 0 ⩽ k ⩽ d ) or not (otherwise).

For every w ∈ V G , POOL ( n + t ) reads the first field of W ( n + t -1) ( w,v ) and W ( n + t -1) ( u, w ) to find that d ( w,v ) = i and d ( u, w ) = j , and then either selects W ( t -1) d -DRFWL(2) ( w,v ) and W ( t -1) d -DRFWL(2) ( u, w ) from the second field of W ( n + t -1) ( w,v ) and W ( n + t -1) ( u, w ) , and applies (7) to get M k ( t ) ij ( u, v ) , if it finds 0 ⩽ i, j ⩽ d , or ignores the contribution from ( W ( n + t -1) ( w,v ) , W ( n + t -1) ( u, w ) ) otherwise.

HASH ( n + t ) calculates W ( t ) d -DRFWL(2) ( u, v ) using (6), and assigns it to the second field of W ( n + t ) ( u, v ) , if it has decided that the color for ( u, v ) should be updated.

For the READOUT part, FWL(2) simply ignores the tuples ( u, v ) with d ( u, v ) > d . It is easy to see the above construction does provide an implementation for d -DRFWL(2).

Similarly, one can prove that ( d +1) -DRFWL(2) also gives an implementation of d -DRFWL(2), for any d ⩾ 1 , by only executing the 'initial color generation' and ' d -DRFWL(2) update' steps stated above. Since the colors for ( u, v ) with d ( u, v ) > d never update, it is sufficient only keeping track of pairs with mutual distance ⩽ d +1 , which ( d +1) -DRFWL(2) is capable of.

Now we will turn to prove the separation result: for any d ⩾ 1 , there exist graphs G and H that FWL(2) (or ( d +1) -DRFWL(2)) can distinguish, but d -DRFWL(2) cannot. We ask G to be two (3 d +1) -cycles, and H a single (6 d +2) -cycle . FWL(2) can distinguish between G and H by (i) calculating distance between every pair of nodes in G and H ; (ii) check if there is ( u, v ) ∈ V 2 such that d ( u, v ) = ∞ at the READOUT step. The above procedure will give G a true label and H a false label.

d +1(1)

To see why ( d +1) -DRFWL(2) can also distinguish between G and H , we designate that POOL ij be a multiset function that counts the elements in the multiset, for all 0 ⩽ i, j ⩽ d +1 , and that HASH (1) d +1 be the sum of all items in ( M d +1(1) ij ( u, v ) ) 0 ⩽ i,j ⩽ d +1 . Briefly speaking, we are asking the first iteration of ( d +1) -DRFWL(2) to count how many nodes w contribute to the update of a distance-( d +1) tuple ( u, v ) , via the form ( W (0) ( w,v ) , W (0) ( u, w ) ) . For any distance-( d +1) tuple in G , the count is 4 for d = 1 and d +5 for d > 1 (actually when d > 1 , a distance-( d +1) tuple ( u, v ) splits a (3 d +1) -cycle into an inferior arc of length ( d +1) and a superior arc of length 2 d ; d

nodes on the inferior arc and 3 nodes on the superior arc, along with u and v , sum up to the number; Figure 3a illustrates the case of d = 3 ). For a distance-( d +1) tuple in H , the count becomes d +2 , which is always different from the count in G . The ( d +1) -DRFWL(2) test then uses this discrepancy to render different colors for a distance-( d +1) tuple in G and in H , thus telling G from H .

The above proof also reveals why d -DRFWL(2) fails to distinguish between G and H . Actually, let ( u, v ) be a distancek tuple in V 2 G or V 2 H with 0 ⩽ k ⩽ d . In the following we use W ( t ) ( u, v ) to denote the d -DRFWL(2) color of ( u, v ) at the t -th iteration, both for ( u, v ) ∈ V 2 G and ( u, v ) ∈ V 2 H . We will now use induction to prove that for all t , W ( t ) ( u, v ) only depends on d ( u, v ) , and does not depend on whether ( u, v ) is in G or H . This implies that for ( u, v ) ∈ V 2 G and ( u ′ , v ′ ) ∈ V 2 H , W ( t ) ( u, v ) = W ( t ) ( u ′ , v ′ ) holds for all iterations t as long as d ( u, v ) = d ( u ′ , v ′ ) .

Base case. Since the initial d -DRFWL(2) color of a distancek tuple (0 ⩽ k ⩽ d ) only depends on k , the t = 0 case is trivial.

Induction step. Now we assume W ( t -1) ( u, v ) only depends on d ( u, v ) no matter which graph ( u, v ) is in, for some t ⩾ 1 . Wewill then prove that no matter what HASH ( t ) k and POOL k ( t ) ij functions we choose, W ( t ) ( u, v ) only depends on d ( u, v ) no matter which graph ( u, v ) is in. Actually, it is sufficient to prove that the multisets

are equal, for any ( u, v ) ∈ V 2 G ∪ V 2 H with d ( u, v ) = k and 0 ⩽ k ⩽ d . This is because the inductive hypothesis leads us to the fact that

can only depend on the number of elements of the multiset on the RHS, because all elements in the RHS multiset must be equal . Therefore, the aforementioned condition guarantees that M k ( t ) ij ( u, v ) are all the same for any distancek tuple ( u, v ) ∈ V 2 G ∪ V 2 H with any fixed i, j, k and any t . This, along with the update rule (6), makes the induction step.

Now we prove that multisets defined by (25) are equal for all ( u, v ) ∈ V 2 G ∪ V 2 H with any given d ( u, v ) = k , 0 ⩽ k ⩽ d . The intuition of the proof can be obtained from Figure 3b, which shows that for any distanced tuple ( u, v ) in a cycle not shorter than (3 d +1) , the nodes that contribute to (25) are exactly those nodes on the inferior arc cut out by ( u, v ) , plus u and v . This means that the distanced tuple is completely agnostic about the length of the cycle in which it lies , as long as the cycle length is ⩾ 3 d +1 . Therefore, any distanced tuple cannot tell whether it is in G or in H , resulting in the conclusion that the multisets in (25) are equal for all distanced tuples ( u, v ) . Similar arguments apply to all distancek tuples in G and H , as long as 0 ⩽ k ⩽ d . Therefore, we assert that W ( t ) ( u, v ) does only depend on d ( u, v ) , no matter which graph ( u, v ) is in, at the t -th iteration. This finishes the inductive proof.

Notice that there are (6 d + 2) distancek tuples in either G or H , for any 0 ⩽ k ⩽ d ; moreover, each of those distancek tuples have identical colors. We then assert that G and H must get identical representations after running d -DRFWL(2) on them. Therefore, d -DRFWL(2) fails to distinguish between G and H .

We remark that for any d ⩾ 1 , d -DRFWL(2) tests cannot distinguish between two k -cycles and a 2 k -cycle, as long as k ⩾ 3 d +1 . The proof for this fact is similar to the one elaborated above.

B.3 Proof of the equivalence in representation power between d -DRFWL(2) tests and d -DRFWL(2) GNNs

At the end of 3.2, we informally state the fact that d -DRFWL(2) GNNs have equal representation power to d -DRFWL(2) tests under certain assumptions. We now restate the fact as the following

Proposition B.3. Let q : G → colors be a d -DRFWL(2) test whose HASH ( t ) k , POOL k ( t ) ij and READOUT functions are injective, ∀ 0 ⩽ i, j, k ⩽ d and ∀ t ⩾ 1 ; in addition, q assigns different initial colors to 2-tuples ( u, v ) with different u, v distances. Let f : G → R p be a d -DRFWL(2) GNN

with T d -DRFWL(2) GNN layers; the initial representations h (0) uv are assumed to depend only on d ( u, v ) .

̸

If two graphs G and H get different representations under f , i.e. f ( G ) = f ( H ) , then q assigns different colors for G and H . Moreover, if G and H are graphs such that q assigns different colors for G and H , there exists a d -DRFWL(2) GNN f such that f ( G ) = f ( H ) .

Proof. For simplicity, we denote

We prove the first part by proving that, if two tuples ( u, v ) and ( u ′ , v ′ ) get different representations h ( T ) uv and h ( T ) u ′ v ′ after applying L T ◦ σ T -1 ◦ · · · ◦ σ 1 ◦ L 1 , then

̸

̸

after we apply q for T iterations. Since W ( ∞ ) ( u, v ) is a refinement of W ( T ) ( u, v ) , (27) implies that W ( ∞ ) ( u, v ) = W ( ∞ ) ( u ′ , v ′ ) . Now, f ( G ) = f ( H ) means for all bijections b from V 2 G, ⩽ d to V 2 H, ⩽ d , there exists a pair ( u, v ) ∈ V 2 G, ⩽ d such that h ( T ) uv = h ( T ) u ′ v ′ with ( u ′ , v ′ ) = b ( u, v ) . If the above statement holds true, this means for ( u, v ) and ( u ′ , v ′ ) we have W ( ∞ ) ( u, v ) = W ( ∞ ) ( u ′ , v ′ ) . Since this is the case for any bijection b , we assert

̸

̸

or simply W ( G ) = W ( H ) , and the first part is proved.

̸

̸

Proof for the above statement can be conducted inductively. When T = 0 , h (0) uv = h (0) u ′ v ′ means d ( u, v ) = d ( u ′ , v ′ ) , since the initial representation h (0) uv of ( u, v ) only depends on the distance d ( u, v ) . By the second condition of q , the above fact further implies W (0) ( u, v ) = W (0) ( u ′ , v ′ ) , and the base case is proved.

̸

̸

̸

Now, assuming that the above statement is true for T = ℓ -1 with ℓ ⩾ 1 , we now prove the T = ℓ case. Given h ( ℓ ) uv = h ( ℓ ) u ′ v ′ , there are two possibilities: (a) h ( ℓ -1) uv = h ( ℓ -1) u ′ v ′ ; (b) for some i, j , a ijk ( ℓ ) uv = a ijk ( ℓ ) u ′ v ′ , where k = d ( u, v ) = d ( u ′ , v ′ ) (we can safely assume d ( u, v ) = d ( u ′ , v ′ ) , since otherwise (27) trivially holds).

̸

̸

For possibility (a), the inductive hypothesis tells us W ( ℓ -1) ( u, v ) = W ( ℓ -1) ( u ′ , v ′ ) , thus W ( ℓ ) ( u, v ) = W ( ℓ ) ( u ′ , v ′ ) . For possibility (b), notice that a ijk ( ℓ ) uv is a function of the multiset

̸

Therefore, a ijk ( ℓ ) uv = a ijk ( ℓ ) u ′ v ′ means that for all bijection b ′ from N i ( u ) ∩ N j ( v ) to N i ( u ′ ) ∩ N j ( v ′ ) , there exists w ∈ N i ( u ) ∩ N j ( v ) such that ( h ( ℓ -1) wv , h ( ℓ -1) uw ) = ( h ( ℓ -1) w ′ v ′ , h ( ℓ -1) u ′ w ′ ) , where w ′ = b ′ ( w ) . Without loss of generality, let us discuss the case where h ( ℓ -1) wv = h ( ℓ -1) w ′ v ′ . In this case, the inductive hypothesis tells us W ( ℓ -1) ( w,v ) = W ( ℓ -1) ( w ′ , v ′ ) , thus ( W ( ℓ -1) ( w,v ) , W ( ℓ -1) ( u, w )) = ( W ( ℓ -1) ( w ′ , v ′ ) , W ( ℓ -1) ( u ′ , w ′ )) . The other case h ( ℓ -1) uw = h ( ℓ -1) u ′ w ′ also leads to this result. Since the bijection b ′ is arbitrary, we assert that the multisets and

̸

̸

are not equal. Since the functions POOL k ( ℓ ) ij and HASH ( ℓ ) k are injective, the above result implies W ( ℓ ) ( u, v ) = W ( ℓ ) ( u ′ , v ′ ) . So far, we have finished the induction step. Therefore, the first part of the theorem is true.

̸

̸

̸

̸

̸

̸

̸

̸

̸

For the second part of the theorem, we quote the well-known result of Xu et al. [55] (Lemma 5 of the paper), that there exists a function f : X → R n on a countable input space X such that any multiset function g can be written as

for some function ϕ , where X is an arbitrary multiset whose elements are in X .

Now, let G and H be graphs that both obtain their d -DRFWL(2) stable colorings under q after T iterations, and that q assigns different colors for them. We will now prove that there exists a d -DRFWL(2) GNN with T d -DRFWL(2) GNN layers that gives G and H different representations.

Since the set of d -DRFWL(2) colors generated by q is countable, we can designate a mapping ν : colors → N that assigns a unique integer for every kind of d -DRFWL(2) color. Under mapping ν , all HASH ( t ) k , POOL k ( t ) ij and READOUT functions can be seen as functions with codomain N .

Then, the initial representation for ( u, v ) with 0 ⩽ d ( u, v ) ⩽ d is h (0) uv = ν ( W (0) ( u, v ) ) . For the update rules, the lemma in [55] tells us that by choosing proper m ( t ) ijk and ϕ ( t ) ijk functions, the following equality can hold,

with X being any multiset of 2-tuples of d -DRFWL(2) colors . Therefore, we can choose m ( t ) ijk in (10) following the instructions of the above lemma, and choose ⊕ as ∑ . We then choose the functions f ( t ) k in (11) as

It is now easy to iteratively prove that h ( t ) uv is exactly ν ( W ( t ) ( u, v ) ) . For the pooling layer, we again leverage the lemma in [55] to assert that there exist functions r and M ′ such that

where X is any multiset of d -DRFWL(2) colors. We then choose

and choose M = ν ◦ M ′ . Now the d -DRFWL(2) GNN constructed above with T d -DRFWL(2) GNN layers produces exactly the same output as q (except that the output is converted to integers via ν ). Because q assigns G and H different colors, the d -DRFWL(2) GNN gives different representations for G and H .

C Proofs of theorems in Section 4

We first give a few definitions that will be useful in the proofs.

Definition C.1 (Pair-wise cycle counts) . Let u, v ∈ V G , we denote C k,l ( u, v ) as the number of ( k + l ) -cycles S that satisfy:

S passes u and v .

There exists a k -path and an l -path (distinct from one another) from u to v , such that every edge in either path is an edge included in S .

For instance, the substructures counted by C 2 , 3 ( u, v ) and C 3 , 4 ( u, v ) are depicted in Figures 4a and 4b respectively.

Definition C.2 (Tailed triangle counts) . Let u, v ∈ V G , T ( u, v ) is defined as the number of tailed triangles in which u and v are at positions shown in Figure 4c.

Definition C.3 (Chordal cycle counts) . Let u, v ∈ V G , CC 1 ( u, v ) and CC 2 ( u, v ) are defined as the numbers of chordal cycles in which u and v are at positions shown in Figures 4d and 4e, respectively. Moreover, we use CC 1 ( u ) and CC 2 ( u ) to denote the node-level chordal cycle counts with the node u located at positions shown in Figures 4f and 4g, respectively.

The notations CC 1 and CC 2 are overloaded in the above definition, which shall not cause ambiguities with a check on the number of arguments.

Definition C.4 (Triangle-rectangle counts) . Let u, v ∈ V G , TR 1 ( u, v ) and TR 2 ( u, v ) are defined as the numbers of triangle-rectangles in which u and v are at positions shown in Figures 4h and 4i, respectively. We also define three types of node-level triangle-rectangle counts, namely TR 1 ( u ) , TR 2 ( u ) and TR 3 ( u ) . They are the numbers of triangle-rectangles where node u is located at positions shown in Figures 4j, 4k and 4l respectively.

v

u

(a)

C

2

,

3

(

u, v

)

v

u

(b)

C

3

,

4

(

u, v

)

u

v

(c)

T

(

u, v

)

v

u

(d)

CC

1

(

u, v

)

v

u

(e)

CC

2

(

u, v

)

u

(f)

CC

1

(

u

)

u

(g)

CC

2

(

u

)

u

v

(h)

TR

1

(

u, v

)

u

v

(i)

TR

2

(

u, v

)

u

(j)

TR

1

(

u

)

u

(k)

TR

2

(

u

)

u

(l)

TR

3

(

u

)

There are relations between the above-defined counts. For chordal cycle counts we have

and

For triangle-rectangle counts we have

and

We also denote P k ( u, v ) as the number of k -paths starting at node u and ending at node v , W k ( u, v ) as the number of k -walks from u to v , and C k ( u ) as the number of k -cycles that pass a node u .

The following lemma will be used throughout this section.

Lemma C.5. Let S be a graph substructure. If there exists a d -DRFWL(2) test q such that for any graph G ∈ G ,

after running q for T iterations on G , then d -DRFWL(2) GNNs can node-level count S .

̸

Proof. Let ( G 1 , u 1 ) , ( G 2 , u 2 ) ∈ G × V , and C ( S, u 1 , G 1 ) = C ( S, u 2 , G 2 ) . Then by assumption,

̸

̸

after running q for T iterations on G 1 and G 2 . From the proof of Proposition B.3 (which is in Appendix B.3), we see that there exists a d -DRFWL(2) GNN f with T d -DRFWL(2) GNN layers such that a tuple ( u, v ) with 0 ⩽ d ( u, v ) ⩽ d gets its representation h ( T ) uv = ν ( W ( T ) ( u, v ) ) , where ν is an injective mapping from the color space of q to N . Therefore, (37) implies h ( T ) u 1 u 1 = h ( T ) u 2 u 2 , or f ( G 1 , u 1 ) = f ( G 2 , u 2 ) . This means that d -DRFWL(2) GNNs can node-level count S .

̸

C.1 Proof of Theorem 4.3

We restate Theorem 4.3 as following,

Theorem C.6. 1-DRFWL(2) GNNs can node-level count 3-cycles, but cannot graph-level count any longer cycles.

Proof. To prove that 1-DRFWL(2) GNNs can node-level count 3-cycles, it suffices to find a 1DRFWL(2) test q such that after running q for T iterations on a graph G ,

Below we explicitly construct q . Notice that if d ( u, v ) = 1 , M 1(1) 11 ( u, v ) can know the count C ( u, v ) = |N 1 ( u ) ∩ N 1 ( v ) | in the first 1-DRFWL(2) iteration. This is essentially the number of triangles in which { u, v } ∈ E G is included as an edge. In the second iteration, we can ask

and we have W (2) ( u, u ) = C 3 ( u ) . Therefore, the positive result is proved.

To prove that 1-DRFWL(2) GNNs cannot graph-level count k -cycles with k ⩾ 4 , we make use of the fact mentioned at the end of Appendix B.2, that 1-DRFWL(2) tests (and thus GNNs) cannot distinguish between two k -cycles and one 2 k -cycle, as long as k ⩾ 4 . This set of counterexamples leads to the negative result.

C.2 Proofs of Theorems 4.4-4.7

Before proving the theorems, we state and prove a series of lemmas.

Lemma C.7. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any 2-tuple ( u, v ) ∈ V 2 G with d ( u, v ) = 1 or d ( u, v ) = 2 , q assigns

for some integer T ⩾ 1 .

Proof. Let q be a 2-DRFWL(2) test that satisfies: at the first iteration, M 1(1) 11 ( u, v ) and M 2(1) 11 ( u, v ) both count the number of nodes in N 1 ( u ) ∩ N 1 ( v ) , and HASH (1) k of q simply selects the M k (1) 11 component, for k = 1 , 2 ; q stops updating colors for any tuple from the second iteration. It is easy to see that q satisfies the condition stated in the lemma.

Lemma C.8. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any node u ∈ V G , q assigns

for some integer T ⩾ 1 . Here deg( u ) is the degree of node u ∈ V G .

Proof. Let q be a 2-DRFWL(2) test that satisfies: at the first iteration, M 0(1) 11 ( u, u ) counts |N 1 ( u ) | = deg( u ) , and HASH (1) 0 of q selects the M 0(1) 11 component, then W (1) ( u, u ) = deg( u ) . q then stops updating colors for any tuple from the second iteration. It is obvious that q satisfies the condition stated in the lemma.

Lemma C.9. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any 2-tuple ( u, v ) ∈ V 2 G with d ( u, v ) = 1 or d ( u, v ) = 2 , q assigns

for some integer T ⩾ 2 .

Proof. By Lemma C.8, there exists a 2-DRFWL(2) test q that assigns W ( T ) ( u, u ) = deg( u ) for distance-0 tuples ( u, u ) with u ∈ V G , where T ⩾ 1 . Now, for the ( T +1) -th iteration of q , we ask

where k = 1 or 2 ; q then assigns

for d ( u, v ) = k and k = 1 , 2 . Now W ( T +1) ( u, v ) = deg( u ) + deg( v ) for T +1 ⩾ 2 .

Lemma C.10. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any 2-tuple ( u, v ) ∈ V 2 G with d ( u, v ) = 1 or d ( u, v ) = 2 , q assigns

̸

for some integer T ⩾ 2 .

Proof. A 3-path starting at u and ending at v is a 3-walk u → x → y → v with u = y and x = v . Therefore,

̸

Here #( u → x → u → v ) is the number of different ways to walk from u to a neighboring node x , then back to u , and finally to v , which is also a neighbor of u ( x can coincide with v ). The term #( u → v → y → v ) can be interpreted analogously. The 1 ( u,v ) ∈E G takes value 1 if ( u, v ) ∈ E G , and 0 otherwise. This term accounts for the count #( u → v → u → v ) which is subtracted twice.

It is easy to see that

Now we construct q as following: at the first iteration, q chooses proper POOL k (1) ij and HASH (1) k functions ( 0 ⩽ i, j, k ⩽ 2 ) such that

u

v

x

y

(

z

)

(a) Setting

u

=

z

v

u

y

z

(

x

)

(b) Setting

x

=

v

v

u

y

x

(

z

)

(c) Setting

x

=

z

u

v

y

(

z

)

(

x

)

(d) Setting

u

=

z

and

x

=

v

̸

̸

From Lemmas C.7 and C.9, this is possible. At the second iteration, we ask M k (2) 11 , M k (2) 12 and M k (2) 21 to be

w

∈N

1

(

u

)

∩N

2

(

v

)

respectively. Here k takes 1 or 2 . We notice that if d ( u, v ) = k with k = 1 , 2 ,

because the RHS of (53) is exactly 1 2 ( ∑ w ∈N 1 ( v ) W 2 ( u, w ) + ∑ w ∈N 1 ( u ) W 2 ( w,v ) ) . Therefore, for a distancek tuple ( u, v ) where k = 1 or 2 ,

is a function of M k (2) 11 , M k (2) 12 , M k (2) 21 and W (1) ( u, v ) , and thus can be assigned to W (2) ( u, v ) by choosing proper HASH (2) k functions. After q assigns W (2) ( u, v ) = P 3 ( u, v ) for those ( u, v ) with d ( u, v ) = 1 or d ( u, v ) = 2 , it then stops updating colors for any tuple from the third iteration. It is now clear that q satisfies the condition stated in the lemma.

Lemma C.11. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any node u ∈ V G , q assigns

for some integer T ⩾ 2 .

Proof. This lemma follows from Theorem 4.3 and the fact that the update rule of 2-DRFWL(2) tests encompasses that of 1-DRFWL(2) tests.

Lemma C.12. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any 2-tuple ( u, v ) ∈ V 2 G with d ( u, v ) = 1 or d ( u, v ) = 2 , q assigns

̸

for some integer T ⩾ 3 .

̸

̸

̸

Proof. A 4-path starting at u and ending at v is a 4-walk u → x → y → z → v with u = y, u = z, x = z, x = v and y = v . We will calculate P 4 ( u, v ) as following: first define

̸

̸

̸

̸

It is easy to see that P 2 , 2 ( u, v ) gives the number of 4-walks u → x → y → z → v with only constraints u = y and y = v . Therefore,

̸

̸

̸

where #( a ) , #( b ) , #( c ) and #( d ) are the numbers of four types of 4-walks illustrated in Figure 5a, 5b, 5c and 5d, respectively. They correspond to setting u = z , x = v , x = z and both setting u = z and x = v in the 4-walk u → x → y → z → v , respectively, while keeping u = y and y = v .

Now, it is not hard to get the following results,

We will briefly explain (59)-(62) in the following. The contribution from (59), (60) and (62) only exists when u and v are neighbors, which accounts for the 1 ( u,v ) ∈E G factor. For #( a ) , if we allow v = y , then the count is simply C 3 ( u ) times 2 (because two directions of a 3-cycle that passes u are counted as two different walks); the contribution of counts from the additional v = y case is exactly #( d ) , and is P 2 ( u, v ) , as long as ( u, v ) ∈ E G . A similar argument applies to #( b ) . Determining #( c ) and #( d ) is easier and the calculation will not be elaborated here.

Combining the above results, we can give a formula for P 4 ( u, v ) ,

Given the explicit formula (63), we now construct q as follows: at the first iteration, q chooses proper POOL k (1) ij and HASH (1) k functions ( 0 ⩽ i, j, k ⩽ 2 ) such that

From Lemmas C.7 and C.9, this is possible. At the second iteration, we let

where 1 ⩽ i, j ⩽ 2 , and

for every ( u, v ) with d ( u, v ) = 1 or d ( u, v ) = 2 . Notice that (65), (66) and (67) can be calculated by an iteration of 2-DRFWL(2) with proper POOL k (2) ij functions, where 1 ⩽ i, j, k ⩽ 2 . Therefore, we assert that

can be calculated by an iteration of 2-DRFWL(2), with proper POOL k (2) ij and HASH (2) k functions, 1 ⩽ i, j, k ⩽ 2 . We now ask q to assign

at the second iteration. From the above discussion and Lemma C.11, this is possible. At the third iteration, q gathers information from (69) and assigns W (3) ( u, v ) = P 4 ( u, v ) for those ( u, v ) with d ( u, v ) = 1 or d ( u, v ) = 2 , using formula (63); since we already have all the terms of (63) prepared in (69) (notice that ∑ 1 ⩽ i,j ⩽ 2 P ij 2 , 2 ( u, v ) = P 2 , 2 ( u, v ) ), it is easy to see that P 4 ( u, v ) can be calculated by an iteration of 2-DRFWL(2) with proper POOL k (3) ij and HASH (3) k functions, 1 ⩽ i, j, k ⩽ 2 . We then ask q to stop updating colors for any tuple from the fourth iteration. Now q satisfies the condition stated in the lemma.

Lemma C.13. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any 2-tuple ( u, v ) ∈ V 2 G with d ( u, v ) = 1 or d ( u, v ) = 2 , q assigns

for some integer T ⩾ 3 .

Proof. Following the notations in the proof of Lemma C.12, we have

where #( u → x → u → z → v ) is the number of different ways to walk from u to a neighboring node x , then back to u , and, through another neighboring node z , finally to v ( x, z, v can coincide with each other). The number #( u → x → v → z → v ) can be interpreted analogously. It is easy to see that

for d ( u, v ) = 1 or d ( u, v ) = 2 . Therefore,

From Lemma C.12, there exists a 2-DRFWL(2) test q such that q assigns W (2) ( u, v ) = P 2 , 2 ( u, v ) for ( u, v ) with d ( u, v ) = 1 or d ( u, v ) = 2 . Moreover, from Lemmas C.7 and C.9, there exists another 2-DRFWL(2) test q ′ such that q ′ assigns W (2) ( u, v ) = (deg( u ) + deg( v )) P 2 ( u, v ) , for d ( u, v ) = 1 or d ( u, v ) = 2 . Synthesizing the colors that q and q ′ assign to ( u, v ) respectively, it is easy to construct a 2-DRFWL(2) test that assigns W (3) ( u, v ) = W 4 ( u, v ) , where d ( u, v ) = 1 or d ( u, v ) = 2 . We then let it stop updating colors for any tuple from the fourth iteration to make it satisfy the condition stated in the lemma.

In the following, we denote P k ( u ) as the number of k -paths that starts at a node u , and W k ( u ) as the number of k -walks that starts at a node u .

Lemma C.14. There exists a 2-DRFWL(2) test q for any k ⩾ 1 such that for any graph G ∈ G and for any node u ∈ V G , q assigns

for some integer T .

Proof. We prove by induction. When k = 1 , W 1 ( u ) = deg( u ) , and by Lemma C.8, the conclusion is obvious. Now we assume that for k = ℓ , there exists a 2-DRFWL(2) test q such that q assigns

for some integer T . Then, at the ( T +1) -th iteration, we ask

At the ( T +2) -th iteration, we ask

for every distance-0 tuple ( u, u ) ∈ V 2 G . It is easy to see that

Therefore, W ( T +2) ( u, u ) = W ℓ +1 ( u ) , and the induction step is finished. We then assert that the lemma is true.

Lemma C.15. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any node u ∈ V G , q assigns

for some integer T ⩾ 3 .

Proof. From Lemma C.10, there exists a 2-DRFWL(2) test q such that q assigns W ( T ) ( u, v ) = C 1 , 3 ( u, v ) for some integer T ⩾ 2 , for any graph G ∈ G and any 2-tuple ( u, v ) ∈ V 2 G with d ( u, v ) = 1 . C 4 ( u ) can be calculated from C 1 , 3 ( u, v ) in another iteration.

Lemma C.16. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any node u ∈ V G , q assigns

for some integer T ⩾ 4 .

Proof. From Lemma C.12, there exists a 2-DRFWL(2) test q such that q assigns W ( T ) ( u, v ) = C 1 , 4 ( u, v ) for some integer T ⩾ 3 , for any graph G ∈ G and any 2-tuple ( u, v ) ∈ V 2 G with d ( u, v ) = 1 . C 5 ( u ) can be calculated from C 1 , 4 ( u, v ) in another iteration.

Lemma C.17. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any node u ∈ V G , q assigns

for some integer T ⩾ 3 .

Proof. By the first equation of (32),

Now, using Lemma C.7, CC 1 ( u ) can be assigned by a 2-DRFWL(2) test to W ( T ) ( u, u ) , for some integer T ⩾ 3 and for all nodes u ∈ V G .

u

v

t

x

y

z

̸

̸

̸

Figure 6: Illustrations accompanying the proof of Lemma C.19. In all subfigures, it is assumed that u → x → y → z → v is a 4-path and u → t → v is a 2-path.

Lemma C.18. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any node u ∈ V G , q assigns

for some integer T ⩾ 4 .

Proof. We first calculate TR 1 ( u, v ) as following,

Therefore, by the first equation of (34),

By Lemmas C.7 and C.10, TR 1 ( u ) can be assigned by a 2-DRFWL(2) test to W ( T ) ( u, u ) , for some integer T ⩾ 4 and for all nodes u ∈ V G .

Lemma C.19. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any node u ∈ V G , q assigns

for some integer T ⩾ 5 .

Proof. We first try to calculate C 2 , 4 ( u, v ) , which is shown in Figure 6a. (If the 6-cycle is thought to be a benzene ring in organic chemistry, then u and v are on meta-positions .) It is easy to see that

u

v

x

y

z

(

t

)

u

v

x

y

z

(

t

)

u

v

x

y

z

(

t

)

We calculate C 2 , 4 ( u, v ) as following,

̸

where #( b ) , #( c ) and #( d ) are the numbers of substructures illustrated in Figure 6b, 6c and 6d, respectively. In those figures, we always assume that u → x → y → z → v is a 4-path and u → t → v is a 2-path. Since every 4-path u → x → y → z → v and every 2-path u → t → v with t = x, t = y and t = z contribute one count to C 2 , 4 ( u, v ) , it is clear that (89) gives the correct number C 2 , 4 ( u, v ) .

̸

̸

We now calculate #( b ) , #( c ) and #( d ) . For #( b ) , we have

̸

Here, #( b , u = y ) or #( b , u = z ) is the count of substructure obtained by coalescing nodes u, y or u, z in Figure 6b, while keeping y = v and x = z . It is easy to see that

̸

The count #( d ) can be calculated analogously, and we summarize the results below.

By Lemmas C.7, C.10 and C.12, there exist 2-DRFWL(2) tests that assign the numbers P 2 ( u, v ) P 4 ( u, v ) and #( b ) + #( d ) to W ( T ) ( u, v ) respectively, for some integer T ⩾ 4 and for all pairs ( u, v ) with d ( u, v ) = 1 or d ( u, v ) = 2 . Therefore, both

and

can be assigned by a specific 2-DRFWL(2) test q to W ( T ) ( u, u ) for some T ⩾ 5 and for all nodes u . To prove the lemma it now suffices to show that

can be assigned by a 2-DRFWL(2) test to W ( T ) ( u, u ) for some T ⩾ 5 and for all nodes u . We calculate #( c ) as following,

̸

Here #( c , u = z ) is the count of substructure obtained by coalescing nodes u, z in Figure 6c, while keeping x = v . The counts #( c , x = v ) , #( c , x = z ) and #( c , u = z, x = v ) can be understood analogously, but one needs to notice that when counting #( c , x = v ) we keep u = z . We have

̸

and

By Lemmas C.7, C.17 and C.18, after summing over all nodes v ∈ N 1 ( u ) ∪N 2 ( u ) , each of the terms in (94) can be assigned by a 2-DRFWL(2) test to W ( T ) ( u, u ) , for some T ⩾ 5 and for all nodes u . Therefore, we have finally reached the end of the proof.

We are now in a position to give the proofs for Theorems 4.4, 4.5, 4.6 and 4.7.

We restate Theorem 4.4 as following.

Theorem C.20. 2-DRFWL(2) GNNs can node-level count 2, 3, 4-paths.

Proof. By Lemma C.5, it suffices to prove that for each k = 2 , 3 , 4 , there exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any node u ∈ V G , q assigns

for some integer T .

In the following proofs, we no longer care the exact value of T ; instead, we are only interested in whether a graph property can be calculated by a 2-DRFWL(2) test, i.e. there exists a finite integer T and a subset of 2-DRFWL(2) colors (for example, W ( T ) ( u, u ) , u ∈ V G for node-level properties, or W ( T ) ( u, v ) , d ( u, v ) ⩽ 2 for pair-level properties) that the subset of 2-DRFWL(2) colors gives exactly the values of the graph property, at the T -th iteration.

For k = 2 ,

But by Lemma C.7, there exists a 2-DRFWL(2) test q and an integer T such that (39) holds for all pairs ( u, v ) with d ( u, v ) = 1 or d ( u, v ) = 2 . Therefore, at the ( T + 1) -th iteration, q can assign W ( T +1) ( u, u ) = P 2 ( u ) using (100).

For k = 3 ,

Using Lemma C.10 and a similar argument to above, the first two terms of (101) can be calculated by a 2-DRFWL(2) test. For the last term of (101), we notice that

By Lemma C.14, W 3 ( u ) can be calculated by a 2-DRFWL(2) test. From the proof of Lemma C.10, one can see that W 3 ( u, v ) can be calculated by a 2-DRFWL(2) test, for d ( u, v ) = 1 or d ( u, v ) = 2 . Therefore, the last two terms of (102) can also be calculated by a 2-DRFWL(2) test. Since W 3 ( u, u ) = 2 C 3 ( u ) , it can be seen from Lemma C.11 that this term can also be calculated by a 2-DRFWL(2) test. Therefore, we conclude that ∑ v ∈N 3 ( u ) P 3 ( u, v ) , thus P 3 ( u ) , can be calculated by a 2-DRFWL(2) test.

For k = 4 ,

Using Lemma C.12 and a similar argument to above, the first two terms of (103) can be calculated by a 2-DRFWL(2) test. Like in the k = 3 case, we treat the last two terms of (103) as following,

By Lemmas C.13 and C.14, the terms W 4 ( u ) , ∑ v ∈N 1 ( u ) W 4 ( u, v ) and ∑ v ∈N 2 ( u ) W 4 ( u, v ) can be calculated by 2-DRFWL(2) tests. Moreover, it is easy to verify that

Therefore, by Lemmas C.7, C.8 and C.15, W 4 ( u, u ) can also be calculated by a 2-DRFWL(2) test. Summarizing the results above, we have proved that P 4 ( u ) can be calculated by a 2-DRFWL(2) test.

We restate Theorem 4.5 as following.

Theorem C.21. 2-DRFWL(2) GNNs can node-level count 3, 4, 5, 6-cycles.

Proof. By Lemma C.5, it suffices to prove that for each k = 3 , 4 , 5 , 6 , C k ( u ) can be calculated by a 2-DRFWL(2) test. From Lemmas C.11, C.15, C.16 and C.19, the above statement is true.

We restate Theorem 4.6 as following.

Theorem C.22. 2-DRFWL(2) GNNs can node-level count tailed triangles, chordal cycles and triangle-rectangles.

Proof. By Lemmas C.17 and C.18, and using a similar argument to above, it is easy to prove that 2-DRFWL(2) GNNs can node-level count chordal cycles and triangle-rectangles. To see why 2DRFWL(2) GNNs can node-level count tailed triangles, we only need to give a 2-DRFWL(2) test q that assigns

for some integer T . Actually, we have

or in a symmetric form,

Using this formula, along with Lemmas C.7, C.8 and C.11, it is straightforward to construct the 2-DRFWL(2) test q that satisfies (106), as long as T ⩾ 3 . Therefore, 2-DRFWL(2) GNNs can also node-level count tailed triangles.

We restate Theorem 4.7 as following.

Theorem C.23. 2-DRFWL(2) GNNs cannot graph-level count more than 7-cycles or more than 4-cliques.

Proof. The theorem is a direct corollary of Theorem 3.2. In the remark at the end of Appendix B.2, we have shown that no d -DRFWL(2) test can distinguish between two k -cycles and a single 2 k -cycle, as long as k ⩾ 3 d +1 . For d = 2 , we assert that no 2-DRFWL(2) test (thus no 2-DRFWL(2) GNN) can distinguish between two k -cycles and a 2 k -cycle, as long as k ⩾ 7 . Therefore, 2-DRFWL(2) GNNs cannot graph-level count more than 7-cycles.

Moreover, Theorem 4.2 of [56] states that for any k ⩾ 2 , there exists a pair of graphs with different numbers of ( k +1) -cliques that FWL( k -1 ) fails to distinguish between. Therefore, we assert that FWL(2) cannot graph-level count more than 4-cliques. By Theorem 3.2, the 2-DRFWL(2) tests (thus 2-DRFWL(2) GNNs) are strictly less powerful than FWL(2) in terms of the ability to distinguish between non-isomorphic graphs. Therefore, 2-DRFWL(2) GNNs cannot graph-level count more than 4-cliques.

C.3 Proof of Theorem 4.8

Before proving the theorem, we state and prove a series of lemmas.

Lemma C.24. There exists a 3-DRFWL(2) test q such that for any graph G ∈ G and for any 2-tuple ( u, v ) ∈ V 2 G with 1 ⩽ d ( u, v ) ⩽ 3 , q assigns

for some integer T .

Proof. If 1 ⩽ d ( u, v ) ⩽ 2 , the result follows from Lemma C.10 since 2-DRFWL(2) tests constitute a subset of 3-DRFWL(2) tests. If d ( u, v ) = 3 , then equations (50)-(54) still hold (with the value of k changed to 3), giving rise to

The RHS of (110) is obviously calculable by 3-DRFWL(2) tests. Therefore, the lemma holds.

Lemma C.25. There exists a 3-DRFWL(2) test q such that for any graph G ∈ G and for any 2-tuple ( u, v ) ∈ V 2 G with 1 ⩽ d ( u, v ) ⩽ 3 , q assigns

for some integer T .

Proof. If 1 ⩽ d ( u, v ) ⩽ 2 , the result follows from Lemma C.12. If d ( u, v ) = 3 , then equation (63) still holds, giving rise to

The definitions of P 2 , 2 ( u, v ) and P ij 2 , 2 ( u, v ) are given in (57) and (65), respectively. In particular, we point out that

is calculable by 3-DRFWL(2) tests, for any 1 ⩽ i, j ⩽ 2 and for d ( u, v ) = 3 . Therefore, the lemma holds.

Lemma C.26. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any 2-tuple ( u, v ) ∈ V 2 G with 1 ⩽ d ( u, v ) ⩽ 2 , q assigns

for some integer T 0 .

Proof. The lemma follows from

Lemma C.27. There exist two 2-DRFWL(2) tests q 1 , q 2 such that for any graph G ∈ G and for any 2-tuple ( u, v ) ∈ V 2 G with 1 ⩽ d ( u, v ) ⩽ 2 , q 1 , q 2 assign

respectively, for some integers T 1 and T 2 .

Proof. The existence of q 1 follows from the proof of Lemma C.17. To see why 2-DRFWL(2) tests can count CC 2 ( u, v ) , notice that

and the result follows from Lemma C.7.

Due to the relations (32) and (33), we assert that 2-DRFWL(2) tests can also node-level count CC 1 ( u ) and CC 2 ( u ) .

Lemma C.28. There exist three 2-DRFWL(2) tests q 1 , q 2 and q 3 such that for any graph G ∈ G and for any node u ∈ V G , q 1 , q 2 and q 3 assign

respectively, for some integers T 1 , T 2 and T 3 .

Proof. From the proof of Lemma C.18 we see that 2-DRFWL(2) tests can count TR 1 ( u, v ) . By (34), 2-DRFWL(2) tests can count TR 1 ( u ) and TR 2 ( u ) . We only need to prove that 2-DRFWL(2) tests can count TR 3 ( u ) .

Notice that

Therefore, by Lemmas C.7, C.26 and C.27, the lemma is true.

Lemma C.29. There exists a 2-DRFWL(2) test q such that for any graph G ∈ G and for any 2-tuple ( u, v ) ∈ V 2 G with 1 ⩽ d ( u, v ) ⩽ 2 , q assigns

Therefore, the lemma follows from Lemmas C.7, C.10 and C.26.

Now, we turn to proving Theorem 4.8. We restate the theorem as following.

Theorem C.30. For any d ⩾ 3 , d -DRFWL(2) GNNs can node-level count 3, 4, 5, 6, 7-cycles, but cannot graph-level count any longer cycles.

Proof. We first prove the negative result: for any d ⩾ 3 , d -DRFWL(2) GNNs cannot graph-level count k -cycles, with k ⩾ 8 . This is because in terms of the ability to distinguish between nonisomorphic graphs, d -DRFWL(2) tests (and thus GNNs) are strictly less powerful than FWL(2), for any d , as shown in Theorem 3.2. Nevertheless, even the more powerful FWL(2) tests fail to graph-level count k -cycles with k ⩾ 8 . [4]

Next, we will prove the positive result. Notice that the update rule of ( d +1) -DRFWL(2) GNNs always encompasses that of d -DRFWL(2) GNNs. Therefore, for any d ⩾ 3 , the node-level cycle for some integer T .

Proof. Notice that

counting power of d -DRFWL(2) GNNs is at least as strong as that of 2-DRFWL(2) GNNs. This implies that d -DRFWL(2) GNNs can node-level count 3, 4, 5, 6-cycles for any d ⩾ 3 .

For the same reason, to prove that d -DRFWL(2) GNNs can node-level count 7-cycles for any d ⩾ 3 , it suffices to prove that 3-DRFWL(2) GNNs can do so. By Lemma C.5, this reduces to proving the existence of a 3-DRFWL(2) test q , such that for any graph G ∈ G and for any node u ∈ V G , q assigns

for some integer T .

We try to calculate C 7 ( u ) via

We assert that

where #( a ) , #( b ) , . . . , #( l ) refer to numbers of the substructures depicted in (a), (b), . . . , (l) of Figure 7, respectively. Since P 3 ( u, v ) and P 4 ( u, v ) are calculable by 3-DRFWL(2) tests (by Lemmas C.24 and C.25), it now suffices to show that 3-DRFWL(2) tests can calculate the counts #( a ) , #( b ) , . . . , #( l ) , summed over all v with 1 ⩽ d ( u, v ) ⩽ 3 .

v

u

(a)

v

u

(b)

v

u

(c)

v

u

(d)

v

u

(e)

v

u

(f)

v

u

(g)

v

u

(h)

v

u

(i)

v

u

(j)

v

u

(k)

v

u

(l)

For the rest of the proof, we express each of the counts #( a ) , #( b ) , . . . , #( l ) (after summing over { v ∈ V G : 1 ⩽ d ( u, v ) ⩽ 3 } ) in terms of numbers already known calculable by 3-DRFWL(2) tests. Since the enumeration procedure is rather tedious, we directly provide the results below.

-

v

4

:1

+

-

-

2

TR

3

w

)

C

5

(

u

∑

⩽

v

:1

2

∈N

⩽

⩽

d

(

u,v

)

∑

⩽

d

(

u,v

CC

(

u

)

,

∑

1

w

3

∈N

-

2

(

u, w

)

P

P

2

(

u, w

2

1

(

u

)

(

v

)

∩N

tailed triangle

2

)

, u, G

)

(

w,v

P

,

(129)

(130)

2

(

u

)

C

(

)

2

(

w,v

(

u

)

∩N

1

∑

)

v

:1

⩽

∑

d

(

u,v

)

⩽

3

#(

g

) =

(

v

)

P

)

Combining the above results, we conclude that the theorem holds.

D Comparison between d -DRFWL(2) and localized FWL(2)

As mentioned in Section 5 of the main paper, d -DRFWL(2) can be seen as a sparse version of FWL(2). Another representative of this class of methods (adding sparsity to FWL(2)) is the localized FWL(2) , proposed by Zhang et al. [58]. In [58], two instances of localized FWL(2) are provided, i.e., LFWL(2) and SLFWL(2). Different from d -DRFWL(2), both LFWL(2) and SLFWL(2) assign a color W ( u, v ) to every 2-tuple ( u, v ) ∈ V 2 G , for any given graph G . Therefore, their space complexity is equal to that of FWL(2), namely O ( n 2 ) . The major difference between LFWL(2)/SLFWL(2) and FWL(2) is in their update rules. Similar to FWL(2), both LFWL(2) and SLFWL(2) update W ( u, v ) using a multiset of the form { { ( W ( w,v ) , W ( u, w )) } } . However, for LFWL(2), we restrict w to be in N ( v ) ; for SLFWL(2), we restrict w to be in N ( u ) ∪ N ( v ) . Since restricting the range of w reduces the multiset size from O ( n ) (as in FWL(2)) to O (deg) , the time complexity of LFWL(2) and SLFWL(2) is O ( n 2 deg) , which is usually much lower than the O ( n 3 ) time complexity of FWL(2).

We now compare d -DRFWL(2) with LFWL(2)/SLFWL(2) in terms of discriminative power. We have the following results:

No d -DRFWL(2) with a fixed d value can be more powerful than LFWL(2) or SLFWL(2). This is because for any finite d , we can construct graph pairs that are separable by LFWL(2) and SLFWL(2) but not separable by d -DRFWL(2). One of such graph pairs can be two (3 d +1) -cycles and one (6 d +2) -cycle, between which d -DRFWL(2) cannot discriminate (as shown in the proof of Theorem 3.2). However, since both LFWL(2) and SLFWL(2) can calculate distance between every pair of nodes (following a procedure constructed in the proof of Theorem 3.2), it is easy for LFWL(2) or SLFWL(2) to distinguish between the aforementioned pair of graphs, since they have different diameters.

With sufficiently large d , d -DRFWL(2) is not less powerful than LFWL(2) or SLFWL(2). This is because for graphs with diameter ⩽ d , d -DRFWL(2) has equal power to FWL(2). Since it is shown in [58] that there exist graph pairs separable by FWL(2) but not separable by LFWL(2) or SLFWL(2), once the value of d grows larger than the diameters of such graph pairs, those graph pairs can be separated by d -DRFWL(2) but not LFWL(2) or SLFWL(2).

Therefore, for sufficiently large d , the discriminative power of d -DRFWL(2) is neither stronger nor weaker than LFWL(2) or SLFWL(2). However, it remains unknown the relation in discriminative power between LFWL(2)/SLFWL(2) and d -DRFWL(2) with a practical d value, such as d = 2 or d = 3 . Are d -DRFWL(2) with those smaller d values less powerful than LFWL(2)/SLFWL(2), or are they incomparable? We leave this question open for future research.

E Experimental details

E.1 Model implementation

Let G be a graph with node features f u , u ∈ V G and edge features e uv , { u, v } ∈ E G . In most of our experiments, we implement d -DRFWL(2) GNNs as following: we generate h (0) uv as

where LIN i , i = 0 , 1 , . . . , d are linear functions. When node and edge features are absent, we assign identical values to h (0) uv with the same d ( u, v ) .

The m ( t ) ijk and f ( t ) k functions in (10) and (11) are chosen as

where LIN ( t ) in (137) is a linear module whose parameters are shared among all i, j, k combinations ; LIN ( t ) { { i,j,k } } in (138) is a linear module whose parameters are shared among all i, j, k that form the same multiset { { i, j, k } } (for example, the contributions a 122 uv , a 212 uv and a 221 uv are all linearly transformed by an identical LIN ( t ) { { 1 , 2 , 2 } } ); MLP ( t ) k is a multilayer perceptron, and can depend on k ; ϵ can be a constant or a learnable parameter. After the d -DRFWL(2) GNN layers, we apply a sum-pooling layer

and then a final MLP.

We point out that our implementation of d -DRFWL(2) GNNs has the property h ( t ) uv = h ( t ) vu , for any u, v ∈ V G with 0 ⩽ d ( u, v ) ⩽ d , and for any t = 0 , 1 , . . . , T . We call an instance of d -DRFWL(2) GNN symmetrized , if it preserves the above property. It is obvious that symmetrized d -DRFWL(2) GNNs only constitute a subspace of the function space of all d -DRFWL(2) GNNs.

Nevertheless, we remark that for the case of d = 2 , symmetrized 2-DRFWL(2) GNNs have equal node-level cycle counting power to 2-DRFWL(2) GNNs . Actually, in the proofs of Lemmas C.7C.19 and Theorems 4.4-4.6 (see Appendix C for details), all the 2-DRFWL(2) tests we constructed have the property W ( t ) ( u, v ) = W ( t ) ( v, u ) , for any u, v ∈ V G with 0 ⩽ d ( u, v ) ⩽ 2 and any t ; on the other hand, it is easy to establish the equivalence between such kind of 2-DRFWL(2) tests and symmetrized 2-DRFWL(2) GNNs. Similar facts can be verified for d -DRFWL(2) GNNs with d = 1 or d = 3 . Therefore, at least for all our experiments (where d ⩽ 3 ), our symmetrized implementation of d -DRFWL(2) GNNs does no harm to their theoretical cycle counting power, although the model is greatly simplified.

E.2 Experimental settings

E.2.1 Substructure counting

Datasets. The synthetic dataset is provided by open-source code of GNNAK on github. The node-level substructure counts are calculated by simple DFS algorithms, which we implement in C language. (This part of code is also available in our repository.)

Models. Implementations of all baseline methods (MPNN, ID-GNN, NGNN, GNNAK+, PPGN and I 2 -GNN) follow [33]. For d -DRFWL(2) GNN ( d = 1 , 2 , 3 ), we use 5 d -DRFWL(2) GNN layers. In each layer (numbered by t = 1 , . . . , 5 ), MLP ( t ) k is a 2-layer MLP, for k = 0 , 1 , . . . , d . (See (138) for the definition of MLP ( t ) k .) The embedding size is 64.

Training settings. Weuse Adam optimizer with initial learning rate 0.001, and use plateau scheduler with patience 10, decay factor 0.9 and minimum learning rate 10 -5 . We train our model for 2,000 epochs. The batch size is 256.

E.2.2 Molecular property prediction

Datasets. The QM9 and ZINC datasets are provided by PyTorch Geometric package [22]. The ogbg-molhiv and ogbg-molpcba datasets are provided by Open Graph Benchmark (OGB) [31]. We rewrite preprocessing code for all four datasets.

The training/validation/test splitting for QM9 is 0.8/0.1/0.1. The training/validation/test splittings for ZINC, ogbg-molhiv and ogbg-molpcba are provided in the original releases.

Models. For QM9, we adopt a 2-DRFWL(2) GNN with 5 2-DRFWL(2) GNN layers. In each layer (numbered by t = 1 , . . . , 5 ), MLP ( t ) k is a 2-layer MLP, for k = 0 , 1 , 2 . The embedding size is 64.

For ZINC, we adopt a simplified version of 3-DRFWL(2) GNN, which only adds terms a 313 uv , a 133 uv and a 331 uv to the update rules (10) and (11) of a 2-DRFWL(2) GNN. Notice that the adopted model architecture is slightly different from the one described in Appendix E.1. This is because we observe that including other kinds of message passing in 3-DRFWL(2) GNN does not improve the performance on ZINC. The adopted 3-DRFWL(2) GNN has 6 3-DRFWL(2) GNN layers. In each layer (numbered by t = 1 , . . . , 6 ), MLP ( t ) k is a 2-layer MLP, for k = 0 , 1 , 2 , 3 . The embedding size is 64. We apply batch normalization between every two 3-DRFWL(2) GNN layers.

For ogbg-molhiv, we also adopt a simplified version of 2-DRFWL(2) GNN; namely, we remove the term a 222 uv in the update rules of the original 2-DRFWL(2) GNN. The adopted 2-DRFWL(2) GNN has 5 2-DRFWL(2) GNN layers. In each layer (numbered by t = 1 , . . . , 5 ), MLP ( t ) k is a 2-layer MLP, for k = 0 , 1 , 2 . The embedding size is 300. We apply a dropout layer with p = 0 . 2 after every 2-DRFWL(2) GNN layer.

For ogbg-molpcba, we adopt a 2-DRFWL(2) GNN with 3 2-DRFWL(2) GNN layers. In each layer (numbered by t = 1 , 2 , 3 ), MLP ( t ) k is a 2-layer MLP, with layer normalization applied after every MLP layer, for k = 0 , 1 , 2 . The embedding size is 256. We apply layer normalization between every two 2-DRFWL(2) GNN layers, and also apply a dropout layer with p = 0 . 2 after every 2-DRFWL(2) GNN layer.

Training settings. For QM9, we use the Adam optimizer, and use plateau scheduler with patience 10, decay factor 0.9 and minimum learning rate 10 -5 . For each of the 12 targets on QM9, we search hyperparameters from the following space: (i) initial learning rate ∈ { 0 . 001 , 0 . 002 , 0 . 005 } ; (ii) whether to apply layer normalization between every two 2-DRFWL(2) GNN layers (yes/no). We train our model for 400 epochs. The batch size is 64.

For both ZINC-12K and ZINC-250K, we use Adam optimizer with initial learning rate 0.001. For ZINC-12K, we use plateau scheduler with patience 20, decay factor 0.5 and minimum learning rate 10 -5 ; for ZINC-250K, we use plateau scheduler with patience 25, decay factor 0.5 and minimum learning rate 5 × 10 -5 . We train our model for 500 epochs on ZINC-12K and 800 epochs on ZINC-250K. The batch size is 128.

For both ogbg-molhiv and ogbg-molpcba, we use Adam optimizer with initial learning rate 0.0005. For ogbg-molhiv, we use step scheduler with step size 20 and decay factor 0.5. We train our model for 100 epochs and the batch size is 64. For ogbg-molpcba, we use step scheduler with step size 10 and decay factor 0.8. We train our model for 150 epochs and the batch size is 256.

F Additional experiments

Experiments that answer Q4 in Section 6 of the main paper;

Ablation studies on the substructure counting power of 2-DRFWL(2) GNNs;

Additional experiments that study the performance of d -DRFWL(2) GNNs on molecular datasets;

Experiments that study the ability of d -DRFWL(2) GNNs to capture long-range interactions;

Additional experiments that study the cycle counting power and empirical efficiency of 2-DRFWL(2) GNNs on larger graphs.

F.1 Discriminative power

Datasets. To answer Q4 , we evaluate the discriminative power on three synthetic datasets: (1) EXP [1], containing 600 pairs of non-isomorphic graphs that cannot be distinguished by the WL(1) test; (2) SR25 [6], containing 15 non-isomorphic strongly regular graphs that the FWL(2) test fails to distinguish; (3) BREC [51], containing 400 pairs of non-isomorphic graphs generated from a variety of sources.

For EXP, we follow the evaluation process in [59], use 10-fold cross validation, and report the average binary classification accuracy; for SR25, we follow [60], treat the task as a 15-way classification, and report the accuracy. Raw data for both EXP and SR25 datasets are provided by open-source code of GNNAK on github.

For BREC, we use Reliable Paired Comparison (RPC) as the evaluation method, following [51]. For every graph pair ( G,H ) , the RPC procedure consists of two stages: major procedure and reliability check .

In the major procedure, we generate q = 32 copies ( G i , H i ) of ( G,H ) , i = 1 , . . . , q . The copies G i and H i ( i = 1 , . . . , q ) are obtained by randomly relabeling nodes in G and H , respectively. We then apply a Hotelling's T 2 test to check whether we should reject the null hypothesis H 0 : E relabel [ f ( G ) -f ( H )] = 0 , meaning that the model f cannot distinguish between G and H .

In the reliability check, we replace H i in each copy ( G i , H i ) with G π i , which is obtained by randomly relabeling nodes in G i , for i = 1 , . . . , q . We then apply a second T 2 test to check whether we should reject the null hypothesis H ′ 0 : E relabel [ f ( G ) -f ( G π )] = 0 , meaning that the representations of two isomorphic graphs will not deviate too much from one another due to numerical errors.

Finally, G and H are considered separable by f iff we should reject H 0 but should not reject H ′ 0 . The detailed description of the RPC procedure is provided in [51].

Baselines. For EXP and SR25, baseline methods are chosen from: (1) Basic MPNNs. These methods have discriminative power upper-bounded by the WL(1) test, and we choose GIN [55], which can achieve WL(1) expressive power theoretically. (2) Subgraph MPNNs. These methods are strictly more powerful than WL(1) but strictly upper-bounded by the FWL(2) test [58]. We choose Nested GIN [59] and GIN-AK+ [60]. (3) Higherorder GNNs with expressive power equal to FWL(2). We choose PPGN [42] and 3-GCN [1]. (4) Methods with discriminative power partially stronger than FWL(2), e.g. I 2 -GNN [33].

For BREC, the complete list of baseline results is given in Table 2 of [51]. We select 3-WL, NGNN, NGNN with distance encoding (DE), SUN [24], SSWL_P [58], GNNAK, I 2 -GNN and PPGN as our baselines.

Models. We evaluate both 2-DRFWL(2) GNN and 3-DRFWL(2) GNN on the three datasets. On all three datasets, we adopt a d -DRFWL(2) GNN with 5 d -DRFWL(2) GNN layers ( d = 2 or 3 ). In each layer (numbered by t = 1 , . . . , 5 ), MLP ( t ) k is a 2-layer MLP, for k = 0 , . . . , d . The embedding size is 64 for EXP/SR25, and 32 for BREC.

Training settings. For both EXP and SR25 datasets, we use Adam optimizer with learning rate 0.001. We train our model for 10 epochs on EXP and 100 epochs on SR25. The batch size is 20 on EXP and 64 on SR25. For BREC, we use Adam optimizer with learning rate 0.0001. We train for 10 epochs and the batch size is 32.

Results. We can see from Table 5 that 2-DRFWL(2) GNN already achieves 99.8% accuracy on the EXP dataset, while 3-DRFWL(2) GNN achieves 100% accuracy. This complies with our theoretical result (Theorem 3.1) that d -DRFWL(2) GNNs are strictly more powerful than WL(1); moreover, 3-DRFWL(2) GNN achieves higher accuracy than 2-DRFWL(2) GNN, verifying the expressiveness hierarchy we obtain in Theorem 3.2. Table 5 also shows that both 2-DRFWL(2) GNN and 3-DRFWL(2) GNN fail on SR25 with a 1/15 accuracy. This verifies our assertion that d -DRFWL(2) GNNs are strictly less powerful than FWL(2).

The experimental results on BREC are shown in Table 6. Since the 400 graph pairs in BREC are categorized into four classes-basic graphs (60 pairs), regular graphs (140 pairs), extension graphs (100 pairs) and CFI graphs (100 pairs), we report the number of distinguishable graph pairs in each class, and also the total number. From Table 6, it is easy to observe the expressiveness gap between d -DRFWL(2) and 3-WL (which has equal discriminative power to FWL(2)), and the gap between 2-DRFWL(2) and 3-DRFWL(2). This again verifies Theorem 3.2.

F.2 Ablation studies on substructure counting

Westudy the impact of different kinds of message passing in 2-DRFWL(2) GNNs on their substructure (especially, cycle) counting power, by comparing the full 2-DRFWL(2) GNN with modified versions in which some kinds of message passing are forbidden. The dataset, model hyperparameters and training settings are the same as those in Appendix E.2.1.

Results. From Table 7, we see that by removing the term a 222 uv in (11) (i.e., by forbidding a distance2 tuple ( u, v ) to receive messages from other two distance-2 tuples ( u, w ) and ( w,v ) ), we make 2-DRFWL(2) GNN unable to node-level count 6-cycles. This complies with our intuitive discussion at the end of Section 4 that such kind of message passing is essential for identifying closed 6-walks, and thus for counting 6-cycles.

Similarly, if we further remove a 122 uv , a 212 uv and a 221 uv , then 2-DRFWL(2) GNN will be unable to count 5-cycles. If a 121 uv , a 211 uv and a 112 uv are also removed, the 2-DRFWL(2) GNN actually degenerates into 1-DRFWL(2) GNN, and becomes unable to count 4-cycles and triangle-rectangles (the latter also containing a 4-cycle as its subgraph). Finally, we can see from Table 7 that even 1-DRFWL(2) GNN can count 3-cycles, tailed triangles and chordal cycles, since the numbers of these substructures only depend on the number of closed 3-walks.

We also study the effect of distance-0 tuples in 2-DRFWL(2) GNN. Removing embeddings for those tuples does not affect the theoretical counting power of 2-DRFWL(2) GNN, but we observe that the empirical performance on counting tasks slightly drops, possibly due to optimization issues.

F.3 Additional experiments on molecular datasets

In the following, we present the results on ZINC, ogbg-molhiv and ogbg-molpcba datasets.

F.4 Long-range interactions

In 6.2 of the main paper, we mention that 2-DRFWL(2) GNN greatly outperforms subgraph GNNs like NGNN or I 2 -GNN on the targets U 0 , U, H and G of the QM9 dataset, a phenomenon we ascribe to the stronger ability of 2-DRFWL(2) GNN to capture long-range interactions. Actually, a bit knowledge from physical chemistry tells us that the targets U 0 , U, H and G not only depend on the chemical bonds between atoms, but also the intermolecular forces (such as hydrogen bonds). Such intermolecular forces can exist between atoms that are distant from each other on the molecular graph (since the two atoms belong to different molecules). However, most subgraph GNNs process the input graphs by extracting k -hop subgraphs around each node, usually with a small k . This operation thus prevents information from propagating between nodes that are more than k -hop away from one another, and makes subgraph GNNs fail to learn the long-range interactions that are vital for achieving good performance on the targets U 0 , U, H and G . In contrast, 2-DRFWL(2) GNN directly performs message passing between distance-restricted 2-tuples on the raw graph , resulting in its capability to capture such long-range interactions.

So far, it remains an interesting question whether 2-DRFWL(2) GNNs' ability to capture long-range interactions is comparable with that of globally expressive models, such as k -GNNs [44] or GNNs based on SSWL, LFWL(2) or SLFWL(2) [58]. Those models keep embeddings for all possible k -tuples v ∈ V k G (instead of only distance-restricted ones). In this sense, they treat the input graphs as fully-connected ones, and presumably have stronger capability to learn long-range dependencies than 2-DRFWL(2) GNNs. We verify this assertion by comparing the performance of 2-DRFWL(2)

GNN on QM9 with that of 1-2-GNN, 1-2-3-GNN, SSWL+ GNN, LFWL(2) GNN and SLFWL(2) GNN, among which the first two are proposed by Morris et al. [44], and the others by Zhang et al. [58]. We also include the result of 1-GNN for reference.

We present the results in Table 9. The baseline results for 1-GNN, 1-2-GNN and 1-2-3-GNN are from [44]. For experiments on SSWL+ GNN, LFWL(2) GNN and SLFWL(2) GNN, we implement the three models following the style of PPGN [42]; namely, we use a B × d × n max × n max matrix to store the embeddings of all 2-tuples within a batch, with B being the batch size, d being the size of the embedding dimension, and n max being the maximal number of nodes among all graphs in the batch.

For each of SSWL+, LFWL(2) and SLFWL(2), we use 5 layers with embedding size 64, and apply layer normalization after each layer. We train for 400 epochs using Adam optimizer with initial learning rate searched from { 0 . 001 , 0 . 002 } , and plateau scheduler with patience 10, decay factor 0.9 and minimum learning rate 10 -5 . The batch size is 64. We report the best MAE (mean absolute error) for each target. The experimental details for 2-DRFWL(2) GNN follow Appendix E.2.2.

From Table 9, we see that although 2-DRFWL(2) GNN outperforms all baseline methods on 8 out of the 12 targets, its performance is inferior to globally expressive models on the targets U 0 , U, H and G . This result implies that the ability of 2-DRFWL(2) GNN to capture long-range interactions is weaker than globally expressive models.

We also evaluate the performance of 2-DRFWL(2) GNN on the Long Range Graph Benchmark (LRGB) [21]. LRGB is a collection of graph datasets suitable for testing GNN models' power to learn from long-range interactions. We select the two graph-level property prediction datasets, Peptides-func and Peptides-struct , from LRGB. The task of Peptides-func is multilabel binary classification, and the average precision (AP) is adopted as the metric. The task of Peptides-struct is multi-label regression, and the mean absolute error (MAE) is adopted as the metric.

For experiments on both datasets, we use 5 2-DRFWL(2) GNN layers with hidden size 120. We train 200 epochs using Adam optimizer with initial learning rate 0.001, and plateau scheduler with patience 10, decay factor 0.5, and minimal learning rate 10 -5 . The batch size is 64. We also list all baseline results given in [21], including MPNNs like GCN [36], GCNII [13], GINE [30, 55] and GatedGCN [11], as well as Transformer-based models like fully connected Transformer [49] with Laplacian PE (LapPE) [19, 20] and SAN [38]. The results are shown in Table 10.

From Table 10, we see that 2-DRFWL(2) GNN outperforms almost all MPNNs. Surprisingly, on Peptides-struct the performance of 2-DRFWL(2) GNN is even comparable to Transformer-based methods (which inherently capture long-range information by treating the graph as fully connected). Therefore, we conclude that although designed to capture local structural information, our model still possesses the ability to learn long-range dependencies.

F.5 Additional experiments on cycle counting power and scalability

Datasets. To further compare the cycle counting power and scalability of d -DRFWL(2) GNNs (especially for the case of d = 2 ) with other powerful GNNs, we perform node-level cycle counting tasks on two protein datasets, ProteinsDB and HomologyTAPE, collected from [29]. We evaluate the performance and empirical efficiency of 2-DRFWL(2) GNN as well as baseline models such as MPNN, NGNN, I 2 -GNN and PPGN, on both datasets.

The official code of [29] provides raw data for three protein datasets-'Enzymes vs Non-Enzymes', 'Scope 1.75' and 'Protein Function'. All three datasets contain graphs that represent protein molecules, in which nodes represent amino acids, and edges represent the peptide bonds or hydrogen bonds between amino acids. Each of the three datasets contains a number of directed graphs, in which only a small portion of edges are directed.

We collect the first two datasets from the three and convert all graphs to undirected graphs by adding inverse edges for the directed edges. Since we are only interested in cycle counting tasks, we remove node features (e.g., amino acid types, spatial positions of amino acids) and edge features (e.g., whether the bond is a peptide bond or a hydrogen bond). The node-level cycle counts are generated by the same method as the one described in Appendix E.2.1. We then rename the two processed datasets as ProteinsDB and HomologyTAPE respectively. (The new names of the datasets correspond to their original .zip file names.)

Some statistics of the two processed datasets are shown in Table 11. We also show the statistics of some other datasets we use in Table 12. From Table 11 and Table 12 we can see that the graphs in ProteinsDB and HomologyTAPE datasets are much larger than those in QM9, ZINC, ogbg-molhiv, ogbg-molpcba or the synthetic dataset, thus suitable for evaluating the scalability of GNN models.

For ProteinsDB, we use 10-fold cross validation and report the average normalized MAE for each counting task. We use the splitting provided in the raw data.

For HomologyTAPE, there are three test splits in the raw data, called 'test_fold', 'test_family' and 'test_superfamily' respectively. The training/validation/test_fold/test_family/test_superfamily splitting is 12,312/736/718/1,272/1,254. We report the weighted average normalized MAE on the three test splits for each counting task.

Models. We adopt MPNN, NGNN, I 2 -GNN and PPGN as our baselines. The implementation details of 2-DRFWL(2) GNN as well as all baseline methods follow Appendix E.2.1, except that (i) for NGNN and I 2 -GNN, the subgraph height is 3 for all tasks, and (ii) for PPGN, the number of PPGN layers is 5 and the embedding size is 64.

Training settings. Weuse Adam optimizer with initial learning rate 0.001, and use plateau scheduler with patience 10, decay factor 0.9 and minimum learning rate 10 -5 . We train all models for 1,500 epochs on ProteinsDB and 2,000 epochs on HomologyTAPE. The batch size is 32.

Results. From Table 13, we see that 2-DRFWL(2) GNN achieves relatively low errors (average normalized MAE < 0 . 002 for ProteinsDB, and < 0 . 0002 for HomologyTAPE) for counting all 3, 4, 5 and 6-cycles on both datasets. This again verifies our theoretical result (Theorem 4.5). Although I 2 -GNN and PPGN are also provably capable of counting up to 6-cycles, PPGN fails to scale to either ProteinsDB or HomologyTAPE dataset, while I 2 -GNN fails to scale to ProteinsDB. Therefore, 2-DRFWL(2) GNN maintains good scalability compared with other existing GNNs that can count up to 6-cycles.

We also measure (i) the maximal GPU memory usage during training, (ii) the preprocessing time, and (iii) the training time per epoch, for all models on both datasets. The result is shown in Table 14. We again see that 2-DRFWL(2) GNN uses much less GPU memory while training, compared with subgraph GNNs. Besides, the training time of 2-DRFWL(2) GNN is shorter than that of I 2 -GNN, while the preprocessing time is comparable.

From Table 14, we also see one limitation of our implementation of 2-DRFWL(2) GNN, which we have mentioned in Section 7-the preprocessing of 2-DRFWL(2) GNN is not efficient enough on larger graphs or graphs with a larger average degree. We believe this limitation can be addressed by

developing more efficient, parallelized operators for the preprocessing of 2-DRFWL(2) GNN, and we leave the exploration of such implementations to future work.

Peer reviews:
comments: SUMMARY:
The paper proposes a version of the 2-dimensional Weisfeiler-Leman algorithm that restricts the distance between nodes in the node tuples processed. A GNN based on this restriction is also proposed and investigated regarding its ability to count cycles and other substructures.

STRENGTHS:
**S1** In general, the paper is well written and clearly structured.

**S2** The idea to restrict the distance is interesting and seems to work reasonably well for counting substructures.

WEAKNESSES:
**W1** The description of the main contribution d-DRFWL(2) could be improved, making the illustration clearer. The general concept behind d-DRFWL(2) could be explained better. What exactly is updated with what information? The formula represents it, but the corresponding text and the example (Figure 1) could be more precise. It does not help that the figure is colored and the colors are referenced. I cannot distinguish red/violet/pink here.

**W2** The theoretical investigation can be extended to show how the approach relates to k-WL and $\delta$-k-LWL.

**W3** More benchmark datasets should be used for evaluation.

QUESTIONS:
**Q1** Is the approach as expressive as FWL(2) theoretically?

**Q2** How does the proposed approach fit into the k-WL/ $\delta$-k-LWL hierarchy?

**Q3** Can you state the differences and connections to $\delta$-k-LWL more clearly?

**Q4** In d-DRFWL(k) what happens for larger values of d and k for theoretical expressiveness, running time and classification accuracy?

### Other remarks:
The spacing between the lines is off at some places (for example l108/109 where characters from different lines overlap).

LIMITATIONS:
To me the theoretical expressiveness did not get discussed to the right extend (see Q2).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with constructing expressive GNNs (i.e., more expressive than 1-WL) with low complexity. The authors focus on cycle count ability, motivated by their importance in real life chemical datasets. With this problem and motivation at hand, the authors propose a family of algorithms possessing a strict hierarchy property called $d$-distcne restricted FWL or $d$-DRFWL(2).  They identify the ability to count cycles in the tuple based coloring and aggregation of the $2$-FWL algorithm compared to MPNNs which are bounded by $1$-WL and cannot even count a 3-cycle, but aim to avoid the $O(n^2)$ and $O(n^3)$ memory and time complexity respectively of $2$-FWL. To that end, they propose only coloring 2-tuples $(u,v)$ which are far from each other by ad most $d$ and propose a distance-dependent aggregation, this algorithm is named $d$-DRFWL(2). They then show that already for $d=2$ the proposed algorithm can count up to 6-cycles and that increasing $d$ strictly increases discriminative power. The proposed algorithm is the most efficient known model that can count 6-cycles, with time and memory complexities of $O(n\\mathrm{deg}^4)$ and $O(n\\mathrm{deg}^2)$, respectively.\n\nSTRENGTHS:\nThe authors clearly present the problem, goal, and motivation for their work. \n\nOn the originality and novelty side, the authors design the most efficient model to this date that can count 6-cycles and show a thorough analysis of one of its instances, the $2$-DRFWL(2).\n\nDue to the importance of cycle counts in chemical and biological structures, I think that this work is valuable to that community.\n\nWEAKNESSES:\n- Although taking a step towards more efficient and expressive GNNs, the efficiency of the method is only when graphs are sparse enough. To my understanding, the proposed method still did not break the limit of running on graphs of sizes $O(100)$ (e.g, ogbg-ppa dataset).\n- Do the authors have any explanation for the large performance gap between their method and $I^2$-GNN on the QM9 dataset? Theoretically, they both can count the same sizes of cycles, so this clouds the motivation for the paper in my opinion, since cycle counting may not be what makes this method more successful.\n\nQUESTIONS:\n- Have the authors considered a baseline model where they add the cycle/substructure counts as features? Is the proposed method indeed better then the naive approach?\n- An ablation that I find missing is to see the empirical effect of $d$. Have the authors experimented with larger $d$ values?\n- How does the proposed hierarchy compare to LFWL(2) and SLFWL(2)? Due to this approaches being closely related, coloring 2-tuples in lower complexity, a more elaborate discussion would add to the work. Also, either theoretical or experimental comparison would be good.\n\nLIMITATIONS:\nThe authors discuss the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the task of counting or detecting substructures such as cycles or small cliques in graphs. This task is (theoretically and practically) impossible for standard GNNs. While higher-order GNNs are able to count such substructures, they are slow, using $n^2$ space and $n^3$ time which is infeasible for large graphs. The paper suggests to make such higher-order GNNs practical by only considering \"local\" pairs of nodes (in the case of a 2-GNN) as substructures such as cycles are inherently local too. The paper proves that such distance-restricted higher-order GNNs are able to count substructures and in particular distance 2 and a 2-GNN suffice to count 6-cycles. Experiments underline the strong substructure counting power of the suggested method while still achieving reasonable runtimes (in contrast to full 2-GNN).\n\nSTRENGTHS:\nStrong cycle counting power without being extremely slow: finally we have an efficient GNN that can not only count triangles (i have seen a few of that) but the much more complicated (and possibly interesting) 6-cycles. This holds both in theory and in practice. The authors provide a clear theory (I have to admit that I did not check the proofs in the appendix) and extensive experiments focusing on the main claim, namely the strong cycle and in general substructure counting power. In terms of efficiency I was surprised that the practical memory consumption is not far from what a standard MPGNN uses (even though in the molecular domain the maximum degree in each graph is typically 3 or 4 and thus not extremely large). Overall a very nice paper!\n\nWEAKNESSES:\nI don't have strong criticisms about the paper (but please see the questions below). Here is a list of things I noted while reading the paper, roughly in the order of appearance:\n- 14: How is subgraph extraction avoided when using DRFWL? The DR-k-FWL procedure somehow needs to operate on small induced subgraphs. \n- 48: I would say it is not obvious why 2-FWL encodes closed walks. It would be nice to have some additional intuition there\n- 62: Many graphs have small diameter, for those the distance restriction does not help too much. And in particular, on the complete graph the distance restriction trivially is useless. The \"deg\" dependency is only introduced in the next paragraph such that the statement that DRFWL(2) should have a lower complexity than 2-FWL is unclear as this obviously does not hold for the worst-case complexity.\n- the related work might also include methods that break the 1-WL barrier by aggregating from multiple distances at once (those methods often call themselves \"higher-order\" in the sense that they use a \"higher-order neighborhood\", often 2 or 3 hops)\n- At Q4: please make explicit that this is a question to be answered empirically (the theory result is already there)\n- How about long-range dependencies? (https://arxiv.org/abs/2206.08164)\n- It would be nice to mention that the model is not suitable for node classification as the corresponding graphs typically have relatively large degrees (at least compared to molecules).\n\n\nTypography:\n- Leman asked to be written without the h. This is why some older papers still use the additional h while most newer ones do not.\n- It might look nicer to use \\bm for the $d$ in section headings\n- I personally prefer it when equations are referenced as Eq. (6) instead of just (6). One way is to use \\cref throughout the document\n- 173: the \"one the other hand\" seems out of context\n- It would be nice to have author names in addition to a citation [99] when [99] acts as the subject of that sentence. This happens several times in the related work section\n- 51: := \\ell -> =: \\ell (as \\ell is to be defined)\n- 296: Similar -> A similar\n- Thanks a lot for proper proofreading!\n\nQUESTIONS:\n- There seems to be a subtle difference between equations 3,4 and equations 6,7. In particular 6,7 explicitly create $M_{i,j}$ for every distance pair $i,j$ while equation 3,4 just create a single multiset of sift operations (colors). Would you like to comment on that small difference? I have the feeling that this change may make the distance-restricted method (with d>diameter) stronger than pure FWL, although your theorem states that this is not the case - effectively the DR variant seems to individualizes one node.\n\n- is 2-DRFWL(2) well suited for long-range dependencies? https://arxiv.org/abs/2206.08164\n- How does it perform on molpcba (which is more commonly evaluated against than molhiv)?\n- Is there a simple intuitive reason why FWL(2) cannot count more than 7-cycles?\n\nLIMITATIONS:\nThe authors addressed the main limitation of the work, namely that it becomes a lot slower when the degree in a graph increases, limiting the applicability to mostly the molecular domain and keeping it from working in the context of node classification.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce the d-DRFWL(2) GNNs to count certain graph substructures, including cycles, cliques, and paths, which is essential for various graph-mining tasks. The authors prove that with d = 2, d-DRFWL(2) GNNs can already count up to 6 cycles, retaining most of the cycle counting power of FWL(2). Since the d-DRFWL(2) GNNs restrict the distance of message propagation, they are much more efficient than other GNN models. Experiments on both synthetic datasets and real molecular datasets are consistent with the theoretical results.\n\nSTRENGTHS:\nS1. The experiment is conducted on both real and synthetic datasets, where the results are convincing.\n\nS2. The theoretical results are sound and easy to follow.\n\nS3. The paper is clear and easy to follow.\n\nWEAKNESSES:\nW1. In my opinion, the scope of this particular line of research, which examines the expressive capabilities of GNNs by quantifying subgraph structures, appears to be rather limited. As stated in [1], GNNs are Turing complete (LOCAL algorithm) under certain conditions, indicating that they can effectively address any graph mining task with the same power as traditional algorithms. Consequently, this line of research primarily relies on observations made in previous graph mining algorithms and attempts to reinterpret them in the framework of GNNs, albeit within specific model designs, i.e., data structure for processing the graph. It would be beneficial if the authors could provide a comprehensive explanation detailing how their approach represents an advancement in utilizing GNNs to solve the subgraph counting problem, surpassing conventional algorithms [2].\n\nW2. Although the authors demonstrate that 2-DRFWL(2) can count 6-cycles, I am curious about the ability of 3-DRFWL(2) (or even larger $d$) to solve the cycle counting problem with larger subgraph sizes. If possible, it would be essential to understand the relationship between the minimal distance and the subgraph size. Additionally, exploring the impact of different values of K in the K-FWL test on the expressive power of GNNs would be interesting.\n\n[1] Loukas, A. (2019, September). What graph neural networks cannot learn: depth vs width. In International Conference on Learning Representations.\\\n[2] Ribeiro, P., Paredes, P., Silva, M. E., Aparicio, D., & Silva, F. (2021). A survey on subgraph counting: concepts, algorithms, and applications to network motifs and graphlets. ACM Computing Surveys (CSUR), 54(2), 1-36.\n\nQUESTIONS:\nQ1. There are other essential subgraph structure such as k-plex and k-core. I wonder whether the proposed framework can count this structures.\n\nQ2. The authors propose the existence of a clear hierarchy of expressiveness for d-DRFWL(2) GNNs as the value of $d$ increases. However, no experiments have been conducted to verify this claim across different values of $d$. Therefore, I would like to suggest the authors include a sensitivity test to support their theoretical findings.\n\nLIMITATIONS:\nPlease check the weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work explores the ability of graph neural network to count certain graph substructures, especially cycles. While past works counts by collecting subgraphs, the work avoids such burdensome procedure and construct a local method. Theoretical analysis are presented to show that Folklore Weisfeiler-Lehman test. Accordingly, the work proposes d-Distance-Restricted FWL(2) GNNs (FWL(2) being 2-dimensional Folklore Weisfeiler-Lehman), which incorporates the Folklore Weisfeiler-Lehman test algorithm into the aggregation steps of a GNN. Subsequent theoretical analysis and experiemental evaluations of d-Distance-Restricted FWL(2) GNNs are provided. In particular, experiment results show that the proposed GNN shows superior results on counting cycles or other structures (in synthetic datasets) and better performance graph regression task on chemical datasets, as well as being empirically more efficient.\n\nSTRENGTHS:\n- The problem of analyzing the ability of GNN to count substructures on a graph is interesting. Intuitively, counting cycles in a graph is non-intuitive for node-centric algorithm, where the operations solely performed on each node which only receives information from its neighbors at each iteration.\n- In light of above, the observation that counting cycles remains intrinsically \\emph{local} is a good and noteworthy observation.\n- The technical content is good and supported by the experiments.\n\nWEAKNESSES:\n### Neglected baseline comparisons with other higher-order GNNs\nAs explained by the author, [35] and [46] presents GNNs utilizing node tuples for better expressivity. As the adaptation of FWL(2) is the main component of this work, it is concerning that [35] and [46] are not included in the baseline comparisons. Since FWL is adopted into GNN to improve over sugraph GNNs, other (representative) works on higher-order GNNs should also be compared.\n\nAs it stands, there is no information in the paper on how this work compares to other higher-order GNNs. Does the proposed work have better performance and better efficiency? Does the proposed work trade-off performance for efficiency and space? We do not know.\n\nIn continuation of above, as [35] is a highly cited paper, further searching founds [a, b, c] that also mentions cycle counting in their paper, but is not discussed or mentioned in this work.\n- [a] A New Perspective on \"How Graph Neural Networks Go Beyond Weisfeiler-Lehman?\" ICLR 2022\n- [b] A Practical, Progressively-Expressive GNN. NeurIPS 2022\n- [c] Graph Neural Networks with Local Graph Parameters. NeurIPS 2021\n\n\n### Unanswered questions: ablation and trade-off for larger d\nWhile the work presents d-DRFWL(2) GNNs, the theoretical analysis and experiments are all limited to d=2. Furthermore, Q4 is not satisfyingly answered, as the experiment results are very close. Why not use a sufficiently large d to represent the FWL(2) test?\n\n### There are some places that are unclear and could be improved by providing more explanation:\n- FWL is left unexplained in the abstract. The paper may improve its readability by briefly stating FWL is a 2-tuple coloring scheme that solves graph isomorphism tests, whereas this work proposes to limit the scope of 2-tuples from all node pairs to only those within d-distance.\n- (line 108), \"divided by a factor only depending on S.\" Isn't the factor simply the number of nodes in S?\n- The syntax of using double curly brakets {{ }} for multisets should be noted when first appeared in Equation (1)\n- (line 125) the meaning of rightarrow with a bar ($\\mapsto$) is not clear. \n- The meaning (or depiction) of the twelve targets of QM9 dataset is not provided in the main paper or the appendix\n\nQUESTIONS:\nCan experiments be provided for comparison between the proposed method with [35] and [46] for performance and empirical efficiency?\n\nCan experiments and analysis be provided for $d\\geq 2$?\n\nCan intuitive explanations be provided for each proof and concepts (currently, only 6-cyle is provided with an intuition)?\n- For example, it seems that the expressivity of up to 4-path and 6-cycle is correlated, as the furtherest node from a certain node in a 6-cycle requires a 4-path back get back to the certain node. Besides, is the 4-path attained by a pair of 2-tuples?\n\nLIMITATIONS:\nIt is provided.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}