Graph Neural Networks with Local Graph  Parameters

1 Introduction . Context . Graph neural networks ( GNNs ) [ Merkwirth and Lengauer , 2005 , Scarselli et al. , 2009 ] , and its important class of Message Passing Neural Networks ( MPNNs ) [ Gilmer et al. , 2017 ] , are one of the most popular methods for graph learning tasks . Such MPNNs use an iterative message passing scheme , based on the adjacency structure of the underlying graph , to compute vertex ( and graph ) embeddings in some real Euclidean space . The expressive ( or distinguishing ) power of MPNNs is , however , rather limited [ Morris et al. , 2019 , Xu et al. , 2019 ] . Indeed , MPNNs will always identically embed two vertices ( graphs ) when these vertices ( graphs ) can not be distinguished by the one-dimensional Weisfeiler-Leman ( WL ) algorithm . Two graphs G1 and H1 and vertices v and w that can not be distinguished by WL ( and thus any MPNN ) are shown in Fig . 1 . The expressive power of WL is well-understood [ Cai et al. , 1992 , Dell et al. , 2018 , Arvind et al. , 2020 ] and basically can only use tree-based structural information in the graphs to distinguish vertices . Hence , no MPNN can detect that vertex v in Fig . 1 is part of a 3-clique , whereas w is not . Similarly , MPNNs can not detect that w is part of a 4-cycle , whereas v is not . Further limitations of WL in terms of graph properties can be found , e.g. , in Arvind et al . [ 2020 ] , Chen et al . [ 2020 ] and Tahmasebi and Jegelka [ 2020 ] . To remedy the weak expressive power of MPNNs , so-called higher-order MPNNs were proposed [ Maron et al. , 2019a , Morris et al. , 2019 , 2020 ] , whose expressive power is well-understood and 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . measured in terms of the k-dimensional WL procedures ( k-WL ) [ Maron et al. , 2019a , Chen et al. , 2019a , Geerts , 2020 , Sato , 2020 , Azizian and Lelarge , 2021 ] . In a nutshell , k-WL operates on k-tuples of vertices and allows to distinguish vertices ( graphs ) based on structural information related to graphs of treewidth k [ Dvorak , 2010 , Dell et al. , 2018 ] . By definition , WL = 1-WL . As an example , 2-WL can detect that vertex v in Fig . 1 belongs to a 3-clique or a 4-cycle since both have treewidth two . While more expressive than WL , the GNNs based on k-WL require O ( nk ) operations in each iteration , where n is the number of vertices , hereby hampering their applicability . A more practical approach is to extend the expressive power of MPNNs whilst preserving their O ( n ) cost in each iteration . Various such extensions [ Kipf and Welling , 2017 , Chen et al. , 2019a , Li et al. , 2019 , Ishiguro et al. , 2020 , Bouritsas et al. , 2020 , Geerts et al. , 2021 ] achieve this by infusing MPNNs with local graph structural information from the start . That is , the iterative message passing scheme of MPNNs is run on vertex labels that contain quantitative information about local graph structures . It is easy to see that such architectures can go beyond the WL test : for example , adding triangle counts to MPNNs suffices to distinguish the vertices v and w and graphs G1 and H1 in Fig . 1 . Moreover , the cost is a single preprocessing step to count local graph parameters , thus maintaining the O ( n ) cost in the iterations of the MPNN . While there are some partial results showing that local graph parameters increase expressive power [ Bouritsas et al. , 2020 , Li et al. , 2019 ] , their precise expressive power and relationship to higher-order MPNNs was unknown , and there is little guidance in terms of which local parameters do help MPNNs and which ones do not . The main contribution of this paper is a precise characterization of the expressive power of MPNNs with local graph parameters and its relationship to the hierarchy of higher-order MPNNs . Our contributions . In order to nicely formalize local graph parameters , we propose to extend vertex labels with homomorphism counts of small graph patterns.1 More precisely , given graphs P and G , and vertices r in P and v in G , we propose to augment the initial features of v with the number of homomorphisms from P to G that map r to v , denoted by hom ( P r , Gv ) , as a way to capture local structural information . More generally , homomorphism counts for a collection of graphs are considered . Indeed , we propose F -MPNNs where F = { P r1 , . . . , P r ` } is a set of ( graph ) patterns , which extend MPNNs by ( i ) first allowing a preprocessing step that labels each vertex v of a graph G with the vector ( hom ( P r1 , G v ) , . . . , hom ( P r ` , G v ) ) , and ( ii ) then run an MPNN on this labelling . Our main contributions are the following : 1 . We precisely characterize the expressive power of F-MPNNs by means of an extension of WL , denoted by F -WL . This characterization gracefully extends the characterization for standard MPNNs , mentioned earlier , by setting F = ∅ , and provides insights in the expressive power of existing MPNN extensions , most notably the Graph Substructure Networks of Bouritsas et al . [ 2020 ] . 2 . We compare F-MPNNs to higher-order MPNNs , which are characterized in terms of the k-WLtest . On the one hand , while F -MPNNs strictly increase the expressive power of the WL-test , for any finite set F of patterns , 2-WL can distinguish graphs which F -MPNNs can not . On the other hand , for each k ≥ 1 there are patterns P such that { P } -MPNNs can distinguish graphs which k-WL can not . 3 . We deal with the challenging problem of pattern selection and comparing F-MPNNs based on the patterns included in F . We prove two partial results : one establishing when a pattern P in F is redundant , and another result indicating when P does add expressive power , based on the treewidth of P compared to the treewidth of other patterns in F . 4 . Our theoretical results are complemented by an experimental study in which we show that for various GNN architectures , datasets and graph learning tasks , all part of the recent benchmark by Dwivedi et al . [ 2020 ] , the augmentation of initial features with homomorphism counts of graph patterns has often a positive effect , and the cost for computing these counts incurs little to no overhead . 1We recall that homomorphisms are edge-preserving mappings between the vertex sets . As such , we believe that F-MPNNs not only provide an elegant theoretical framework for understanding local graph parameter enabled MPNNs , they are also a valuable alternative to higher-order MPNNs as a way to increase the expressive power of MPNNs . In addition , and as will be explained in Section 2 , F -MPNNs provide a unifying framework for understanding the expressive power of several other existing extensions of MPNNs . Proofs of our results and further details on the relationship to existing approaches and experiments can be found in the supplementary material . Related Work . Works related to the distinguishing power of the WL-test , MPNNs and their higherorder variants are cited throughout the paper . Beyond distinguishability , GNNs are analyzed in terms of universality and generalization properties [ Maron et al. , 2019b , Keriven and Peyré , 2019 , Chen et al. , 2019b , Garg et al. , 2020 , Azizian and Lelarge , 2021 ] , local distributed algorithms [ Sato et al. , 2019 , Loukas , 2020 ] , randomness in features [ Sato et al. , 2021 , Abboud et al. , 2021 ] and using local context matrix features [ Vignac et al. , 2020 ] . Other extensions of GNNs are surveyed , e.g. , in Wu et al . [ 2021 ] , Zhou et al . [ 2018 ] and Chami et al . [ 2021 ] . Related are also the Graph Homomorphism Convolutions by NT and Maehara [ 2020 ] which apply SVMs directly on a homomorphism count representation of vertices . Finally , our approach is reminiscent of graph representations by means of graphlet kernels [ Shervashidze et al. , 2009 ] , but then on the level of vertices . Limitations of our approach . One of the limitations of F-MPNNs is that their expressive power depends on the set F of patterns . Our work offers tools and guidelines to help in this search , but the best set of patterns must still be found by trial-and-error . However , as we show in Section 6 , MPNNs almost always benefit from any set of additional features , assuming that the resulting model goes beyond the WL test . Cliques and cycles are two types of simple patterns that are guaranteed to extend the WL test , and indeed we show that striking gains can be obtained by simply adding these features to existing benchmarks . For simplicity of exposition we focus on vertex-labelled undirected graphs but all our results can be extended to edge-labelled directed graphs . 2 Local Graph Parameter Enabled MPNNs We here introduce our MPNNs with local graph parameters . We first recall some graph concepts . Graphs . We consider undirected vertex-labelled graphs G = ( V , E , χ ) , with V the set of vertices , E the set of edges and χ a mapping assigning a label to each vertex in V . The set of neighbors of a vertex is denoted by NG ( v ) = { u ∈ V ∣∣ { u , v } ∈ E } . A rooted graph is a graph in which one of its vertices is declared as its root . We denote a rooted graph by Gv , where v ∈ V is the root and depict them as graphs in which the root is a blackened vertex , such as e.g. , . Given graphs G = ( VG , EG , χG ) and H = ( VH , EH , χH ) , an homomorphism h is a mapping h : VG → VH such that ( i ) { h ( u ) , h ( v ) } ∈ EH for every { u , v } ∈ EG , and ( ii ) χG ( u ) = χH ( h ( u ) ) for every u ∈ VG . For rooted graphs Gv and Hw , an homomorphism must additionally map v to w. We denote by hom ( G , H ) the number of homomorphisms from G to H ; similarly for rooted graphs . MPNNs with local graph parameters . Let F = { P r1 , . . . , P r ` } be a set of rooted graphs , which we refer to as patterns . As a uniform formalization of local graph structural information , we propose the use of homomorphism counts of patterns in F to enhance the initial feature of vertices . To illustrate the idea , consider the graphs in Fig . 1 . As mentioned , these graphs can not be distinguished by the WL-test , and therefore can not be distinguished by the broad class of MPNNs . If we allow a preprocessing stage , however , in which the initial labelling of every vertex v is extended with the number of ( homomorphic images of ) 3-cliques in which v participates ( indicated by numbers between brackets in Fig . 1 ) , then clearly vertices v and w ( and the graphs G1 and H1 ) can be distinguished based on this extra structural information . In fact , the initial labelling already suffices for this purpose . In our setting , this will correspond to selecting the rooted 3-clique and attach to each vertex v , hom ( , Gv ) . Information about 4-cycles requires adding hom ( , Gv ) as well . We therefore propose F-enabled MPNNs , or just F-MPNNs , defined in the same way as MPNNs [ Gilmer et al. , 2017 ] with the crucial difference that the initial feature vector of a vertex v ( a one-hot encoding of its label χG ( v ) ) is augmented with all the homomorphism counts from patterns in F . Formally , in each round d an F-MPNN M labels each vertex v in graph G with a feature vector x ( d ) M , F , G , v which is inductively defined as follows : x ( 0 ) M , F , G , v : = ( χG ( v ) , hom ( P r 1 , G v ) , . . . , hom ( P r ` , G v ) ) x ( d ) M , F , G , v : = UPD ( d ) ( x ( d−1 ) M , F , G , v , COMB ( d ) ( { { x ( d−1 ) M , F , G , v | u ∈ NG ( v ) } } ) ) , for d > 0 , where COMB ( d ) and UPD ( d ) are an aggregating and update function , respectively , as in standard MPNNs , and where { { } } denotes a multi-set . We note that standard MPNNs are F-MPNNs with F = ∅ . As for MPNNs , we can equip F -MPNNs with a READOUT function that aggregates all final feature vectors into a single feature vector in order to classify or distinguish graphs . We remark that any MPNN architecture can be turned into an F -MPNN by a simple homomorphism counting preprocessing step . As such , we propose a generic plug-in for a large class of GNN architectures . Better still , hom ( P r , Gv ) can be computed in time O ( |VG|tw ( P r ) +1 ) [ Díaz et al. , 2002 ] , where tw ( P r ) denotes the treewidth of pattern P r. The treewidth of patterns used in FMPNNs is typically small . And indeed , homomorphism counts of small graph patterns can be efficiently computed in practice , even on large datasets [ Zhang et al. , 2020 ] . We also remark that the use of rooted patterns is important because different vertices in a pattern may embed differently around a target vertex in a graph . For example , hom ( , Gv ) can be different from hom ( , Gv ) . The choice of a root in a graph P can be avoided by including different rooted versions of P in F . In fact , it suffices to include one rooted version P r for each vertex r in a distinct orbit in P . We note that for symmetric graphs , such as cliques and cycles , all vertices lie in the same orbit and a single , arbitrary , choice of root vertex suffices . Alternatively , one could define hom ( P , Gv ) : = ∑ r∈VP hom ( P r , Gv ) which ignores how the pattern is locally mapped into a graph . We speculate , however , that this results in a model that is less powerful than F-MPNNs . Despite their simplicity , we will show that F-MPNNs can substantially increase the power of MPNNs by varying F , only paying a one-time preprocessing cost . F -MPNNs as unifying framework . An important aspect of F -MPNNs is that they allow a principled analysis of the power of existing extensions of MPNNs . For example , taking F = { } suffices to capture degree-aware MPNNs [ Geerts et al. , 2021 ] , such as the Graph Convolution Networks ( GCNs ) [ Kipf and Welling , 2017 ] , which use the degree of vertices ; taking F = { L1 , L2 , . . . , L ` } for rooted paths Li of length i suffices to model the walk counts used in Chen et al . [ 2019a ] ; and taking F as the set of labeled trees of depth one precisely corresponds to the use of the WL-labelling obtained after one round by Ishiguro et al . [ 2020 ] . Furthermore , { C ` } -MPNNs , where C ` denotes the cycle of length ` , correspond to the extension proposed in Section 4 in Li et al . [ 2019 ] . In addition , F-MPNNs are close in spirit to the Graph Substructure Networks ( GSNs ) by Bouritsas et al . [ 2020 ] , which use subgraph isomorphism counts of graph patterns . We recall that an isomorphism from G to H is a bijective homomorphism h from G to H which additionally satisfies ( i ) { h−1 ( u ) , h−1 ( v ) } ∈ EG for every { u , v } ∈ EH , and ( ii ) χG ( h−1 ( u ) ) = χH ( u ) for every u ∈ VH . When G and H are rooted graphs , isomorphisms should preserve the roots as well . Now , in a GSN , the feature vector of each vertex v is augmented with the subgraph isomorphism counts , sub ( P r , Gv ) , for rooted patterns P r in a set of P of patterns , and this is followed by the execution of an MPNN , just as for our F-MPNNs . Importantly , F-MPNNs can be used to bound the expressive power of GSNs , as we will show later . We note that computing sub ( P r , Gv ) is , in general , more costly than computing hom ( P r , Gv ) [ Curticapean et al. , 2017 ] . This partially motivates our choice of using homomorphism counts instead of subgraph isomorphism counts . Moreover , homomorphism counts of tree patterns underly existing characterizations of the expressive power of MPNNs [ Dell et al. , 2018 , Grohe , 2020a ] . As we will see shortly , these characterizations gracefully extend to F -MPNNs . 3 The Expressive Power of F -MPNNs We next provide exact characterizations of the expressive power of F-MPNNs . Our results extend those for standard MPNNs [ Xu et al. , 2019 , Morris et al. , 2019 , Dell et al. , 2018 ] . Characterization in terms of F -WL . We bound the expressive power of F-MPNNs in terms of what we call the F-WL-test . This test extends the WL-test [ Weisfeiler and Lehman , 1968 , Grohe , 2017 ] in the same way as F -MPNNs extend standard MPNNs : by including homomorphism counts of patterns in F in the initial labelling . The F-WL-test , for F = { P r1 , . . . , P r ` } , is a vertex labelling algorithm that iteratively computes a label χ ( d ) F , G , v for each vertex v of a graph G , as follows : χ ( 0 ) F , G , v : = ( χG ( v ) , hom ( P r 1 , G v ) , . . . , hom ( P r ` , G v ) ) χ ( d ) F , G , v : = HASH ( χ ( d−1 ) F , G , v , { { χ ( d−1 ) F , G , u | u ∈ NG ( v ) } } ) , for d > 0 . The F-WL-test stops in round d when no new pair of vertices are distinguished , that is , χ ( d−1 ) F , G , v1 = χ ( d−1 ) F , G , v2 implies χ ( d ) F , G , v1 = χ ( d ) F , G , v2 , for any vertices v1 and v2 in G. The standard WL-test corresponds to { ∅ } -WL . We can use the F-WL-test to compare vertices of the same graphs , or different graphs . We say that the F-WL-test can not distinguish vertices if their final labels are the same , and that the F-WL-test can not distinguish graphsG andH if the multiset containing each label computed for G is the same as that of H . Similarly as for MPNNs and the WL-test [ Xu et al. , 2019 , Morris et al. , 2019 ] , the F-WL-test provides an upper bound for the expressiveness of F-MPNNs . Proposition 1 . If two vertices of a graph can not be distinguished by the F-WL-test , then they can not be distinguished by any F-MPNN either . Moreover , if two graphs can not be distinguished by the F-WL-test , then they can not be distinguished by any F-MPNN either . Furthermore , simply adding local parameters from a set F of patterns to the GIN architecture of Xu et al . [ 2019 ] results in F-MPNNs that match the expressive power of the F-WL-test . Characterization in terms of F -pattern trees . At the core of several results about the WL-test lies a characterization linking the test with homomorphism counts of ( rooted ) trees [ Dell et al. , 2018 , Grohe , 2020a ] . In view of the connection to MPNNs , it tells that MPNNs only use quantitative tree-based structural information from the underlying graphs . We next extend this characterization to F-WL by using homomorphism counts of so-called F-pattern trees . Proposition 1 then reveals that F -MPNNs can use quantitative information of richer graph structures than the trees used by MPNNs . To define F-pattern trees we need the graph join operator ? . Given two rooted graphs Gv and Hw , the join graph ( G ? H ) v is obtained by taking the disjoint union of Gv and Hw , followed by identifying w with v. The root of the join graph is v. For example , the join of and is . Figure 2 : Examples of F-pattern trees . Furthermore , ifG is a graph and P r is a rooted graph , then joining a vertex v in G with P r results in the disjoint union ofG and P r , where r is identified with v. Let F = { P r1 , . . . , P r ` } . An F-pattern tree T r is obtained from a standard rooted tree Sr = ( V , E , χ ) , called the backbone of T r , followed by joining every vertex s ∈ V with any number of copies of patterns from F . Examples of F -pattern trees , for F = { } are shown in Fig . 2 , where grey colored vertices are part of the backbones of the F -pattern trees . We define the depth of an F -pattern tree as the depth of its backbone . Standard trees are F-pattern trees in which no patterns are joined with backbone vertices . We next use F-pattern trees to characterize the expressive power of F-WL and thus , by Proposition 1 , of F-MPNNs . Theorem 1 . For any finite collection F of patterns , vertices v and w in a graph G are indistinguishable by the F-WL-test if and only if hom ( T r , Gv ) = hom ( T r , Gw ) for every rooted F-pattern tree T r. Similarly , G and H are indistinguishable by the F-WL-test if and only if hom ( T , G ) = hom ( T , H ) for every ( unrooted ) F-pattern tree T . The proof of this Theorem requires a generalization of the techniques from Dell et al . [ 2018 ] and Grohe [ 2020a , b ] , used to characterize the expressiveness of WL in terms of homomorphism counts of trees . In fact , we can use our proof to recover said results simply by setting F = ∅ . We note that F-MPNNs are more expressive than MPNNs ( recall the graphs G1 and H1 and F = { } ) . We can even make the above theorem more precise . When F-WL is run for d rounds , then only F-patterns trees of depth d are required . This tells that increasing the number of rounds of F-WL results in that more complicated structural information is taken into account . As an illustration , consider the graphs G2 and H2 and vertices v and w shown in Fig . 3 and let us consider F-WL with F = { } . We argue that v and w can not be distinguished by F-WL based on the initial labelling only , but they can be distinguished after one round . v G2 w H2 F-MPNNs to detect more complex patterns based on F-pattern trees . This is in contrast to , e.g. , the Graph Homomorphism Convolutions by NT and Maehara [ 2020 ] in which only homomorphism counts of patterns in F , and not the counts of the derived F-patterns trees , are taken into account . Implications for other MPNN extensions . Importantly , Theorem 1 discloses the boundaries of F - MPNNs . To illustrate this for some specific instances of F -MPNNs mentioned earlier , the expressive power of degree-based MPNNs [ Kipf and Welling , 2017 , Geerts et al. , 2021 ] is captured by { L1 } pattern trees , and walk count MPNNs [ Chen et al. , 2019a ] are captured by { L1 , . . . , L ` } -pattern trees . These pattern trees are just trees , since joining paths to trees only results in bigger trees . Thus , Theorem 1 tells that all these extensions are still bounded by WL ( albeit needing fewer rounds ) . In contrast , beyond WL , { C ` } -pattern trees capture cycle count MPNNs [ Li et al. , 2019 ] . Our results also shed light on the expressive power of GSNs [ Bouritsas et al. , 2020 ] which , as already mentioned , use subgraph isomorphism counts of patterns P ∈ P . More precisely , a P-GSN augments vertex features by including sub ( P r , Gv ) for rooted versions P r of P ∈ P , where r ranges over representative vertices of the different orbits in P . Let us denote by P+ this set of rooted patterns . Fact 1 ( Curticapean et al . [ 2017 ] ) . For any set P of rooted patterns , there exists a pattern set s ( P ) such that for each P r ∈ P , sub ( P r , Gv ) can be computed based on { hom ( Qr , Gv ) | Qr ∈ s ( P ) } . In other words , subgraph isomorphism counts can be computed in terms of homomorphism counts , albeit by using different pattern sets . For example , if P = { } , then s ( P ) = { , , , } which contains all different homomorphic images of . As a consequence , we obtain : Proposition 2 . P-GSNs can not distinguish more vertices than s ( P+ ) -MPNNs can . So , we can bound the expressive power of GSNs in terms of F-MPNNs . In the supplementary material we also show that F-MPNNs are bounded by GSNs that use some special set of patterns derived from F . We conclude by observing that the proof technique underlying Theorem 1 can be directly applied to GSNs . The key insight is to redefine the notion of homomorphism from pattern trees to graphs . Indeed , consider a P-GSN and a corresponding P+-pattern tree T r. We define shom ( T r , Gv ) as the number of mappings h : VT → VG such that ( i ) h ( r ) = v ; ( ii ) h is a homomorphism when restricted to the backbone of T r ; and ( iii ) for each P r joined with a backbone vertex , h ( P r ) is isomorphic to P r. We then have the following counterpart of Theorem 1 for GSNs : Theorem 2 . For any finite collection P of patterns , vertices v and w in a graph G are indistinguishable by P-GSNs if and only if shom ( T r , Gv ) = shom ( T r , Gw ) for every rooted P+-pattern tree T r. Similarly , G and H are indistinguishable by P-GSNs if and only if shom ( T , G ) = shom ( T , H ) for every ( unrooted ) P+-pattern tree T . 4 A Comparison with the k-WL-test Compared to the computationally intensive higher-order MPNNs based on the k-WL-tests [ Maron et al. , 2019a , Morris et al. , 2019 , 2020 ] , F-MPNNs are an alternative and efficient way to extend the expressive power of MPNNs ( and thus the WL-test ) . In this section we situate F-WL in the k-WL hierarchy . The definition of the k-WL-test can be found , e.g. , in Morris et al . [ 2020 ] , and is provided in the supplementary material as well . We also use the standard notion of treewidth of a graph ( see e.g. , Bodlaender [ 1993 ] ) . Intuitively , treewidth measures the tree-likeness of a graph . For example , trees have treewidth one , cycles have treewidth two , and the k-clique Kk has treewidth k − 1 . Furthermore , the treewidth of a pattern P r is the treewidth of its unrooted version P . We have seen that F-WL can distinguish graphs that WL can not : just consider { K3 } -WL for the 3-clique K3 . To generalize this observation we need some notation . Let F and G be two sets of patterns and consider an F-MPNN M and a G-MPNN N . We say that M is upper bounded in expressive power by N if for any graph G , if N can not distinguish vertices v and w,2 then neither can M . A similar notion is in place for pairs of graphs : if N can not distinguish graphs G and H , then neither can M . More generally , letM be a class of F -MPNNs and N be a class of G-MPNNs . We say that the classM is upper bounded in expressive power by N if every M ∈ M is upper bounded in expressive power by an N ∈ N ( which may depend on M ) . WhenM is upper bounded by N and vice versa , thenM and N are said to have the same expressive power . A class N is more expressive than a classM whenM is upper bounded in expressive power by N , but there exist graphs that can be distinguished by MPNNs in N but not by any MPNN inM . Our first result is a consequence of the characterization of k-WL in terms of homomorphism counts of graphs of treewidth k [ Dvorak , 2010 , Dell et al. , 2018 ] . Proposition 3 . For each finite set F of patterns , the expressive power of F-WL is bounded by k-WL , where k is the largest treewidth of a pattern in F . For example , since the treewidth of K3 is 2 , we have that { K3 } -WL is bounded by 2-WL . Similarly , { Kk+1 } -WL is bounded in expressive power by k-WL . Our second result tells how to increase the expressive power of F-WL beyond k-WL . A pattern P r is a core if any homomorphism from P to itself is injective . For example , any clique Kk and cycle of odd length is a core . Theorem 3 . Let F be a finite set of patterns . If F contains a pattern P r which is a core and has treewidth k , then there exist graphs that can be distinguished by F-WL but not by ( k − 1 ) -WL . In other words , for such F , F-WL is not bounded by ( k − 1 ) -WL . For example , since K3 is a core , { K3 } -WL is not bounded in expressive power by WL = 1-WL . More generally , { Kk } -WL is not bounded by ( k − 1 ) -WL . The proof of Theorem 3 is based on extending deep techniques developed in finite model theory , and that have been used to understand the expressive power of finite variable logics [ Atserias et al. , 2007 , Bova and Chen , 2019 ] . This result is stronger than the one underlying the strictness of the k-WL hierarchy [ Otto , 2017 ] , which states that k-WL is strictly more expressive than ( k − 1 ) -WL . Indeed , Otto [ 2017 ] only shows the existence of a pattern P r of treewidth k such that ( k− 1 ) -WL is not bounded by { P r } -WL . In Theorem 3 we provide an explicit recipe for finding such a pattern P r , that is , P r can be taken a core of treewidth k. In summary , we have shown that there is a set F of patterns such that ( i ) F-WL can distinguish graphs which can not be distinguished by ( k − 1 ) -WL , yet ( ii ) F-WL can not distinguish more graphs than k-WL . This begs the question whether there is a finite set F such that F-WL is equivalent in expressive power to k-WL . We answer this negatively . Proposition 4 . For any k > 1 , there does not exist a finite set F of patterns such that F-WL is equivalent in expressive power to k-WL . In the proof of Proposition 4 we show a stronger claim . Indeed , we construct two graphs that can be distinguished by 2-WL but can not be distinguished by any F-WL . Since any two graphs that can be distinguished by 2-WL can also be distinguished by k-WL , for any k > 2 , the proposition follows . In view of the connection between F -MPNNs and GSNs mentioned earlier , we thus show that no GSN can match the power of k-WL , which was a question left open in Bouritsas et al . [ 2020 ] . We remark that if we allow F to consist of all ( infinitely many ) patterns of treewidth k , then F-WL is equivalent in expressive power to k-WL [ Dvorak , 2010 , Dell et al. , 2018 ] .

Summary: This paper provides a theoretical framework attempting to unify a variety of recent methods in the Graph Neural Network literature, which all follow a similar conceptual approach: they introduce extra structural features in the input of the Neural Network that cannot be computed by traditional message passing. The main formalism underlying the framework is that of homomorphism, which is an adjacency-preserving mapping between two graphs. Differently from subgraph isomorphism, homomorphisms are not necessarily bijective (think of walks vs paths).  The authors define $\mathcal{F}$-MPNNs as a message passing framework where node features are enhanced with the homomorphism counts of every pattern in a collection $\mathcal{F}$. The authors derive a deep characterization of the expressive power of $\mathcal{F}$-MPNNs, both in comparison to the k-WL hierarchy and between different instantiations thereof,  show how it can incorporate other approaches, and answer numerous research questions that have been left open by the most related paper, that of Bouritsas et al., arxiv’20. The method is also validated experimentally, by enhancing GNNs with homomorphism counts of cycles and cliques, on node, edge and graph level tasks, showing consistent improvement against various conventional MPNN counterparts.