Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting

1 INTRODUCTION . The field of graph representation learning has undergone a rapid growth in the past few years . In particular , Graph Neural Networks ( GNNs ) , a family of neural architectures designed for irregularly structured data , have been successfully applied to problems ranging from social networks and recommender systems ( Ying et al. , 2018a ) to bioinformatics ( Fout et al. , 2017 ; Gainza et al. , 2020 ) , chemistry ( Duvenaud et al. , 2015 ; Gilmer et al. , 2017 ; Sanchez-Lengeling et al. , 2019 ) and physics ( Kipf et al. , 2018 ; Battaglia et al. , 2016 ) , to name a few . Most GNN architectures are based on message passing ( Gilmer et al. , 2017 ) , where at each layer the nodes are updated by information aggregated from their neighbours . A crucial difference from traditional neural networks operating on grid-structured data is the absence of canonical ordering of the nodes in a graph . To address this , the aggregation function is constructed to be invariant to neighbourhood permutations and , as a consequence , to graph isomorphism . This kind of symmetry is not always desirable and thus different inductive biases that disambiguate the neighbours have been proposed . For instance , in geometric graphs , such as 3D molecular graphs and meshes , directional biases are usually employed in order to model the positional information of the nodes ( Masci et al. , 2015 ; Monti et al. , 2017 ; Bouritsas et al. , 2019 ; Klicpera et al. , 2020 ; de Haan et al. , 2020b ) ; for proteins , ordering information is used to disambiguate amino-acids at different positions in the sequence ( Ingraham et al. , 2019 ) ; in multi-relational knowledge graphs , a different aggregation is performed for each relation type ( Schlichtkrull et al. , 2018 ) . The structure of the graph itself does not usually explicitly take part in the aggregation function . In fact , most models rely on multiple message passing steps as a means for each node to discover the global structure of the graph . However , since message-passing GNNs are at most as powerful as the Weisfeiler Lehman test ( WL ) ( Xu et al. , 2019 ; Morris et al. , 2019 ) , they are limited in their abilities to adequately exploit the graph structure , e.g . by counting substructures ( Arvind et al. , 2019 ; Chen et al. , 2020 ) . This uncovers a crucial limitation of GNNs , as substructures have been widely recognised as important in the study of complex networks . For example , in molecular chemistry , functional groups and rings are related to a plethora of chemical properties , while cliques are related to protein complexes in Protein-Protein Interaction networks and community structure in social networks , respectively ( Granovetter , 1982 ; Girvan & Newman , 2002 ) . Motivated by this observation , in this work we propose Graph Substructure Network ( GSN ) , a new symmetry breaking mechanism for GNNs based on introducing structural biases in the aggregation function . In particular , each message is transformed differently depending on the topological relationship between the endpoint nodes . This relationship is expressed by counting the appearance of certain substructures , the choice of which allows us to provide the model with different inductive biases , based on the graph distribution at hand . The substructures are encoded as structural identifiers that are assigned to either the nodes or edges of the graph and can thus disambiguate the neighbouring nodes that take part in the aggregation . We characterise the expressivity of substructure encoding in GNNs , showing that GSNs are strictly more expressive than traditional GNNs for the vast majority of substructures while retaining the locality of message passing , as opposed to higher-order methods ( Maron et al. , 2019b ; c ; a ; Morris et al. , 2019 ) that follow the WL hierarchy ( see Section 2 ) . In the limit , our model can yield a unique representation for every isomorphism class and is thus universal . We provide an extensive experimental evaluation on hard instances of graph isomorphism testing ( strongly regular graphs ) , as well as on real-world networks from the social and biological domains , including the recently introduced large-scale benchmarks ( Dwivedi et al. , 2020 ; Hu et al. , 2020 ) . We observe that when choosing the structural inductive biases based on domain-specific knowledge , GSN achieves state-ofthe-art results . 2 PRELIMINARIES . Let G = ( VG , EG ) be a graph with vertex set VG and undirected edge set EG . A subgraph GS = ( VGS , EGS ) of G is any graph with VGS ⊆ VG , EGS ⊆ EG . When EGS includes all the edges of G with endpoints in VGS , i.e . EGS = { ( v , u ) ∈ E : v , u ∈ VGS } , the subgraph is said to be induced . Isomorphisms Two graphs G , H are isomorphic ( denoted H ' G ) , if there exists an adjacencypreserving bijective mapping ( isomorphism ) f : VG → VH , i.e . ( v , u ) ∈ EG iff ( f ( v ) , f ( u ) ) ∈ EH . Given some small graph H , the subgraph isomorphism problem amounts to finding a subgraph GS of G such that GS ' H . An automorphism of H is an isomorphism that maps H onto itself . The set of all the unique automorphisms form the automorphism group of the graph , denoted as Aut ( H ) , contains all the possible symmetries of the graph . The automorphism group yields a partition of the vertices into disjoint subsets of VH called orbits . Intuitively , this concept allows us to group the vertices based on their structural roles , e.g . the end vertices of a path , or all the vertices of a cycle ( see Figure 1 ) . Formally , the orbit of a vertex v ∈ VH is the set of vertices to which it can be mapped via an automorphism : Orb ( v ) = { u ∈ VH : ∃g ∈ Aut ( H ) s.t . g ( u ) = v } , and the set of all orbits H \Aut ( H ) = { Orb ( v ) : v ∈ VH } is usually called the quotient of the automorphism when it acts on the graph H . We are interested in the unique elements of this set that we will denote as { OVH,1 , OVH,2 , . . . , OVH , dH } , where dH is the cardinality of the quotient . Analogously , we define edge structural roles via edge automorphisms , i.e . bijective mappings from the edge set onto itself , that preserve edge adjacency ( two edges are adjacent if they share a common endpoint ) . In particular , every vertex automorphism g induces an edge automorphism by mapping each edge { u , v } to { g ( u ) , g ( v ) } . 1 In the same way as before , we construct the edge automorphism group , from which we deduce the partition of the edge set in edge orbits { OEH,1 , OEH,2 , . . . , OEH , dH } . Weisfeiler-Lehman tests : The Weisfeiler-Lehman graph-isomorphism test ( Weisfeiler & Leman , 1968 ) , also known as naive vertex refinement , 1-WL , or just WL ) , is a fast heuristic to decide if two graphs are isomorphic or not . The WL test proceeds as follows : every vertex v is initially assigned a colour c0 ( v ) that is later iteratively refined by aggregating neighbouring information : 1Note that the edge automorphism group is larger than that of induced automorphisms , but strictly larger only for 3 trivial cases ( Whitney , 1932 ) . However , induced automorphisms provide a more natural way to express edge structural roles . ct+1 ( v ) = HASH ( ct ( v ) , Hct ( u ) Iu∈N ( v ) ) , where H·I denotes a multiset ( a set that allows element repetitions ) and N ( v ) is the neighbourhood of v. The WL algorithm terminates when the colours stop changing , and outputs a histogram of colours . Two graphs with different histograms are nonisomorphic ; if the histograms are identical , the graphs are possibly , but not necessarily , isomorphic . Note that the neighbour aggregation in the WL test is a form of message passing , and GNNs are the learnable analogue . A series of works on improving GNN expressivity mimic the higher-order generalisations of WL , known as k-WL and k-Folklore WL ( WL hierarchy ) and operate on k-tuples of nodes ( see Appendix B.1 ) . The ( k + 1 ) -FWL is strictly stronger than k-FWL , k-FWL is as strong as ( k + 1 ) -WL and 2-FWL is strictly stronger than the simple 1-WL test . 3 GRAPH SUBSTRUCTURE NETWORKS . Graphs consist of nodes ( or edges ) with repeated structural roles . Thus , it is natural for a neural network to treat them in a similar manner , akin to weight sharing between local patches in CNNs for images ( LeCun et al. , 1989 ) or positional encodings in language models for sequential data ( Sukhbaatar et al. , 2015 ; Gehring et al. , 2017 ; Vaswani et al. , 2017 ) . Nevertheless , GNNs are usually unaware of the nodes ’ different structural roles , since all nodes are treated equally when performing local operations . Despite the initial intuition that the neural network would be able to discover these roles by constructing deeper architectures , it has been shown that GNNs are ill-suited for this purpose and are blind to the existence of structural properties , e.g . triangles or larger cycles ( Chen et al. , 2020 ; Arvind et al. , 2019 ) . To this end , we propose to explicitly encode structural roles as part of message passing , in order to capture richer topological properties . Our method draws inspiration from Loukas ( 2020 ) , where it was shown that GNNs become universal when the nodes in the graph are uniquely identified , i.e when they are equipped with different features . However , it is not clear how to choose these identifiers in a way that can allow the neural network to generalise . Structural roles , when treated as identifiers , although not necessarily unique , are more amenable to generalisation due to their repetition across different graphs . Thus , they can constitute a trade-off between uniqueness and generalisation . Structural features : Structural roles are encoded into features by counting the appearance of certain substructures . The larger the substructure collection , the more fine-grained the partition of nodes into roles will be . LetH = { H1 , H2 . . . HK } be a set of small ( connected ) graphs , for example cycles of fixed length or cliques . For each graph H ∈ H , we first find its isomorphic subgraphs in G denoted as GS . For each node v ∈ VGS we infer its role w.r.t . H by obtaining the orbit of its mapping f ( v ) in H , OrbH ( f ( v ) ) . By counting all the possible appearances of different orbits in v , we obtain the vertex structural feature xVH ( v ) of v , defined as follows : xVH , i ( v ) = | { GS ' H : v ∈ VGS s.t . f ( v ) ∈ OVH , i } | |Aut ( H ) | , i = 1 , . . . , dH . ( 1 ) We divide the counts by the number of the automorphisms of H , since for every matched subgraph GS there will always be |Aut ( H ) | different ways to map it to H , thus these repetitions will be uninformative . By combining the counts from different substructures inH and different orbits , we obtain the feature vector xV ( v ) = [ xVH1 ( v ) , . . . , x V HK ( v ) ] ∈ ND×1 of dimension D = ∑ Hi∈H dHi . Similarly , we can define edge structural features xEH , i ( { u , v } ) by counting occurrences of edge automorphism orbits : xEH , i ( { u , v } ) = | { GS ' H : { u , v } ∈ EGS s.t . { f ( u ) , f ( v ) } ∈ OEH , i } | |Aut ( H ) | , ( 2 ) and the combined edge features xE ( { u , v } ) = [ xEH1 ( { u , v } ) , . . . , x E HK ( { u , v } ) ] . An example of vertex and edge structural features is illustrated in Figure 1 . Structure-aware message passing : The key building block of our architecture is the graph substructure layer , defined in a general manner as a Message Passing Neural Network ( MPNN ) ( Gilmer et al. , 2017 ) , where now the messages from the neighbouring nodes also contain the structural information . In particular , each node v updates its state ht ( v ) by combining its previous state with the aggregated messages : ht+1 ( v ) = UPt+1 ( ht ( v ) , MSGt+1 ( v ) ) , where the UPt+1 function is a multilayer perceptron ( MLP ) and the message aggregation is a summation of features transformed by an MLP M t+1 as follows : MSGt+1 ( v ) = ∑ u∈N ( v ) M t+1 ( ht ( v ) , ht ( u ) , xV ( v ) , xV ( u ) , e ( { u , v } ) ) ( 3 ) MSGt+1 ( v ) = ∑ u∈N ( v ) M t+1 ( ht ( v ) , ht ( u ) , xE ( { u , v } ) , e ( { u , v } ) ) ( 4 ) where the two variants , named GSN-v and GSN-e , correspond to vertex- or edge-counts , respectively , and e ( { u , v } ) denotes edge features . The two variants are analogous to absolute and relative positional encodings in language models ( Shaw et al. , 2018 ; Dai et al. , 2019 ) . It is important to note here that contrary to identifier-based GNNs ( Loukas , 2020 ; Sato et al. , 2020 ; Dasoulas et al. , 2020 ) that obtain universality at the expense of permutation equivariance ( since the identifiers are arbitrarily chosen with the sole requirement of being unique ) , GSNs retain the attractive inductive bias of the permutation equivariant property . This stems from the fact that the process generating our structural identifiers ( i.e . subgraph isomorphism ) is permutation equivariant itself ( proof provided in the appendix ) . How powerful are GSNs ? We now turn to the expressive power of GSNs in comparison to MPNNs and the WL tests , a key tool for the theoretical analysis of the expressivity of graph neural networks so far . Since GSN is a generalisation of MPNNs , it is easy to see that it is at least as powerful . Importantly , GSNs have the capacity to learn functions that traditional MPNNs can not learn . The following observation derives directly from the analysis of the counting abilities of the 1-WL test ( Arvind et al. , 2019 ) and its extension to MPNNs ( Chen et al. , 2020 ) ( for proofs , see Appendix A.2 ) . Proposition 3.1 . GSN is strictly more powerful than MPNN and the 1-WL test when one of the following holds : a ) H is any graph except for star graphs of any size and structural features are inferred by subgraph matching , i.e . we count all subgraphs GS ∼= H for which it holds that EGS ⊆ EG , or b ) H is any graph except for single edges and single nodes and structural features are inferred by induced subgraph matching , i.e . we count all subgraphs GS ∼= H for which it holds that EGS = EG ∩ ( VGS × VGS ) . An important question is which substructures are the most informative and whether they can completely characterise the graph . As of today , we are not aware of any results in graph theory that can guarantee the reconstruction of a graph from a smaller collection of its subgraphs . In fact , the Reconstruction Conjecture ( Kelly et al. , 1957 ; Ulam , 1960 ) , states that a graph with size n ≥ 3 can be reconstructed from its vertex-deleted subgraphs . Moreover , it is known ( Dasoulas et al. , 2020 ; Chen et al. , 2019 ) that solving graph isomorphism is equivalent to universal approximation of permutation invariant functions defined on graphs . Thus , we can state the following for GSN : Corollary 3.1 . If the Reconstruction Conjecture holds , GSN can distinguish all non-isomorphic graphs of size n and is therefore universal , whenH contains all substructures of size k = n− 1 . How to choose the substructures ? The Reconstruction Conjecture has only been proven for n ≤ 11 ( McKay , 1997 ) and still remains open for larger graphs , while to the best of our knowledge , there is no similar hypothesis for smaller values of k. However , in practice we observe that small substructures of size k = O ( 1 ) are sufficient both for resolving hard instances of graph isomorphism as well as for tasks related to real-world networks . In particular , although our method does not attempt to align with the WL hierarchy , an interesting observation that we make is that small substructures have the capacity to distinguish strongly regular graphs , where 2-FWL provably fails ( proven in Appendix B.3 ) . This is illustrated in Figure 2 ( right ) , where counting 4-cliques is sufficient to disambiguate this exemplary pair , and in Section 5 , where a large scale experimental study is conducted . In real-world scenarios , using too many and too large subgraphs will not only be expensive to compute , but they might also lead to overfitting . As an alternative , one can choose only the most discriminative subgraphs , i.e . the ones that can achieve the maximum possible node disambiguation , similarly to identifier-based approaches . Note that this approach solely takes into account expressivity and does not provide any guarantee about generalisation . This is empirically validated in Figure 4 , where we observe that choosing substructures with strong node disambiguation properties allows for better fitting of the training data , but the test set performance does not necessarily improve . Aiming at better generalisation , it is desirable to make use of substructures for which there is substantial prior knowledge of their importance in certain network distributions and have been observed to be intimately related to various properties . For example , small substructures ( graphlets ) have been extensively analysed in protein-protein interaction networks ( Pržulj et al. , 2004 ) , triangles and cliques characterise the structure of ego-nets and social networks in general ( Granovetter , 1982 ) , simple cycles ( rings ) are central in molecular distributions , directed and temporal motifs have been shown to explain the working mechanisms of gene regulatory networks , biological neural networks , transportation networks and food webs ( Milo et al. , 2002 ; Paranjape et al. , 2017 ; Benson et al. , 2016 ) . In Figure 6 in the Appendix , we showcase the importance of these inductive biases : a cycle-based GSN predicting molecular properties achieves smaller generalisation gap compared to a traditional MPNN , while at the same time generalising better with less training data . Choosing the best substructure collection is still an open problem ; various heuristics can be used ( motif frequencies , feature selection strategies ) or ideally by learning the substructure dictionary in an end-to-end manner . Answering this question is left for future work . GSN-v vs GSN-e : Finally , we examine the expressive power of the two proposed variants . A crucial observation that we make , is that for each graph H in the collection , the vertex structural identifiers can be reconstructed by the corresponding edge identifiers . Thus , we can show that for every GSN-v there exists a GSN-e that can simulate the behaviour of the former ( proof provided in Appendix A.3 ) . Proposition 3.2 . For a given subgraph collectionH , let CV the set of functions that can be expressed by a GSN-v with arbitrary depth and with , and CE the set of functions that can be expressed by a GSN-e with the same properties . Then , it holds that CE ⊇ CV , or in other words GSN-e is at least as expressive as GSN-v . Complexity : The complexity of GSN comprises two parts : precomputation ( substructure counting ) and training/testing . The key appealing property is that training and inference are linear w.r.t the number of edges , O ( |E| ) , as opposed to higher-order methods ( Maron et al. , 2019a ; Morris et al. , 2019 ) with O ( nk ) training complexity and relational pooling ( Murphy et al. , 2019 ) with O ( n ! ) training complexity in absence of approximations . The worst-case complexity of subgraph isomorphism of fixed size k is O ( nk ) , by examining all the possible k-tuples in the graph . However , for specific types of subgraphs , such as paths and cycles , the problem can be solved even faster ( see e.g . Giscard et al . ( 2019 ) ) . Approximate counting algorithms are also widely used , especially for counting frequent network motifs ( Kashtan et al. , 2004 ; Wernicke , 2005 ; 2006 ; Wernicke & Rasche , 2006 ) , and can provide a considerable speed-up . Furthermore , recent neural approaches ( Ying et al. , 2020b ; a ) provide fast approximate counting . In our experiments , we performed exact counting using the common isomorphism algorithm VF2 ( Cordella et al. , 2004 ) . Although its worst case complexity is O ( nk ) , it scales better in practice , for instance when the candidate subgraph is infrequently matched or when the graphs are sparse , and is also trivially parallelisable . In Figure 3 , we show a quantitative analysis of the empirical runtime of the counting algorithm against the worst case , for three different graph distributions : molecules , protein contact maps , social networks . It is easy to see that when the graphs are sparse ( for the first two cases ) and the number of matches is small , the algorithm is significantly faster than the worst case , while it scales better with the size of the graph n. Even , in the case of social networks , where several examples are near-complete graphs , both the runtime and the growth w.r.t both n and k are better than the worst case . Overall , the preprocessing computational burden in most of the cases remains negligible for relatively small and sparse graphs , as it is the case of molecules .

Summary: This work proposes the Graph Substructure Network (GSN) to encode structural roles for different nodes so that the expressivity of Graph Neural Networks is improved. The core idea is to count the number of certain substructures, such as cycles, cliques, and triangles. Then the proposed MPNN encodes such substructure counting information into the message passing. Experimental results show that the proposed method can obtain better performance than the comparing methods.