Automorphic Equivalence-aware Graph Neural Network

1 Introduction . The past few years have witnessed the phenomenal success of GNNs in numerous graph learning tasks , such as node classification [ 24 ] , link prediction [ 59 ] , and community detection [ 19 ] , which is largely due to their capability of simultaneously modelling the connecting patterns and feature distribution in each node ’ s local neighborhood . As a result , it leads to a surge of interests from both academia and industry to develop more powerful GNN models [ 52 ] . Despite of various architectures , the most popular GNNs , like GCN [ 24 ] , GraphSAGE [ 19 ] , GAT [ 48 ] and GIN [ 55 ] , apply permutation invariant aggregate function on each node ’ s local neighborhood to learn node embeddings , which leads to concerns about their representational power [ 17 , 55 ] . In this paper , we investigate GNN ’ s expressiveness from an important but largely overlooked angle , i.e. , the capacity to distinguish automorphic equivalence within each node ’ s local neighborhood . Automorphic equivalence ( AE ) [ 14 ] is a classic concept to differentiate the structural role of each node in a given graph . Specifically , two nodes are considered to be AE only if they are interchangeable in some index permutations that preserve the connection matrix , i.e. , graph automorphisms [ 35 ] . AE can identify the nodes that exhibit identical structural features in a graph , which makes it a central topic in computational biologist , social network analysis and other scientific domains [ 33 , 34 ] . For example , empirical studies show AE is an important indicator of social position and behavior similarity in social network [ 16 , 37 ] , which thus might significantly benefit GNN architecture design . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Empirically efficient heuristics have been proposed to identify the AE in moderate scale graphs by enumerating all the possible automorphisms [ 6 , 45 ] . However , previous analytic methods only classify nodes into categorical equivalence sets . Although categorical features can be jointly optimized with GNNs under various frameworks [ 30 , 54 ] , little previous efforts are invested to principally incorporate AE into GNNs . Thus , we aim to design a novel GNN model that is provably expressive in capturing AE features and can be tuned in a data-dependent manner based on the graph data and targeted applications , which will effectively allow us to learn expressive function to harness the power of AE feature . Here , we propose GRaph AutomorPhic Equivalent network , i.e. , GRAPE , a novel variant of GNN that can learn expressive representation by differentiating the automorphic equivalences of each node ’ s neighbors . First , GRAPE extends the classic AE concept into a localized setting , i.e. , Ego-AE , to accommodate the local nature of GNNs . Specifically , Ego-AE identifies the local neighborhoods of each node by mapping with given subgraph templates and then partitions the neighboring nodes into Ego-AE sets based on the graph automorphisms in neighborhood . Second , we design a learnable AEaware aggregators to model the node features in these Ego-AE sets , which adaptively assigns different weights to neighboring nodes in different Ego-AE sets and explicitly model the interdependency among them . Moreover , in order to capture complex structural features , GRAPE proposes to fuse the embeddings learned from Ego-AE sets identified by different subgraph templates with a squeeze-andexcitation module [ 22 ] . Finally , to alleviate the barrier of subgraph template design , we propose an efficient genetic algorithm to automatically search for optimal subgraph templates . Specifically , it gradually optimizes a randomly initiated population of subgraph templates by iteratively exploring the adjacency of good performing candidates and eliminating the bad performing ones . To accelerate the search process , we further design an incremental subgraph matching algorithm that can leverage the similarity between subgraphs to greatly reduce the complexity of finding matched instances . We theoretically prove that the proposed GRAPE is expressive in terms of learning distinct representations for nodes with different Ego-AE sets , which fundamentally makes up the shortcomings of popular GNN variants , e.g. , GCN [ 24 ] , GraphSAGE [ 19 ] , GAT [ 48 ] and GIN [ 55 ] . Moreover , we empirically validate GRAPE on eight real-world datasets , which cover the scenarios of social network , citation network and e-commerce co-purchase network . Experiments show GRAPE is consistently the best performing model across all datasets with up to 26.7 % accuracy improvement . Besides , case studies indicate GRAPE can effectively differentiate the structural roles of each node ’ s neighbors . Moreover , the proposed genetic algorithm efficiently generates high quality subgraph templates that have comparable performance with the hand-crafted ones . 2 Related Works . In the sequel , we define a graph as G = ( V , E ) , where V = { v1 , · · · , vn } is the set of nodes and E = { ( vi , vj ) } is the set of edges . Let the feature vector of node vi be X ( vi ) , and N ( vi ) represents the set of vi ’ s neighbor nodes . N is the number of node and M is the embedding size . 2.1 Automorphic Equivalence ( AE ) in Graph Analysis . Here , we investigate one of the most popular structural equivalence concepts , i.e. , automorphic equivalence ( AE ) [ 14 ] which plays a central role in computational biologist , social network analysis and other scientific domains [ 33 , 34 ] . The most interesting part of AE is that it identifies the nodes with exact same structural patterns , e.g. , degree and centrality [ 15 ] , but not necessarily connecting to the same neighboring nodes . Basically , AE is defined as follows . Definition 2.1 . Given a graph G = ( V , E ) , an automporphism π ( ? ) is a node permutation that preserves the adjacency matrix , i.e. , the permuted nodes π ( va ) and π ( vb ) are connected if and only if nodes va and vb are connected . Two nodes va and vb are considered to be Automorphic Equivalence ( AE ) if there is a graph automorphism that maps one onto the other , i.e. , π ( va ) = vb . Nodes in V that are AE with each other constitutes a AE set . An example of such sets is in Figure 1 ( a ) . Specifically , AE sets can be identified by enumerating the automorphism group of the given graph with efficient Nauty algorithm [ 35 ] , and the nodes that are mapped onto each others in different automorphisms will be partitioned into same AE sets . Previous works have attempted to preserve the structural similarities on graph via various node embedding algorithms [ 12 , 2 , 40 ] . Recently , GraphWave [ 12 ] was proposed to leverage wavelet diffusion patterns to capture the structural roles in node representations . Besides , role2vec [ 2 ] introduced a generalized feature-based random walks that aims to represent the structural similarities among nodes . However , the connection between AE and GNN has not been examined in existing literature . Moreover , AE is defined on whole graph level , which is infeasible to compute in large graphs and goes against the inherent local nature of most GNN frameworks . In this paper , we aim to extend the AE concept to local setting , and propose a novel GNN model to harness its power . 2.2 The Expressive Power of GNNs . The recent success of GNNs draws increasing interests in investigating their capability in capturing structural properties [ 55 , 17 ] . Specifically , message passing neural network ( MPNN ) is the most popular GNN framework [ 18 , 19 , 55 ] . Let AGG be the aggregate function that collects feature from neighbors , and COMB be the combine function that integrates each node ’ s self feature with those from AGG . Generally , MPNN generates representation for a node vt at k-th layer as hkvt = COMB ( hk−1vt , AGG ( H k−1 vt ) ) , ( 1 ) where h0 ( vi ) = X ( vi ) , andHk−1vt = { hk−1vj |vj ∈ N ( vt ) } . Existing MPNNs use local permutation invariant AGGs to compute node embeddings , which subsumes a large class of popular GNN models such as GCN [ 24 ] , GraphSAGE [ 19 ] , GAT [ 48 ] , Geniepath [ 31 ] and GIN [ 55 ] . The family of MPNNs is proven to be theoretically linked to Weisfeiler-Lehman ( WL ) subtree kernel [ 19 ] . Subsequently , they are at most as powerful as 1-WL test on discriminating graph isomorphisms [ 55 ] . More recently , several attempts have been made to improve the expressiveness of GNN beyond MPNN framework . They can mainly be classified into two categories : augmenting node features and designing more powerful architectures . In terms of augmenting node feature , recent works proposed to introduce various additional feature [ 42 , 25 , 59 ] . 3D-GCN uses additional 3D point cloud feature to differentiate neighbors and facilitate learnable graph kernels [ 29 ] . Moreover , GNN variants can in theory achieve universal approximation on graph by equipping nodes with randomly initialized feature vector [ 43 ] , but they are difficult to generalize to different graphs in practice [ 5 ] . Previous work also proposed to augment node feature with substructure count [ 5 ] , which however can not reveal the local structural roles in each node ’ s neighborhood . On the other hand , the previous efforts of designing more powerful GNN architecture are dedicated to different structural properties , e.g. , graph isomorphism [ 55 ] and graph moment [ 11 ] . Although AE is an important concept for graph data analysis , it has not been addressed by previous GNN research . More detailed comparison with the existing GNN variants is provided in Appendix A . However , previous works have shown that identifying automorphic equivalence is a strictly more difficult task than discriminating graph isomorphisms [ 46 ] . Specifically , two AE nodes always have isomorphic neighborhood , while the nodes with isomorphic neighborhood are not necessary AE [ 15 ] . Therefore , it raises concerns about MPNN ’ s expressive power of AE feature , which has not been adequately investigated in previous research . In this paper , we aim to design a novel GNN model that is provably expressive in modeling Ego-AE , which falling in the category of designing novel GNN architecture . To the best of our knowledge , we are the first to empower GNN with the capability of capturing automorphic equivalence . 2.3 Genetic Algorithm . Genetic algorithm [ 13 ] is a widely adopted algorithm for combinatorial optimization problems , e.g. , traveling salesman problem . Recently , it has also been used to tune hyper-parameters [ 32 ] and search neural architectures for deep networks [ 57 , 53 ] . Basically , genetic algorithm mimics the natural selection process to iteratively to search for better solutions by exploring the adjacency of promising candidates and eliminating the worst-performing ones [ 10 ] . Therefore , it can iteratively optimize the candidate population from parent generation to children generation . Specifically , genetic algorithms are often made up the following four components : ( i ) Mutation : explores the adjacency of promising candidates in parent generation by generating slightly different candidates in children generation ; ( ii ) Crossover : search for different combinations of genetic features in the candidates in parent generation ; ( iii ) Evaluation : measure the fitness of candidates in given tasks ; ( iv ) Selection : eliminating the candidates with worst performance . Here , we design a genetic algorithm to automatically optimize the subgraph templates in the proposed GRAPE . It effectively allows us to search the architecture of GRAPE in a data-dependent manner , which significantly reduces the barrier of subgraph template design .

Summary:  This paper studies the problem of improving the expressive power of GNN models. It computes the automorphism equivalence set for each node and uses the set to enhance the message passing. This method is proven to be more expressive than standard MPNN.