Recursive Neighborhood Pooling for Graph Representation Learning

1 INTRODUCTION . Graph Neural Networks ( GNNs ) are powerful tools for graph representation learning ( Scarselli et al. , 2008 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ) , and have been successfully used in applications such as encoding molecules , simulating physics , social network analysis , knowledge graphs , and many others ( Duvenaud et al. , 2015 ; Defferrard et al. , 2016 ; Battaglia et al. , 2016 ; Jin et al. , 2018 ) . An important class of GNNs is the set of Message Passing Graph Neural Networks ( MPNNs ) ( Gilmer et al. , 2017 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Xu et al. , 2019 ; Scarselli et al. , 2008 ) , which follow an iterative message passing scheme to compute a graph representation . Despite the empirical success of MPNNs , their expressive power has been shown to be limited . For example , their discriminative power , at best , corresponds to the one-dimensional Weisfeiler-Leman ( 1-WL ) graph isomorphism test ( Xu et al. , 2019 ; Morris et al. , 2019 ) , so they can not , e.g. , distinguish regular graphs . Moreover , they also can not count any induced subgraph with at least three vertices ( Chen et al. , 2020 ) , or learn graph parameters such as clique information , diameter , or shortest cycle ( Garg et al. , 2020 ) . Still , in several applications , e.g . in computational chemistry , materials design or pharmacy ( Elton et al. , 2019 ; Sun et al. , 2020 ; Jin et al. , 2018 ) , we aim to learn functions that depend on the presence or count of specific substructures . To strengthen the expressive power of GNNs , higher-order representations such as k−GNNs ( Morris et al. , 2019 ) and k−Invariant Graph Networks ( k−IGNs ) ( Maron et al. , 2019 ) have been proposed . k−GNNs are inspired by the k-dimensional WL ( k−WL ) graph isomorphism test , a message passing algorithm on k−tuples of vertices , and k−IGNs are based on equivariant linear layers of a feed-forward neural network applied to the input graph as a matrix , and are at least as powerful as k−GNN . These models are provably more powerful than MPNNs and can , e.g. , count any induced substructure with at most k vertices . But , this power comes at the computational cost of at least Ω ( nk ) operations for n vertices . The necessary tradeoffs between expressive power and computational complexity are still an open question . The expressive power of a GNN is often measured in terms of a hierarchy of graph isomorphism tests , i.e. , by comparing it to a k−WL test . Yet , there is limited knowledge about how the expressive power of higher-order graph isomorphism tests relates to various functions of interest ( Arvind et al. , 2020 ) . A different approach is to take the perspective of specific functions that are of practical interest , and quantify a GNN ’ s expressive power via those . Here , we focus on counting induced substructures to measure the power of a GNN , as proposed in ( Chen et al. , 2020 ) . In particular , we study whether it is possible to count given substructures with a GNN whose complexity is between that of MPNNs and the existing higher-order GNNs . To this end , we study the scheme of many higher-order GNNs ( Morris et al. , 2019 ; Chen et al. , 2020 ) : select a collection of subgraphs of the input graph , encode these , and ( possibly iteratively ) compute a learned function on this collection . First , we propose a new such class of GNNs , Recursive Neighborhood Pooling Graph Neural Networks ( RNP-GNNs ) . Specifically , RNP-GNNs represent each vertex by a representation of its neighborhood of a specific radius . Importantly , this neighborhood representation is computed recursively from its subgraphs . As we show , RNP-GNNs can count any induced substructure with at most k vertices . Moreover , for any set of substructures with at most k vertices , there is a specifiable RNP-GNN that can count them . This flexibility allows to design a GNN that is adapted to the power needed for the task of interest , in terms of counting ( induced ) substructures . The Local Relational Pooling ( LRP ) architecture too has been introduced with the goal of counting substructures ( Chen et al. , 2020 ) . While it can do so , it is polynomial-time only if the encoded neighborhoods are of size o ( log ( n ) ) . In contrast , RNP-GNNs use almost linear operations , i.e. , n1+o ( 1 ) , if the size of each encoded neighborhood is no ( 1 ) . This is an exponential theoretical improvement in the tolerable size of neighborhoods , and a significant improvement over the complexity of O ( nk ) in k−GNN and k−IGN . Finally , we take a broader perspective and provide an information-theoretic lower bound on the complexity of a general class of GNNs that can provably count substructures with at most k vertices . This class includes GNNs that represent a given graph by aggregating a number of encoded graphs , where the encoded graphs are related to the given graph with an arbitrary function . In short , in this paper , we make the following contributions : • We introduce Recursive Neighborhood Pooling Graph Neural Networks ( RNP-GNNs ) , a flexible class of higher-order graph neural networks , that provably allow to design graph representation networks with any expressive power of interest , in terms of counting ( induced ) substructures . • We show that RNP-GNNs offer computational gains over existing models that count substructures : an exponential improvement in terms of the “ tolerable ” size of the encoded neighborhoods compared to LRP networks , and much less complexity in sparse graphs compared to k−GNN and k−IGN . • We provide an information-theoretic lower bound on the complexity of a general class of GNN that can count ( induced ) substructures with at most k vertices . 2 BACKGROUND . Message Passing Graph Neural Networks . Let G = ( V , E , X ) be a labeled graph with |V| = n vertices . Here , Xv ∈ X denotes the initial label of v ∈ V , where X ⊆ N is a ( countable ) domain . A typical Message Passing Graph Neural Network ( MPNN ) first computes a representation of each vertex , and then aggregates the vertex representations via a readout function into a representation of the entire graph G. The representation h ( i ) v of each vertex v ∈ V is computed iteratively by aggregating the representations h ( i−1 ) u of the neighboring vertices u : m ( i ) v = AGGREGATE ( i ) ( { { h ( i−1 ) u : u ∈ N ( v ) } } ) , ( 1 ) h ( i ) v = COMBINE ( i ) ( h ( i−1 ) v , m ( i ) v ) , ( 2 ) for any v ∈ V , for k iterations , and with h ( 0 ) v = Xv . The AGGREGATE/COMBINE functions are parametrized , learnable functions , and { { . } } denotes a multi-set , i.e. , a set with ( possibly ) repeating elements . A graph-level representation can be computed as hG = READOUT ( h ( k ) v : v ∈ V ) , ( 3 ) where READOUT is a learnable function . For representational power , it is important that the learnable functions above are injective , which can be achieved , e.g. , if the AGGREGATE function is a summation and COMBINE is a weighted sum concatenated with an MLP ( Xu et al . ( 2019 ) ) . Higher-Order GNNs . To increase the representational power of GNNs , several higher-order GNNs have been proposed . In a k−GNN , a message passing algorithm is applied to the k−tuples of vertices , in a similar fashion as GNNs do on vertices ( Morris et al. , 2019 ) . At initialization , each k−tuple is labeled with its type , that is , two k−tuples are labeled differently if their induced subgraphs are not isomorphic . As a result , k−GNNs can count ( induced ) substructures with at most k vertices even at initialization . Another class of higher-order networks are k−IGNs , which are constructed with linear invariant/equivariant feed-forward layers , whose inputs consider graphs via adjacency matrices ( Maron et al. , 2019 ) . k−IGNs are at least as powerful as k−GNNs , and hence they too can count substructures with at most k vertices . However , both methods need O ( nk ) operations . Specifically for counting substructures , Chen et al . ( 2020 ) propose Local Relational Pooling ( LRP ) networks . LRPs apply Relational Pooling ( RP ) networks ( Murphy et al. , 2019b ; a ) on the neighborhoods around each vertex . RP networks use permutation-variant functions and convert them to a permutation-invariant function by summing over all permutations . This summation is computationally expensive . 3 OTHER RELATED WORKS . Expressive power . Several other works have studied the expressive power of GNNs ( Azizian & Lelarge , 2020 ) . Scarselli et al . ( 2009 ) extend universal approximation from feedforward networks to GNNs , using the notion of unfolding equivalence . Chen et al . ( 2019 ) establish an equivalence between the graph isomorphism problem and the power to approximate permutation invariant functions on graphs . Maron et al . ( 2019 ) and Keriven & Peyré ( 2019 ) propose higher-order , tensor-based GNN models that provably achieve universal approximation of permutation-invariant functions on graphs , and Loukas ( 2019 ) studies expressive power under depth and width restrictions . Studying GNNs from the perspective of local algorithms , Sato et al . ( 2019 ) show that GNNs can approximate solutions to certain combinatorial optimization problems . Subgraphs and GNNs . The idea of considering local neighborhoods to have better representations than MPNNs is considered in several works ( Liu et al. , 2019 ; Monti et al. , 2018 ; Liu et al. , 2020 ; Yu et al. , 2020 ; Meng et al. , 2018 ; Cotta et al. , 2020 ; Alsentzer et al. , 2020 ; Huang & Zitnik , 2020 ) . For example , in link prediction , one can use local neighborhoods around links and apply GNNs , as suggested in ( Zhang & Chen , 2018 ) . A novel method based on combining GNNs and a clustering algorithm is proposed in ( Ying et al . ( 2018 ) ) . For graph comparison ( i.e. , testing whether a given possibly large subgraph exists in the given model ) , Ying et al . ( 2020 ) compare the outputs of GNNs for small subgraphs of the two graphs . To improve the expressive power of GNNs , Bouritsas et al . ( 2020 ) use features that are counts of specific subgraphs of interest . Another related work is ( Vignac et al. , 2020 ) , where an MPNN is strengthened by learning local context matrices around vertices . 4 RECURSIVE NEIGHBORHOOD POOLING . Next , we construct Recursive Neighborhood Pooling Graph Neural Networks ( RNP-GNNs ) , GNNs that can count any set of induced substructures of interest , with lower complexity than previous models . We represent each vertex by a representation of its radius r1-neighborhood , and then combine these representations . The key question is how to encode these local neighborhoods in a vector representation . To do so , we introduce a new idea : we view local neighborhoods as small subgraphs , and recursively apply our model to encode these neighborhood subgraphs . When encoding the local subgraphs , we use a different radius r2 , and , recursively , a sequence of radii ( r1 , r2 , . . . , rt ) ∈ Nt to obtain the final representation h ( t ) v of vertices after t recursion steps . While MPNNs also encode a representation of a local neighborhood of certain radius , the recursive representations differ as they essentially take into account intersections of neighborhoods . As a result , as we will see in Section 5.1 , they retain more structural information and are more expressive . Models such as k-GNN and LRP also compute encodings of subgraphs , and then update the resulting representations via message passing . We can do the same with the neighborhood representations computed by RNP-GNNs to encode more global information , although our representation results in Section 5.1 hold even without that . In Section 6 , we will compare the computational complexity of RNP-GNN and these other models . Formally , an RNP-GNN is a parametrized learnable function f ( . ; θ ) : Gn → Rd , where Gn is the set of all labeled graphs on n vertices . Let G = ( V , E , X ) be a labeled graph with |V| = n vertices , and let h ( 0 ) v = Xv be the initial representation of each vertex v. LetNr ( v ) denote the neighborhood of radius r of vertex v , and let G ( t−1 ) v ( Nr1 ( v ) \ { v } ) ) denote the induced subgraph of G on the set of vertices Nr1 ( v ) \ { v } , with augmented vertex label X ( t−1 ) u = ( h ( t−1 ) u ,1 [ ( u , v ) ∈ E ] ) for any u ∈ Nr1 ( v ) \ { v } . This means we add information about whether vertices are direct neighbors ( with distance one ) of v. Given a recursion sequence ( r1 , r2 , . . . , rt ) or radii , the representations are updated as m ( t ) v = RNP-GNN ( t−1 ) ( G ( t−1 ) v ( Nr1 ( v ) \ { v } ) ) , ( 4 ) h ( t ) v = COMBINE ( t ) ( h ( t−1 ) v , m ( t ) v ) , ( 5 ) for any v ∈ V , and hG = READOUT ( { { h ( t ) v : v ∈ V } } ) . ( 6 ) Different from MPNNs , the recursive update ( 4 ) is in general applied to a subgraph , and not a multi-set of vertex representations . RNP-GNN ( t−1 ) is an RNP-GNN with recursion parameters ( r2 , . . . , rt ) ∈ Nt−1 . The final READOUT is an injective , permutation-invariant learnable multi-set function . If t = 1 , then m ( t ) v = AGGREGATE ( t ) ( { { ( h ( t−1 ) u ,1 [ ( u , v ) ∈ E ] ) : u ∈ Nr1 ( v ) } } ) ( 7 ) is a permutation-invariant aggregation function as used in MPNNs , only over a potentially larger neighborhood . For r1 = 1 and t = 1 , RNP-GNN reduces to MPNN . In Figure 1 , we illustrate an RNP-GNN with recursion parameters ( 2 , 2 , 1 ) as an example . We also provide pseudocode for RNP-GNNs in Appendix C .

Summary: The goal of the paper is to show that GNN's (without exponential computational complexity) can be constructed with the ability to count subgraphs. To this effect, the authors propose a principled neighborhood pooling strategy and theoretically characterize their expressive power - with respect to other models proposed earlier. More specifically, the authors propose a recursive neighborhood pooling strategy which characterizes graphs based on the counts of subgraphs . Furthermore, they show that if the tuple of recursion parameters are chosen well, their proposed model can capture all induced subgraphs (universality) of sizes smaller than the first value in the tuple of recursion parameters plus 1 - and show a relationship to the reconstruction conjecture (Kelly et al. 1957). The authors also provide a bound on the number of iterations required to learn the expressive representations.