Recursive Neighborhood Pooling for Graph Representation Learning

1 INTRODUCTION . Graph Neural Networks ( GNNs ) are powerful tools for graph representation learning ( Scarselli et al. , 2008 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ) , and have been successfully used in applications such as encoding molecules , simulating physics , social network analysis , knowledge graphs , and many others ( Duvenaud et al. , 2015 ; Defferrard et al. , 2016 ; Battaglia et al. , 2016 ; Jin et al. , 2018 ) . An important class of GNNs is the set of Message Passing Graph Neural Networks ( MPNNs ) ( Gilmer et al. , 2017 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Xu et al. , 2019 ; Scarselli et al. , 2008 ) , which follow an iterative message passing scheme to compute a graph representation . Despite the empirical success of MPNNs , their expressive power has been shown to be limited . For example , their discriminative power , at best , corresponds to the one-dimensional Weisfeiler-Leman ( 1-WL ) graph isomorphism test ( Xu et al. , 2019 ; Morris et al. , 2019 ) , so they can not , e.g. , distinguish regular graphs . Moreover , they also can not count any induced subgraph with at least three vertices ( Chen et al. , 2020 ) , or learn graph parameters such as clique information , diameter , or shortest cycle ( Garg et al. , 2020 ) . Still , in several applications , e.g . in computational chemistry , materials design or pharmacy ( Elton et al. , 2019 ; Sun et al. , 2020 ; Jin et al. , 2018 ) , we aim to learn functions that depend on the presence or count of specific substructures . To strengthen the expressive power of GNNs , higher-order representations such as k−GNNs ( Morris et al. , 2019 ) and k−Invariant Graph Networks ( k−IGNs ) ( Maron et al. , 2019 ) have been proposed . k−GNNs are inspired by the k-dimensional WL ( k−WL ) graph isomorphism test , a message passing algorithm on k−tuples of vertices , and k−IGNs are based on equivariant linear layers of a feed-forward neural network applied to the input graph as a matrix , and are at least as powerful as k−GNN . These models are provably more powerful than MPNNs and can , e.g. , count any induced substructure with at most k vertices . But , this power comes at the computational cost of at least Ω ( nk ) operations for n vertices . The necessary tradeoffs between expressive power and computational complexity are still an open question . The expressive power of a GNN is often measured in terms of a hierarchy of graph isomorphism tests , i.e. , by comparing it to a k−WL test . Yet , there is limited knowledge about how the expressive power of higher-order graph isomorphism tests relates to various functions of interest ( Arvind et al. , 2020 ) . A different approach is to take the perspective of specific functions that are of practical interest , and quantify a GNN ’ s expressive power via those . Here , we focus on counting induced substructures to measure the power of a GNN , as proposed in ( Chen et al. , 2020 ) . In particular , we study whether it is possible to count given substructures with a GNN whose complexity is between that of MPNNs and the existing higher-order GNNs . To this end , we study the scheme of many higher-order GNNs ( Morris et al. , 2019 ; Chen et al. , 2020 ) : select a collection of subgraphs of the input graph , encode these , and ( possibly iteratively ) compute a learned function on this collection . First , we propose a new such class of GNNs , Recursive Neighborhood Pooling Graph Neural Networks ( RNP-GNNs ) . Specifically , RNP-GNNs represent each vertex by a representation of its neighborhood of a specific radius . Importantly , this neighborhood representation is computed recursively from its subgraphs . As we show , RNP-GNNs can count any induced substructure with at most k vertices . Moreover , for any set of substructures with at most k vertices , there is a specifiable RNP-GNN that can count them . This flexibility allows to design a GNN that is adapted to the power needed for the task of interest , in terms of counting ( induced ) substructures . The Local Relational Pooling ( LRP ) architecture too has been introduced with the goal of counting substructures ( Chen et al. , 2020 ) . While it can do so , it is polynomial-time only if the encoded neighborhoods are of size o ( log ( n ) ) . In contrast , RNP-GNNs use almost linear operations , i.e. , n1+o ( 1 ) , if the size of each encoded neighborhood is no ( 1 ) . This is an exponential theoretical improvement in the tolerable size of neighborhoods , and a significant improvement over the complexity of O ( nk ) in k−GNN and k−IGN . Finally , we take a broader perspective and provide an information-theoretic lower bound on the complexity of a general class of GNNs that can provably count substructures with at most k vertices . This class includes GNNs that represent a given graph by aggregating a number of encoded graphs , where the encoded graphs are related to the given graph with an arbitrary function . In short , in this paper , we make the following contributions : • We introduce Recursive Neighborhood Pooling Graph Neural Networks ( RNP-GNNs ) , a flexible class of higher-order graph neural networks , that provably allow to design graph representation networks with any expressive power of interest , in terms of counting ( induced ) substructures . • We show that RNP-GNNs offer computational gains over existing models that count substructures : an exponential improvement in terms of the “ tolerable ” size of the encoded neighborhoods compared to LRP networks , and much less complexity in sparse graphs compared to k−GNN and k−IGN . • We provide an information-theoretic lower bound on the complexity of a general class of GNN that can count ( induced ) substructures with at most k vertices . 2 BACKGROUND . Message Passing Graph Neural Networks . Let G = ( V , E , X ) be a labeled graph with |V| = n vertices . Here , Xv ∈ X denotes the initial label of v ∈ V , where X ⊆ N is a ( countable ) domain . A typical Message Passing Graph Neural Network ( MPNN ) first computes a representation of each vertex , and then aggregates the vertex representations via a readout function into a representation of the entire graph G. The representation h ( i ) v of each vertex v ∈ V is computed iteratively by aggregating the representations h ( i−1 ) u of the neighboring vertices u : m ( i ) v = AGGREGATE ( i ) ( { { h ( i−1 ) u : u ∈ N ( v ) } } ) , ( 1 ) h ( i ) v = COMBINE ( i ) ( h ( i−1 ) v , m ( i ) v ) , ( 2 ) for any v ∈ V , for k iterations , and with h ( 0 ) v = Xv . The AGGREGATE/COMBINE functions are parametrized , learnable functions , and { { . } } denotes a multi-set , i.e. , a set with ( possibly ) repeating elements . A graph-level representation can be computed as hG = READOUT ( h ( k ) v : v ∈ V ) , ( 3 ) where READOUT is a learnable function . For representational power , it is important that the learnable functions above are injective , which can be achieved , e.g. , if the AGGREGATE function is a summation and COMBINE is a weighted sum concatenated with an MLP ( Xu et al . ( 2019 ) ) . Higher-Order GNNs . To increase the representational power of GNNs , several higher-order GNNs have been proposed . In a k−GNN , a message passing algorithm is applied to the k−tuples of vertices , in a similar fashion as GNNs do on vertices ( Morris et al. , 2019 ) . At initialization , each k−tuple is labeled with its type , that is , two k−tuples are labeled differently if their induced subgraphs are not isomorphic . As a result , k−GNNs can count ( induced ) substructures with at most k vertices even at initialization . Another class of higher-order networks are k−IGNs , which are constructed with linear invariant/equivariant feed-forward layers , whose inputs consider graphs via adjacency matrices ( Maron et al. , 2019 ) . k−IGNs are at least as powerful as k−GNNs , and hence they too can count substructures with at most k vertices . However , both methods need O ( nk ) operations . Specifically for counting substructures , Chen et al . ( 2020 ) propose Local Relational Pooling ( LRP ) networks . LRPs apply Relational Pooling ( RP ) networks ( Murphy et al. , 2019b ; a ) on the neighborhoods around each vertex . RP networks use permutation-variant functions and convert them to a permutation-invariant function by summing over all permutations . This summation is computationally expensive . 3 OTHER RELATED WORKS . Expressive power . Several other works have studied the expressive power of GNNs ( Azizian & Lelarge , 2020 ) . Scarselli et al . ( 2009 ) extend universal approximation from feedforward networks to GNNs , using the notion of unfolding equivalence . Chen et al . ( 2019 ) establish an equivalence between the graph isomorphism problem and the power to approximate permutation invariant functions on graphs . Maron et al . ( 2019 ) and Keriven & Peyré ( 2019 ) propose higher-order , tensor-based GNN models that provably achieve universal approximation of permutation-invariant functions on graphs , and Loukas ( 2019 ) studies expressive power under depth and width restrictions . Studying GNNs from the perspective of local algorithms , Sato et al . ( 2019 ) show that GNNs can approximate solutions to certain combinatorial optimization problems . Subgraphs and GNNs . The idea of considering local neighborhoods to have better representations than MPNNs is considered in several works ( Liu et al. , 2019 ; Monti et al. , 2018 ; Liu et al. , 2020 ; Yu et al. , 2020 ; Meng et al. , 2018 ; Cotta et al. , 2020 ; Alsentzer et al. , 2020 ; Huang & Zitnik , 2020 ) . For example , in link prediction , one can use local neighborhoods around links and apply GNNs , as suggested in ( Zhang & Chen , 2018 ) . A novel method based on combining GNNs and a clustering algorithm is proposed in ( Ying et al . ( 2018 ) ) . For graph comparison ( i.e. , testing whether a given possibly large subgraph exists in the given model ) , Ying et al . ( 2020 ) compare the outputs of GNNs for small subgraphs of the two graphs . To improve the expressive power of GNNs , Bouritsas et al . ( 2020 ) use features that are counts of specific subgraphs of interest . Another related work is ( Vignac et al. , 2020 ) , where an MPNN is strengthened by learning local context matrices around vertices . 4 RECURSIVE NEIGHBORHOOD POOLING . Next , we construct Recursive Neighborhood Pooling Graph Neural Networks ( RNP-GNNs ) , GNNs that can count any set of induced substructures of interest , with lower complexity than previous models . We represent each vertex by a representation of its radius r1-neighborhood , and then combine these representations . The key question is how to encode these local neighborhoods in a vector representation . To do so , we introduce a new idea : we view local neighborhoods as small subgraphs , and recursively apply our model to encode these neighborhood subgraphs . When encoding the local subgraphs , we use a different radius r2 , and , recursively , a sequence of radii ( r1 , r2 , . . . , rt ) ∈ Nt to obtain the final representation h ( t ) v of vertices after t recursion steps . While MPNNs also encode a representation of a local neighborhood of certain radius , the recursive representations differ as they essentially take into account intersections of neighborhoods . As a result , as we will see in Section 5.1 , they retain more structural information and are more expressive . Models such as k-GNN and LRP also compute encodings of subgraphs , and then update the resulting representations via message passing . We can do the same with the neighborhood representations computed by RNP-GNNs to encode more global information , although our representation results in Section 5.1 hold even without that . In Section 6 , we will compare the computational complexity of RNP-GNN and these other models . Formally , an RNP-GNN is a parametrized learnable function f ( . ; θ ) : Gn → Rd , where Gn is the set of all labeled graphs on n vertices . Let G = ( V , E , X ) be a labeled graph with |V| = n vertices , and let h ( 0 ) v = Xv be the initial representation of each vertex v. LetNr ( v ) denote the neighborhood of radius r of vertex v , and let G ( t−1 ) v ( Nr1 ( v ) \ { v } ) ) denote the induced subgraph of G on the set of vertices Nr1 ( v ) \ { v } , with augmented vertex label X ( t−1 ) u = ( h ( t−1 ) u ,1 [ ( u , v ) ∈ E ] ) for any u ∈ Nr1 ( v ) \ { v } . This means we add information about whether vertices are direct neighbors ( with distance one ) of v. Given a recursion sequence ( r1 , r2 , . . . , rt ) or radii , the representations are updated as m ( t ) v = RNP-GNN ( t−1 ) ( G ( t−1 ) v ( Nr1 ( v ) \ { v } ) ) , ( 4 ) h ( t ) v = COMBINE ( t ) ( h ( t−1 ) v , m ( t ) v ) , ( 5 ) for any v ∈ V , and hG = READOUT ( { { h ( t ) v : v ∈ V } } ) . ( 6 ) Different from MPNNs , the recursive update ( 4 ) is in general applied to a subgraph , and not a multi-set of vertex representations . RNP-GNN ( t−1 ) is an RNP-GNN with recursion parameters ( r2 , . . . , rt ) ∈ Nt−1 . The final READOUT is an injective , permutation-invariant learnable multi-set function . If t = 1 , then m ( t ) v = AGGREGATE ( t ) ( { { ( h ( t−1 ) u ,1 [ ( u , v ) ∈ E ] ) : u ∈ Nr1 ( v ) } } ) ( 7 ) is a permutation-invariant aggregation function as used in MPNNs , only over a potentially larger neighborhood . For r1 = 1 and t = 1 , RNP-GNN reduces to MPNN . In Figure 1 , we illustrate an RNP-GNN with recursion parameters ( 2 , 2 , 1 ) as an example . We also provide pseudocode for RNP-GNNs in Appendix C .

Summary: The paper has a fully theoretical flair while proposing a novel and seemingly efficient procedure to recursively compute the higher-order (i.e. more than 1-hop) neighbourhood of a node that are used for learning discriminative graph embedding. The paper contributes with the model above (RNP-GNN) and by providing a proof of its representational power and a general theorem supplying an information theoretic lower bound on the complexity of GNNs that can count induced substructures.