Breaking the Expressive Bottlenecks of Graph Neural Networks

1 INTRODUCTION . Graphs are ubiquitous in the real world . Social networks , traffic networks , knowledge graphs , and molecular structures are typical graph-structured data . Graph Neural Networks ( GNNs ) ( Scarselli et al. , 2008 ; Gori et al. , 2005 ) , leveraging the power of neural networks to graph-structured data , have a rapid development recently ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Bronstein et al. , 2017 ; Gilmer et al. , 2017 ; Duvenaud et al. , 2015 ) . Expressive power of GNNs measures their abilities to represent different graph structures ( Sato , 2020 ) . It decides the performance of GNNs where the awareness of graph structures is required , especially on large graphs with complex topologies . The neighborhood aggregation scheme ( or message passing ) follows the same pattern with weisfiler-lehman ( WL ) graph isomorphism test ( Weisfeiler & Leman , 1968 ) to encode graph structures , where node representations are computed iteratively by aggregating transformed representations of its neighbors with structural information learned implicitly . Therefore , the WL test is used to measure the expressiveness of GNNs . Unfortunately , general GNNs are at most as powerful as 1-order WL test ( Morris et al. , 2019 ; Xu et al. , 2019 ) . There is also work trying to improve the expressiveness that are beyond 1-order WL test ( Maron et al. , 2019 ; Morris et al. , 2019 ; Chen et al. , 2019 ; Li et al. , 2020b ; Vignac et al. , 2020 ) . However , the weak distinguishing strength of aggregators is the fundamental limitation . The expressiveness analysis measured by the WL test assumes that aggregators are injective , which is usually unreachable . Therefore , this motivates us to investigate the following questions : What are the key factors to limit the expressiveness of GNN ? and how to break these limitations ? Aggregators are permutation invariant functions that operate on sets while preserving permutation invariance . ( Zaheer et al. , 2017 ) first theoretically studied permutation invariant functions and provided a family of functions to which any permutation invariant function must belong . ( Xu et al. , 2019 ) extended it on multisets but only for countable space . ( Corso et al. , 2020 ) further extended it to uncountable space . ( Murphy et al. , 2018 ) and ( Murphy et al. , 2019 ) expressed a permutation invariant function by approximating an average over permutation-sensitive functions with tractability strategies . ( Dehmamy et al. , 2019 ) showed that a single propagation rule applied in general GNNs is rather restrictive in learning graph moments ( Lin & Skiena , 1995 ) . They and ( Corso et al. , 2020 ) improved the distinguishing strength of aggregation by leveraging multiple basic aggregators ( SUM , MEAN , NORMALIZED MEAN , MAX/MIN , and STD ) . This strategy showed its effectiveness on tasks taken from classical graph theory . In contrast to existing studies towards aggregators in GNNs , we provide a new GNN formulation , where the aggregation is represented as the multiplication of the corresponding hidden feature matrix of neighbors and the aggregation coefficient matrix . This new formulation enables us to answer the following questions : ( i ) when a GNN will lose its expressive power ; ( ii ) How to build aggregators with higher distinguishing strength , even injective aggregators . Based on our theoretical analysis , we propose two GNN layers : ExpandingConv and CombConv , and evaluate them on general graph classification and graph regression tasks . Our key contributions are summarized as follows : • We formalize the distinguishing strength of aggregators as a partial order , and theoretically show that the choice of aggregators can be bottlenecks of expressiveness . We also propose to apply nonlinear units ahead of aggregations to break the distinguishing strength limitations of aggregators as well as to achieve an implicit sampling mechanism . • We reformulate the neighborhood aggregation with the aggregation coefficient matrix and then provide a theoretical point of view on building powerful aggregators and even injective aggregators . • We propose ExpandingConv and CombConv layers which achieve state-of-the-art performance on a variety of graph tasks . We also show that multi-head GAT is one of the ExpandingConv implementations , which brings a theoretical explanation for its effectiveness . 2 PRELIMINARIES . 2.1 NOTATIONS . For a graph GpV , Eq , we denote the set of edges , nodes and node feature vectors respectively by EG , VG and XG . N pvq represents the set of neighbors of v including itself , i.e. , N pvq “ tu P VG|pu , vq P EGuY tvu . We use rns to denote the set t1 , 2 , ... , nu . tt ... uu represents a multi-set , i.e. , a set with possibly repeating elements . Πn represents the set of all permutations of the integers 1 to n. hπ , where π P Π|h| , is a reordering of the elements of a sequence h according to π . Given a matrix X P Raˆb , XT represents the transpose of X , and vecpXq P Rabˆ1 represents the column stack ofX . 2.2 GRAPH NEURAL NETWORKS . Most GNNs adopt the neighborhood aggregation scheme ( Gilmer et al. , 2017 ) to learn the node representations , which utilizes both node features and graph structures . In the k-th layer , the representation of node v hpkqv “ Updatephpk´1qv , Aggregateptthpk´1qu |u P N pvquuqq . ( 1 ) Aggregators in GNNs . An aggregator is a permutation invariant function ( Zaheer et al. , 2017 ) with bounded size inputs . It satisfies : ( i ) size insensitive : an aggregator can take an arbitrary but finite size of inputs ; ( ii ) permutation invariant : an aggregator is invariant to the permutation of input . There are a limited number of basic aggregators such as SUM , MEAN , NORMALIZED MEAN , MAX/MIN , STD , etc . Most proposed GNNs apply one of these aggregators . Sum-of-power mapping ( Zaheer et al. , 2017 ) and normalized moments ( Corso et al. , 2020 ) can also be used as aggregators and they allow for a variable number of aggregators . 3 PROPOSED MODEL . In this section , we first formalize the distinguishing strength of aggregators as a partial order , and show why basic aggregators used in popular GNNs become bottlenecks of expressiveness . Then , we analyze the requirements for building powerful aggregators and even injective aggregators . Finally , we introduce two GNN layers based on our theoretical analysis . 3.1 DISTINGUISHING STRENGTH OF AGGREGATORS . To ensure generality , our analysis of aggregators is always considered in multisets and uncountable case , where the inputs are continuous and with possibly repeating elements . We first introduce distinguishing strength under the concept of partial order ( Schmidt , 2011 ) . Distinguishing strength . The distinguishing strength of aggregator faggr1 is stronger than faggr2 , denoted by faggr1 ľ faggr2 , if and only if for any two multisets x1 and x2 where the number of elements can be different , faggr2px1q ‰ faggr2px2q ñ faggr1px1q ‰ faggr1px2q . Meanwhile , if there exist x11 and x 1 2 such that faggr1px11q ‰ faggr1px12q but faggr2px11q “ faggr2px12q , faggr1 is strictly stronger than faggr2 , denoted by faggr1 ą faggr2 . If faggr1 ľ faggr2 ľ faggr1 , we say these two aggregators have the same distinguishing strength , denoted by faggr1 “ faggr2 . If there exist multisets x1 and x2 such that faggr1px1q ‰ faggr1px2q , faggr2px1q “ faggr2px2q , and there also exist x11 and x 1 2 such that faggr1px11q “ faggr1px12q , faggr2px11q ‰ faggr2px12q , we say faggr1 and faggr2 are incomparable . Distinguishing strength is a partial order , and the set of all aggregators form a poset . In this poset , the aggregators with the greatest distinguishing strength should be injective . With the definition of distinguishing strength , we can compare any two aggregators . The distinguishing strength of widely used aggregators SUM , MEAN , MAX/MIN is incomparable . One can easily give two multisets of elements that are distinguished by one aggregator but are not distinguished by the others as showed in ( Corso et al. , 2020 ) . Equivariant aggregator . faggr : ttRduu Ñ Rd is an equivariant aggregator if and only if faggrpttT ¨ x1 , T ¨ x2 , ¨ ¨ ¨ , T ¨ xnuuq “ T ¨ faggrpttx1 , x2 , ¨ ¨ ¨ , xnuuq for any T P Rmˆd and ttxi P Rd|i P rnsuu . Widely used SUM and MEAN are equivariant aggregators but MAX/MIN is not . We denote faggr1 b faggr2 a new aggregator by combing faggr1 and faggr2 with faggr1 b faggr2pXq “ rfaggr1pXq||faggr2pXqs , where || denotes concatenation . Lemma 1 . ( i ) For any continuous function g , we have g ˝ faggr ĺ faggr , and when g is injective , faggr “ g ˝ faggr ; ( ii ) faggr1 b faggr2 ľ faggr1 and faggr1 b faggr2 ľ faggr2 . If faggr1 and faggr2 are incomparable , faggr1 b faggr2 ą faggr1 and faggr1 b faggr2 ą faggr2 ; ( iii ) If faggr is an equivariant aggregator , then faggrpT ¨ x1 , T ¨ x2 , ¨ ¨ ¨ , T ¨ xnq ĺ faggrpx1 , x2 , ¨ ¨ ¨ , xnq for any T P Rmˆd and ttxi P Rd|i P rnsuu . We prove Lemma 1 in Appendix B. Lemma 1 indicates that aggregators become bottlenecks of distinguishing strength . For the equivariant aggregator , any linear transformation before aggregation and any transformation after aggregation have no contribution to the distinguishing strength . For SUM and MEAN , we have gpSUMpT ¨ x1 , T ¨ x2 , ¨ ¨ ¨ , T ¨ xnqq ĺ SUMpx1 , x2 , ¨ ¨ ¨ , xnq and gpMEANpT ¨ x1 , T ¨ x2 , ¨ ¨ ¨ , T ¨ xnqq ĺ MEANpx1 , x2 , ¨ ¨ ¨ , xnq , where T P Rmˆd , and g can be any continuous function . Based on Lemma 1 , we can now compare the distinguishing strength of aggregations in some popular GNNs . GIN-0 sums all hidden features of neighbors at first , and then pass them to a 2-layer MLP . Therefore , when considering in a continuous input features space , the distinguishing strength of GIN-0 is at most as powerful as the SUM aggregator . GCN uses a NORMALIZED MEAN ( denoted by nMEAN ) aggregator . Given a node v and its neighbors , nMEANpv , u1 , ¨ ¨ ¨ , unq “ 1 ? |N pvq| ¨p hv ? |N pvq| ` hu1 ? |N pu1q| ` ¨ ¨ ¨ ` hun ? |N punq|´1 q. nMEAN is also an equivariant aggregator , and the distinguishing strength of aggregation in GCN is at most as powerful as nMEAN . GAT corresponds to the weighted SUM aggregation , where the weight coefficients are the functions of hidden features . This makes the distinguishing strength of GAT and SUM incomparable . Based on these observations , a potential approach to breaking the distinguishing strength limitation is to apply a nonlinear processing on inputs before aggregation . 3.2 BUILDING POWERFUL AGGREGATORS . In this section , we analyze the requirements for building more powerful aggregators and further injective aggregators . We first introduce a new representation of GNN layers which unifies several popular GNN layers . Given a node v and its neighbors N pvq , our new formulation represents the GNN operation as follows : $ ’ & ’ % mv “ flocalpvq / * aggregation coefficients generation * / r ptq v “ mvπh̄pt´1qTvπ / * neighborhood aggregation * / h ptq v “ fNNprptqv q / * feature/structure extraction * / . ( 2 ) Here , mv P R|N pvq| is the aggregation coefficients vector of node v. Note that mv should be the mapping of local structures such as node degrees , node or edge features of the k-hop neighbors assigned on node v to ensure the same encoding of isomorphic graphs . h̄pt´1qTvπ “ phpt´1qv , hpt´1qu1 , ¨ ¨ ¨ , h pt´1q u|N pvq|q T P R|N pvq|ˆd is the matrix representation of v ’ s neighbors according to a permutation π. fNN : Rd Ñ Rd 1 is a neural network that extracts task-relevant information from the aggregated representation rptqv , and is used to update hidden feature h ptq v of node v. According to Equation 2 , the aggregation should be with high distinguishing strength to avoid indistinguishability among neighbors . Meanwhile , the extraction should be powerful enough to efficiently extract task-relevant structural patterns from the aggregated representation of neighbors . Based on these observations , we reformulate GCN , GIN0 and GAT with their corresponding threestage representations as follows : GCN : $ ’ ’ & ’ ’ % mv “ 1 ? |N pvq| p 1 ? |N pvq| , 1 ? |N pu1q| , ¨ ¨ ¨ , 1 ? |N pu|Npvq|´1q| q r ptq v “ mvπh̄pt´1qTvπ h ptq v “ σpWrptqv ` bq , GIN0 : $ ’ & ’ % mv “ 11ˆ|N pvq| r ptq v “ mvπh̄pt´1qTvπ h ptq v “ MLPprptqv q , GAT : $ ’ & ’ % m ptq v “ pattphpt´1qv , hpt´1qv q , attphpt´1qv , hpt´1qu1 q , ¨ ¨ ¨ , attph pt´1q v , h pt´1q u|Npvq´1|qq r ptq v “ mptqvπh̄pt´1qTvπ h ptq v “ σpWrptqv ` bq . Their default formulations are given in Appendix A . In the aggregation step , GCN ’ s mv is the mapping of neighbors ’ degrees ; GIN0 ’ s mv is the mapping of node v ’ s degree which is equivalent to SUM aggregator ; GAT ’ smv is the mapping of neighbors ’ features . All of them are the mappings of local structures as given in Equation 2 . In this three-stage representation , the aggregation is reformulated as the multiplication of the aggregation coefficients vector and the feature matrix of neighbors . It provides insights on improving the distinguishing strength of aggregations . First , we show how to characterize the permutation invariance in this formulation . Let M P Rsˆn denote an aggregation coefficient matrix where s ě 1 . Note that in GCN , GIN and GAT , s is restricted to be 1. hπ P Rnˆd is the matrix representation of n input elements according to π . The aggregation computation in the second step of Equation 2 is faggrpM , hπq “ MPπhπ “ Mπhπ , where Pπ is the permutation matrix according to π. Pπhπ ensures the same output for all hπ , π P Π|h| . Mπ “ MPπ is the reordering of columns of M according to π . For any π1 , π2 P Πn , faggrpM , hπ1q “ faggrpM , hπ2q , thus permutation invariance holds . Once M is decided , we obtain a unique aggregator denoted by fM . For any sequence of input elements h , fM phq “ faggrpM , hπq , where π P Πn can be any ordering of neighbors . Next , we analyze the distinguishing strength of fM . Proposition 1 . For any two matrices M P Rsˆn and M 1 P Rs1ˆn with s , s1 ď n , we have ( i ) f ` M M 1 ˘ ľ fM , where ` M M 1 ˘ means stacking these two matrices ; ( ii ) f ` M M 1 ˘ ą fM if and only if rankp ` M M 1 ˘ q ą rankpMq ; ( iii ) Any multiset of size n is distinguishable with fM if and only if rankpMq “ n. We prove Proposition 1 in Appendix C. Proposition 1 shows that the distinguishing strength of fM is decided by the rank of the correspondingM . Yet , the distinguishing strength analysis in Proposition 1 is only suitable for multisets aggregated with shared fM . Next , we extend the analysis for the case of different aggregators . Let RespfM q denote the set of all outputs of fM . Our proposed three-stage representation also provides useful insight on the constraints among different aggregators . That is , in order to fully distinguish different local structures , for any two different fM1 and fM2 , RespfM1qXRespfM2q “ H. This is because to fully distinguish different local structures , we should ensure their aggregated representations are different . SinceM is restricted to be the mapping of local structures ( such as khop neighbors ) , different M means that the corresponding local structures are different . Therefore , the aggregation results of different fM must be different . However , it is not satisfied by existing GNNs , and there are few studies on distinguishing multisets aggregated by different aggregators . In Proposition 2 , we present a detailed analysis of it . Proposition 2 . For any M1 , M2 P Rsˆn1 and M 11 , M 12 P Rs 1ˆn2 , ( i ) Respf ` M1 M 11 ˘q X Respf ` M2 M 12 ˘q Ď RespfM1q X RespfM2q ; ( ii ) If Respf ` M1 M 11 ˘q X Respf ` M2 M 12 ˘q Ă RespfM1q X RespfM2q , then rankp ` M1 M2 M 11 M 1 2 ˘ q ą rankp ` M1 M2 ˘ q ; We prove Proposition 2 in Appendix D. Proposition 2 shows the necessity of preserving the rank of aggregation coefficient matrix when considering the distinguishing strength among different aggregators . Next , we provide a sufficient condition for building desired multiple injective aggregators with the outputs having no intersections . Proposition 3 . For any two aggregators fM1 and fM2 with M1 P Rsˆn1 and M2 P Rsˆn2 , if rankp ` M1 M2 ˘ q “ n1 ` n2 , then fM1 and fM2 are injective and RespfM1q X RespfM2q “ H. We prove Proposition 3 in Appendix E. Proposition 1 , 2 and 3 provide a new perspective for building powerful aggregators and even injective aggregators . Compared with the distinguishing strength studies in ( Xu et al. , 2019 ) and ( Corso et al. , 2020 ) , as well as existing strategies for building injective aggregators , e.g. , sum-of-power mapping ( Zaheer et al. , 2017 ) and normalized moments ( Corso et al. , 2020 ) , we reformulate the aggregation with aggregation coefficients matrix and show the relations of the distinguishing strength of aggregators and the rank of the corresponding aggregation coefficients matrices . Besides , the aggregation of this method is controlled by aggregation coefficients which can be learned from graph data to better leverage structural information . In this paper , to simplify the analysis , we only consider the aggregations within one-hop neighbors . The results can be easily extended to more sophisticated aggregators with the overall framework unchanged In the perspective of preserving the rank of hidden features among neighbors , r “ Mh indicates that rankprq ď minprankpMq , rankphqq . To preserve the rank of hidden features in aggregations such that rankprq “ rankphq , we need rankpMq ě rankphq . This builds a connection between improving the distinguishing strength of aggregators and preserving the rank of hidden features among neighbors , both of which have the requirements on the rank ofM . General aggregators such as ones in GCN , GIN-0 and GAT have rankpMq “ 1 . Thus , rankprq is always fixed to 1 no matter what the rank of the input features is . Correspondingly , they have a weak distinguishing strength . Equation 2 splits the aggregation and feature/structure extraction into two independent steps , which helps to figure out that the expressive power loss happens in the aggregation step , and then the model extracts feature/structure information on the distorted encodings of neighbors . From Equation 2 , the aggregation can be considered as a representation regularization step , which unifies different multisets of neighbors into the same representation style while holding permutation invariance . Then , the model can extract structural information on this regulated data with a shared trainable matrix as the third step in Equation 2 . Based on this observation , we propose two novel GNN layers : ExpandingConv and CombConv .

Summary: The authors propose two new layers for GNNs. CombConv and ExpandingConv are motivated by the insight that a GNN is only as expressive as the rank of the matrix that represents the coefficients of the aggregation function. To arrive at this statement, the authors formalize all GNNs as being composed of three steps: 1) generation of aggregation coefficients, 2) actual aggregation of the neighbourhood, and 3) feature extraction from the aggregation. Furthermore, it is shown that current approaches have very low distinguishing strength and that CombConv and ExpandingConv, by their construction, yield higher expressive power.