Diverse Message Passing for Attribute with Heterophily

1 Introduction . Originated from the spectral graph theory , Graph Neural Networks ( GNNs ) [ 1 , 2 ] have demonstrated superior performances on modeling ubiquitous irregular data in many tasks , such as computer vision , natural language processing , information retrieval , etc . By reducing the computational complexity of graph Fourier transformation via approximation , most of the GNNs , especially Graph Convolutional Networks ( GCN ) [ 3 ] and its variants ( such as SGC [ 4 ] ) , actually perform Laplacian smoothing [ 5 ] and low-passing filtering [ 4 ] . Although this intuitive strategy achieves state-of-the-art performances on networks with homophily , such as Cora , CiteSeer and Pubmed , it possesses two fatal drawbacks . Firstly , it only performs well on shallow networks , while induces serious over-smoothing issue when more layers are stacked [ 5 , 6 , 7 ] . Secondly , it shows obvious ineffectiveness in handling the networks ⇤Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . with heterophily , where its performance is even worse than the simple multiple layer perceptron ( MLP ) [ 8 ] . There are many attempts to overcome the above two drawbacks . The common philosophy behind these attempts is weakening the smoothing effect . One type of methods directly modifies the smoothed node representations based on local information . For example , APPNP [ 9 ] and GCNII [ 10 ] average the smoothed node representations with the corresponding original node attributes . JKNet [ 7 ] and PPNP [ 9 ] combine multi-scale information by summation , while MixHop [ 11 ] , truncated block Krylov network [ 12 ] and H2GCN [ 8 ] combine multi-scale information by concatenation . Other methods modifies the network topology , which indirectly reduces the over-smoothing issue . For instance , DropEdge [ 13 ] and GRAND [ 14 ] randomly remove edges and nodes , respectively , while FAGCN [ 15 ] and GPRGNN [ 16 ] relax the edges with positive values , which tends to induce the smoothing effect , to possess real values ( which can be either positive or negative ) . A recent research [ 17 ] has tried to bridge and unify over-smoothing issue and networks with heterophily . Most of the GNNs , which can be designed for networks with either homophily or heterophily , can be unified by a message passing framework . These GNNs usually consider all the attributes of one node in its entirety , and propagate them via the same ( positive or negative ) propagation weight along each edge . Intuitively , most of the Graph Neural Networks ( GNNs ) are restricted in the uniform message passing framework and focus on uniform weight learning , such as GCN [ 3 ] , GAT [ 18 ] and FAGCN [ 15 ] . This phenomenon is induced by simplifying the network homophily/heterophily to node-level property [ 19 ] . In Geom-GCN [ 20 ] and H2GCN [ 8 ] , which are designed to handle the networks with heterophily , the network homophily rate is defined by only utilizing network topology and node labels . Actually , different attributes possess diverse characteristics . To explicitly model the diversity , the concept of network homophily rate defined in [ 20 ] is extended to attribute level . Specifically , attribute homophily rate for each attribute is defined by replacing the real label of each node with the corresponding attributes to form weak labels . Statistics in Figure 1 reveal that the attribute homophily rates of the networks with either homophily and heterophily all tend to vary in a large range . Then , GNNs are desired to explicitly consider the diversity of attribute homophily rates . To achieve this objective , we propose a Diverse Message Passing ( DMP ) framework , which specifies every attribute propagation weight on each edge , to adaptively handle the diverse attribute homophily . Besides , we further propose two strategies to significantly reduce the model complexity of DMP to alleviate the overfitting issue . By investigating the spectral characteristics , existing spectral GNNs are actually equivalent to a degenerated version of our DMP . From the perspective of numerical optimization , we provide a theoretical analysis to demonstrate DMP ’ s powerful representation ability and the ability of alleviating the over-smoothing issue . Specifically , different from Uniform Message Passing , which directly generates a graph partition , our DMP generates multiple groups of graph partition candidates , and then the classifier in semi-supervised task will determine the specific combining approach to form the final partition . This mechanism actually enables our DMP to possess more representation ability , which can also alleviate the over-smoothing issue , compared to classic Uniform Message Passing . 2 Notations and Preliminaries . Let G = ( V , E ) denote a graph with a set of nodes V = { v1 , v2 , · · · , vN } and a set of edges E , where N is the number of nodes . The topology of graph G can be represented by its adjacency matrix A = [ aij ] 2 { 0 , 1 } N⇥N , where aij = 1 if and only if there exists an edge eij = ( vi , vj ) between nodes vi and vj . The degree matrix D is a diagonal matrix with each diagonal element di = PN i=1 aij being the degree of node vi . N ( vi ) = { vj | ( vi , vj ) 2 E } stands for the neighbourhoods of node vi . X 2 R N⇥F and H 2 RN⇥F 0 respectively denote the collection of node attributes and representation of the ith row , i.e. , xi 2 RF and hi 2 RF 0 , which both correspond to node vi . Note that F and F 0 stand for the dimensions of node attributes and representations . Most of the Graph Neural Networks ( GNNs ) follow an aggregation-combination strategy [ 21 ] , where each node representation is iteratively updated by aggregating node representations in the local neighbourhoods and combining the aggregated representations with the node representation itself as h̄ k v = AGGREGATE k h k 1 u |u 2 N ( v ) , hkv = COMBINATE k h k 1 v , h̄ k v , ( 1 ) where h̄kv stands for the aggregated representation from local neighbourhoods . Besides of the concatenation based implementation , such as GraphSAGE [ 22 ] and H2GCN [ 8 ] , averaging ( or summation ) has been widely adopted to implement COMBINATEk ( · , · ) , such as GCN [ 3 ] , GAT [ 18 ] , GIN [ 23 ] , etc . Except for the MAX and LSTM implementations in GraphSAGE [ 22 ] , most of the GNNs utilize averaging function to implement AGGREGATEk . Therefore , they can be unified as h k v = ✓ ckvvh k 1 v + X u2N ( v ) ckuvh k 1 u ◆ W k ! , ( 2 ) where Wk represents the learnable parameters and ( · ) denotes the nonlinear mapping function . Note that the scalar cuv is the averaging weight . For example , GCN [ 3 ] sets ckuv = 1/ ( p ( du + 1 ) ( dv + 1 ) , GIN [ 23 ] sets ckuv = 1 for u 6= v and ckvv = 1 + ✏k , and GAT [ 18 ] learns non-negative ckuv based on the attention mechanism . Recently , to handle the network with heterophily via high-passing filtering , GPRGNN [ 16 ] sets ckuv = k/ ( p ( du + 1 ) ( dv + 1 ) with k being a learnable real value , while FAGCN [ 15 ] directly relaxes the learnable cuv in GAT to real value . 3 Diverse Message Passing . In this section , the motivation of Diverse Message Passing is firstly provided . Then , the diverse aggregation framework is given , followed by two implementations to reduce model complexity . 3.1 Motivations . The unified formulation in Eq . ( 2 ) can be regarded as the uniform message passing framework , where all the attributes which are propagated from node v to node u , i.e. , all the elements of hk 1u , share the common weight ckuv . Then , all the attributes can be aggregated with the same weight assignment scheme in the local neighbourhood N ( v ) [ { v } . Since aggregation can be interpreted from the perspectives of both Laplacian smoothing [ 5 ] and low-passing filtering [ 4 ] , the same weight assignment scheme indicates that all the attributes tend to share similar distributions over the entire graph . To investigate the distributions of attributes and measure their similarities , the attribute homophily rate is proposed . The network homophily rates , including both the node version [ 20 ] and edge version [ 8 ] , quantify similarity between connected nodes . The node similarity in network homophily rate is measured according to the node labels . Here , the attribute homophily rate is defined by measuring the node similarity according to each attribute . Specifically , the attribute homophily rate of attribute f is formulated as f = 1P v2V xvf X v2V vf = 1P v2V xvf X v2V xvf P u2N ( v ) xuf dv ! , ( 3 ) where P u2N ( v ) xuf dv is the rate of neighbourhoods with attribute f , vf = xvf P u2N ( v ) xuf dv is the rate of neighbourhoods which share the same attribute f as node v , and the attribute homophily rate of attribute f , i.e. , f , is the average of vf over all the nodes possessing attribute f . Figure 1 shows the distribution of { f } Ff=1 on networks with different homophily , computed via the method in [ 20 ] . For better visualization , both the macroscopic and microscopic perspectives are provided . Figure 1 ( a ) gives the distribution of attribute homophily ( in macroscopic perspective ) on each network ( each column ) . Figure 1 ( b ) describes the attribute homophily of different attributes , i.e. , { f } Ff=1 from the microscopic perspective . It can be observed that networks with higher homophily also possess higher averaged attribute homophily , and vice versa . However , attribute homophily { f } Ff=1 distributes in a large range , with little correlations to the degree of attribute homophily of network . It shows that different attributes possess remarkably different attribute homophily , which reveals that the distributions of attributes over the entire graph are diverse , instead of uniform ones . Therefore , diverse aggregation scheme , i.e. , averaging scheme , should be designed to fully explore the diverse distributions of attributes over the entire graph for effective node representation learning . 3.2 Diverse Aggregation Framework . To achieve diverse aggregation , the Uniform Message Passing in Eq . ( 2 ) is revised to make different attributes possess different propagation weights . Then , the scaler ckvu in Eq . ( 2 ) is augmented to vector ckvu , which has the same length as the node representation hk 1u . Each element in ckvu stands for the weight of the corresponding attribute in hk 1u propagated from node u to node v. Thus , Eq . ( 2 ) is modified to h k v = ✓ c k vv h k 1 v + X u2N ( v ) c k uv h k 1 u ◆ W k ! , ( 4 ) where denotes the element-wise product of vectors . Then , the remaining issue is the learning strategy of ckuv ’ s . If all ckuv ’ s are set as free parameters to be directly learned , the model complexity , i.e. , the number of parameters , of directly learning ckuv ’ s is O ( |E|F ) , which may induce overfitting . In the following paragraphs , two strategies are designed to reduce the number of learnable parameters . The first strategy constrains all the parameters ckuv ’ s to be determined by a learnable function . Since c k uv can be regarded as the averaging weight between nodes v and u , the attention scheme can be adopted as c k uv = tanh [ h k 1 v ||h k 1 u ] W k c , ( 5 ) where [ hk 1v ||hk 1u ] represents the concatenation of representations hk 1v and hk 1u . Note that the learnable mapping function Wkc 2 RF⇥F is shared across all the edges , where F is the dimension of h k 1 v and ckuv . Then , the model complexity is significantly reduced to O ( F 2 ) . Different from the attention mechanism employed in GAT [ 18 ] , which constrains the learned propagation weights ckuv ’ s to be non-negative via a softmax nonlinear function , the learned weight vectors ckuv ’ s are relaxed to real values by adopting a tanh nonlinear function tanh ( x ) = exp ( x ) exp ( x ) exp ( x ) +exp ( x ) = sigmoid ( 2x ) 1 , whose output is zero-centered and ranged in ( 1 , 1 ) . Intuitively , the positive weights are equivalent to low-passing filtering , while the negative weights actually facilitate the filtering beyond low-frequency . The second strategy simplifies Eq . ( 4 ) by assuming that all the neighbourhoods employs identical propagation vector , i.e. , ckuv = c̄kv for every u 2 N ( v ) . This simplification , which has also been adopted in GraphSAGE [ 22 ] , H2GCN [ 8 ] and GIN [ 23 ] , assumes that the neighbourhoods of fixed-hop equally contribute to the node representations , Accordingly , Eq . ( 4 ) can be simplified to h k v = ✓ c k v h k 1 v + c̄ k v X u2N ( v ) h k 1 u ◆ W k ! . ( 6 ) Eq . ( 6 ) can be considered as the weighted averaging of the node representation hk 1v and the neighbourhood representation h̄k 1v = P u2N ( v ) h k 1 u , where both ckv and c̄kv can be regarded as the element-wise averaging weights . Thus , the learning of ckv and c̄kv can be determined by node and neighbourhood representations . Similar to Eq . ( 5 ) , the attention mechanism is adopted to learn them as c k v = tanh [ h k 1 v ||h̄ k 1 v ] W k c . ( 7 ) Compared to Eq . ( 5 ) , the learning process of c̄kv is similar , yet with a different parameter Wkc̄ . This strategy possesses identical model complexity to the first strategy .

Summary: This paper extends the node-level network homophily based on the node label to attribute-level one by considering each attribute as a weak label. Then, it observes that different attributes have diverse attribute homophily rates. To alleviate the issue of existing uniform message passing on capturing this diverse attribute homophily rates, this paper proposes diverse message passing, where different attributes are assigned different propagation weights, and two implementations. Finally, it theoretically investigates the ability of diverse message passing on overcoming over-smoothing by analyzing the roles of the uniform and diverse propagation weights.