Universal Graph Convolutional Networks

1 Introduction . Real-world complex systems can often be viewed as networks , such as social networks , biological networks and citation networks . Recently , research of analyzing networks with deep learning has received widespread attention both in academia and industry . In particular , Graph Convolutional Networks ( GCNs ) [ 14 ] , which obtain the meaningful representation of nodes in the network by integrating the neighborhood information , have achieved great success and been widely applied in tackling network analytics tasks , such as node classification [ 23 , 28 ] , link prediction [ 33 ] and recommendation [ 30 , 17 ] . While the success of GCNs and their variants [ 1 , 6 ] , a key weakness is the homophily assumption of networks , which restricts their performance on general network data . To be specific , most GCNs †Equal contributions . * Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . seem to be tailor-made to work on homophily networks [ 18 ] , where nodes within the same class tend to connect with each other . In fact , heterophily [ 20 ] networks , where nodes of different classes tend to link together , are also widespread in real-world . For example , different types of amino acids are more likely to be connected in protein structure [ 35 ] , and fraudsters tend to connect to accomplices than to other fraudsters in transaction networks [ 21 ] . Furthermore , the random networks also often exist in real-world , such as railway networks , where the edges between nodes are more likely to be randomly generated . Most popular GCNs typically obtain node embeddings of these networks using 1-hop network neighbors as neighborhoods for information propagation [ 12 , 29 ] . However , considering the different structural properties of networks ( e.g. , homophily or heterophily ) , whether different networks should adopt a unified or different propagation mechanisms ? This is a very important question for GCNs since they mainly gain better performance through the propagation of information . A well informed answer can help us better understand the essence of GCNs , such as how different types of nodes affect the propagation , and what type of nodes are really required to achieve a certain level of predictive accuracy aiming to different networks . Several recent works have studied the networks with different structural properties . For example , Pei et al . [ 22 ] consider the heterophlily property of networks , and propose a geometric aggregation scheme to overcome neighborhood structural information losing and long-range dependencies lacking . Zhu et al . [ 35 ] design an effective model which improves the representation power of GCNs under heterophily through theoretical and empirical analysis . Chien et al . [ 4 ] introduce a new generalized pageRank ( GPR ) architecture to jointly optimize node feature and topological information extraction . Bo et al . [ 2 ] assess the roles of low-frequency and high-frequency signals , and propose an efficient method that can adaptively integrate different signals in the process of message passing . However , there is still a lack of insightful understanding from the perspective of propagation mechanism . As the first contribution of this study , we conduct experiments analysing the propagation mechanism of GCNs in networks with different structural properties . Surprisingly , our experiments clearly illustrate that for networks with complete homophily , complete heterophily and randomness , 1-hop , 2-hop and k-nearest neighbor ( kNN ) neighbors are more suitable as neighborhoods for information propagation , respectively . This means that the depicting ability of the current propagation mechanism of GCNs is limited , and networks with different structural properties may need to adopt different propagation mechanisms . In fact , while these segmentation rules seem to be able to select appropriate nodes as neighborhoods in an ideal way , the real-world networks are complex , and may present diverse properties , e.g. , the network dominated by homophily may contain a small amount of randomness or heterophily . A natural question is , “ Can we reasonably utilize these segmentation rules to design a universal propagation mechanism independent of the network structural assumption ? '' To tackle this challenge , we propose a novel and universal GCN model , i.e. , U-GCN , for general network data . The central idea is that we learn node embeddings by making full use of the information from 1-hop , 2-hop and kNN neighbors , and fuse them adaptively to derive deeper correlation information for the given learning objectives . To be specific , we first introduce a multi-type convolution mechanism . It uses 1-hop network ( i.e. , original input network ) , 2-hop network and kNN network that constructed by 1-hop , 2-hop and kNN neighbors for direct information propagation separately , and utilizes a node-level attention mechanism for each network , to extract three specific embeddings . We then make a discriminative aggregation to learn out the importance of these three embeddings , thereby extracting the most correlated information aiming to the ground truth such as node classification . Extensive experiments on a series of benchmark datasets demonstrate the superiority of U-GCN over some state-of-the-arts . 2 Notations and Preliminaries . Let G = ( A , X ) be an undirected attributed network , where A ∈ Rn×n represents the symmetric adjacency matrix with n nodes , and X ∈ Rn×p is the attribute ( content ) matrix of p attributes per node . Concretely , aij = 1 denotes there is an edge between nodes vi and vj , or 0 otherwise ; and xi represents the attribute vectors of node vi . Given an attribute network G , and a labeled node set VL containing u |V | nodes , where each node vi ∈ VL contains a unique class label yi ∈ Y . The goal of semi-supervised node classification is to infer the labels of nodes in V \VL by learning a classification function F . Homophily . In this work , the level of homophily ratio of edges [ 35 ] is used to define networks with strong homophily/heterophily . Specifically , the level of homophily ratio of edges is the fraction of edges in a network which connect nodes that have the same class label ( i.e. , intra-class edges ) , described by : α = | ( vi , vj ) : aij = 1 ∧ yi = yj | |m| , ( 1 ) where m is the number of edges . Networks with α closer to 1 tend to have more edges connecting nodes within the same class , or stronger homophily ; whereas networks with α closer to 0 have more edges connecting nodes in different classes , or stronger heterophily . Graph Convolutional Network . Graph Convolutional Network ( GCN ) [ 14 ] is a variant of multilayer convolutional neural networks that operates directly on networks . It learns embedding of each node by iteratively aggregating the information from its neighbors . Mathematically , let H ( l ) be the feature representation of the l-th layer , and H ( 0 ) be the node attribute matrix , the forward propagation can be defined as : H ( l ) = σ ( D̃− 1 2 ÃD̃− 1 2H ( l−1 ) W ( l ) ) , ( 2 ) where Ã = A + I stands for the adjacency matrix with self-loops , D̃ the node degree matrix of Ã , i.e. , D̃ii = ∑ j Ãij , W ( l ) a trainable weight matrix and σ the non-linear activation function . While GCN works well on several network analysis tasks such as node classification [ 10 , 15 ] , it still has a fundamental problem , that is , homophily assumption of networks , which leads to the main contribution in this work , i.e. , analyse what type of nodes are more suitable as neighborhoods for direct information propagation independent of the network structural assumption . 3 Motivating Observations . Here , we present a simple yet intuitive case study to illustrate and analyze the performance of GCN changes with different propagation mechanisms . The main idea is that we will apply GCN to networks with different structural properties utilizing three types of nodes : 1-hop , 2-hop and k-nearest neighbor ( kNN ) neighbors , which are often believed to be the effective neighborhoods for node classification in networks [ 28 , 35 ] , to realize the information propagation , respectively . Then , we will check the performance of GCN on these cases . A universal propagation mechanism should provide a good result in general network data . However , if the performance drops sharply in comparison with the other two situations , this will demonstrate that networks with different structural properties may need to use different propagation mechanisms . Setup . We conduct experiments on the Newman artificial networks [ 7 ] with different properties . The network consists 128 nodes divided into 4 classes , where each node has on average zin edges ( i.e. , intra-class edges ) connecting to nodes of the same class and zout edges ( i.e. , inter-class edges ) to nodes of other classes , and zin + zout = 16 . Note that here we utilize two indicators : ρin = zin/32 and ρout = zout/96 , to determine the network property , i.e. , ρin > ρout , ρin = ρout and ρin < ρout means the network with homophily , randomness and heterophily , respectively . For node attributes , we generate a 4h-dimensional binary attributes ( i.e. , xi ) for each node to form 4 attribute clusters , corresponding to the 4 classes [ 9 ] . To be specific , for every node in the m-th class , we use a binomial distribution with mean pin = hin/h to generate a h-dimensional binary vector as its ( ( m − 1 ) × h + 1 ) -th to ( m × h ) -th attributes , and generated the rest attributes using a binomial distribution with mean pout = hout/ ( 3h ) . In our experiments , we set 4h = 200 and hout = 4 ( hin+hout = 16 ) , so that pin > pout , the h-dimensional attributes are associated with the m-th class with a higher probability , whereas the rest 3h attributes are irrelevant . As shown in Figure 1 , for networks with strong homophily ( e.g. , ρout = 0.075 ) , it is easy to obtain high accuracy using 1-hop network neighbors . However , as the inter-class edges increase , the accuracy is rapidly reduced . This mainly due to the homophily assumption , preventing GCN from effectively fusing information . On the other hand , for networks with strong heterophily ( e.g. , ρout = 0.165 ) , it is surprising that , the accuracy of GCN of using 2-hop neighbors as neighborhoods ( i.e. , 83.15 % ) is much higher than that of using 1-hop network neighbors ( i.e. , 32.85 % ) . Since the homophily ratio of 2-hop neighbors may rise with the increase of inter-class edges , GCN of using 2-hop neighbors is more effective to some extent . Interestingly , we can find that GCN of utilizing kNN is easy to get the staple accuracy , i.e. , 71.46 % . In particular , it is much higher than those of using 1-hop and 2-hop neighbors on complete random network ( i.e. , ρout = 0.125 ) . Summary . This case study shows that the current propagation mechanism of GCN is not universal for general network data , but we can find that there are rules in several special situations ( i.e. , complete random network ) . This motivates us that networks with different structural properties may need adopt different propagation mechanisms . We conduct extra experiments on Newman networks [ 7 ] with complete homophily , randomness and complete heterophily utilizing GCN , so as to discover more appropriate propagation mechanism to select valuable nodes as neighborhoods , and thus improving the performance of GCNs for different networks . One straightforward strategy is to learn network embeddings with different GCNs using different types of nodes ( i.e. , 1-hop , 2-hop and kNN neighbors ) , and concatenate the embeddings into a single vector , so as to use the discriminative aggregation mechanism ( which will be introduced in Section 4.2 below ) to learn their importance for node classification . We show the attention values as a function of the number of training iterations in Figure 2 . Observation 1 : Network with complete homophily tends to obtain better performance utilizing 1-hop network neighbors for direct information propagation . For the network in Figure 2A , the edges exist only in nodes within the same class , or complete homophily . As shown , 1-hop network neighbors show great importance as the increase of training iterations . This partly validates that under the setting of complete homophily , 1-hop neighbors are more effective for direct information propagation compared to 2-hop and kNN neighbors . Observation 2 : Network with randomness tends to get better performance utilizing kNN for direct information propagation . The network in Figure 2B exhibits more randomness than Figure 2A , that is , complete random network . Obviously , with the increase of training iterations , the attention value of kNN is much higher than those of 1-hop and 2-hop neighbors . Since kNN typically constructed according to the similarity of node attributes , it can still realize the information fusion effectively , compared with the other two types of neighbors , in case that the network topology contains noise . Observation 3 : Network with complete heterophily tends to obtain better performance utilizing 2-hop neighbors for direct information propagation . For the network in Figure 2C , the edges exist only in nodes within different classes , or complete heterophily . While the learned attention values of these three types of neighbors differ slightly , the importance of 2-hop neighbors is relatively higher . This is mainly due to the fact that the homophily ratio of 2-hop neighbors becomes higher with the increase of inter-class edges ( which is often the real life in many network analysis tasks ) . While networks with different structural properties provide better performance utilize different propagation mechanisms , the real-world networks are complex , and may show diverse properties , e.g. , the network dominated by homophily may contain a small amount of heterophily . Therefore , it is imperative to explore a universal propagation mechanism for GCNs independent of the network structural assumption . Theorem 1 . The real-world networks can be approximately decomposed into a mixture of three kinds of simple networks , namely complete homophily , complete random and complete heterophily , in different proportions . Proof . Given a network G , where n and m denote the number of nodes and edges , respectively . Assuming that r edges ( 0 ≤ r ≤ m ) are generated randomly , i.e. , the probability of nodes connecting nodes within the same class or different classes is the same , which form a complete random network ( with n nodes and r edges ) . The remaining edges connecting two nodes within the same class can then be regarded as composing a complete homophily network , or a complete heterophily network . Summary . Now , we can conclude that a universal GCN model may not only consider the 1-hop ( Observation 1 ) , but also the 2-hop ( Observation 2 ) and kNN neighbors ( Observation 3 ) for direct information propagation . More importantly , considering different network properties can be more correlated with one of them or even their combinations , the model itself should adaptively learn their corresponding importance , so as to achieve feature fusion more effectively . This case study , although leveraging specific artificial networks , is representative because real-world networks can often be considered as the combination of these three simple network cases .

Summary: The paper proposes Universal GCN, an architecture built on top of GCN that utilizes 3 different neighborhoods per node to generate 3 node vectors that are then fused together via an attention mechanism. The 3 different neighborhoods are: 1. 1-hop neighborhood of the node 2. 2-hop neighborhood of the node 3. kNN based neighborhood of the node where proximity is calculated based on cosine similarity of node attributes  To motivate the need for this architecture, the authors first elaborate on phenomena like homophily, heterophily, and presence of random connections in real-world graphs to highlight that GNNs that solely rely on homophily may not perform well on real-world graphs.