Universal Graph Convolutional Networks

1 Introduction . Real-world complex systems can often be viewed as networks , such as social networks , biological networks and citation networks . Recently , research of analyzing networks with deep learning has received widespread attention both in academia and industry . In particular , Graph Convolutional Networks ( GCNs ) [ 14 ] , which obtain the meaningful representation of nodes in the network by integrating the neighborhood information , have achieved great success and been widely applied in tackling network analytics tasks , such as node classification [ 23 , 28 ] , link prediction [ 33 ] and recommendation [ 30 , 17 ] . While the success of GCNs and their variants [ 1 , 6 ] , a key weakness is the homophily assumption of networks , which restricts their performance on general network data . To be specific , most GCNs †Equal contributions . * Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . seem to be tailor-made to work on homophily networks [ 18 ] , where nodes within the same class tend to connect with each other . In fact , heterophily [ 20 ] networks , where nodes of different classes tend to link together , are also widespread in real-world . For example , different types of amino acids are more likely to be connected in protein structure [ 35 ] , and fraudsters tend to connect to accomplices than to other fraudsters in transaction networks [ 21 ] . Furthermore , the random networks also often exist in real-world , such as railway networks , where the edges between nodes are more likely to be randomly generated . Most popular GCNs typically obtain node embeddings of these networks using 1-hop network neighbors as neighborhoods for information propagation [ 12 , 29 ] . However , considering the different structural properties of networks ( e.g. , homophily or heterophily ) , whether different networks should adopt a unified or different propagation mechanisms ? This is a very important question for GCNs since they mainly gain better performance through the propagation of information . A well informed answer can help us better understand the essence of GCNs , such as how different types of nodes affect the propagation , and what type of nodes are really required to achieve a certain level of predictive accuracy aiming to different networks . Several recent works have studied the networks with different structural properties . For example , Pei et al . [ 22 ] consider the heterophlily property of networks , and propose a geometric aggregation scheme to overcome neighborhood structural information losing and long-range dependencies lacking . Zhu et al . [ 35 ] design an effective model which improves the representation power of GCNs under heterophily through theoretical and empirical analysis . Chien et al . [ 4 ] introduce a new generalized pageRank ( GPR ) architecture to jointly optimize node feature and topological information extraction . Bo et al . [ 2 ] assess the roles of low-frequency and high-frequency signals , and propose an efficient method that can adaptively integrate different signals in the process of message passing . However , there is still a lack of insightful understanding from the perspective of propagation mechanism . As the first contribution of this study , we conduct experiments analysing the propagation mechanism of GCNs in networks with different structural properties . Surprisingly , our experiments clearly illustrate that for networks with complete homophily , complete heterophily and randomness , 1-hop , 2-hop and k-nearest neighbor ( kNN ) neighbors are more suitable as neighborhoods for information propagation , respectively . This means that the depicting ability of the current propagation mechanism of GCNs is limited , and networks with different structural properties may need to adopt different propagation mechanisms . In fact , while these segmentation rules seem to be able to select appropriate nodes as neighborhoods in an ideal way , the real-world networks are complex , and may present diverse properties , e.g. , the network dominated by homophily may contain a small amount of randomness or heterophily . A natural question is , “ Can we reasonably utilize these segmentation rules to design a universal propagation mechanism independent of the network structural assumption ? '' To tackle this challenge , we propose a novel and universal GCN model , i.e. , U-GCN , for general network data . The central idea is that we learn node embeddings by making full use of the information from 1-hop , 2-hop and kNN neighbors , and fuse them adaptively to derive deeper correlation information for the given learning objectives . To be specific , we first introduce a multi-type convolution mechanism . It uses 1-hop network ( i.e. , original input network ) , 2-hop network and kNN network that constructed by 1-hop , 2-hop and kNN neighbors for direct information propagation separately , and utilizes a node-level attention mechanism for each network , to extract three specific embeddings . We then make a discriminative aggregation to learn out the importance of these three embeddings , thereby extracting the most correlated information aiming to the ground truth such as node classification . Extensive experiments on a series of benchmark datasets demonstrate the superiority of U-GCN over some state-of-the-arts . 2 Notations and Preliminaries . Let G = ( A , X ) be an undirected attributed network , where A ∈ Rn×n represents the symmetric adjacency matrix with n nodes , and X ∈ Rn×p is the attribute ( content ) matrix of p attributes per node . Concretely , aij = 1 denotes there is an edge between nodes vi and vj , or 0 otherwise ; and xi represents the attribute vectors of node vi . Given an attribute network G , and a labeled node set VL containing u |V | nodes , where each node vi ∈ VL contains a unique class label yi ∈ Y . The goal of semi-supervised node classification is to infer the labels of nodes in V \VL by learning a classification function F . Homophily . In this work , the level of homophily ratio of edges [ 35 ] is used to define networks with strong homophily/heterophily . Specifically , the level of homophily ratio of edges is the fraction of edges in a network which connect nodes that have the same class label ( i.e. , intra-class edges ) , described by : α = | ( vi , vj ) : aij = 1 ∧ yi = yj | |m| , ( 1 ) where m is the number of edges . Networks with α closer to 1 tend to have more edges connecting nodes within the same class , or stronger homophily ; whereas networks with α closer to 0 have more edges connecting nodes in different classes , or stronger heterophily . Graph Convolutional Network . Graph Convolutional Network ( GCN ) [ 14 ] is a variant of multilayer convolutional neural networks that operates directly on networks . It learns embedding of each node by iteratively aggregating the information from its neighbors . Mathematically , let H ( l ) be the feature representation of the l-th layer , and H ( 0 ) be the node attribute matrix , the forward propagation can be defined as : H ( l ) = σ ( D̃− 1 2 ÃD̃− 1 2H ( l−1 ) W ( l ) ) , ( 2 ) where Ã = A + I stands for the adjacency matrix with self-loops , D̃ the node degree matrix of Ã , i.e. , D̃ii = ∑ j Ãij , W ( l ) a trainable weight matrix and σ the non-linear activation function . While GCN works well on several network analysis tasks such as node classification [ 10 , 15 ] , it still has a fundamental problem , that is , homophily assumption of networks , which leads to the main contribution in this work , i.e. , analyse what type of nodes are more suitable as neighborhoods for direct information propagation independent of the network structural assumption . 3 Motivating Observations . Here , we present a simple yet intuitive case study to illustrate and analyze the performance of GCN changes with different propagation mechanisms . The main idea is that we will apply GCN to networks with different structural properties utilizing three types of nodes : 1-hop , 2-hop and k-nearest neighbor ( kNN ) neighbors , which are often believed to be the effective neighborhoods for node classification in networks [ 28 , 35 ] , to realize the information propagation , respectively . Then , we will check the performance of GCN on these cases . A universal propagation mechanism should provide a good result in general network data . However , if the performance drops sharply in comparison with the other two situations , this will demonstrate that networks with different structural properties may need to use different propagation mechanisms . Setup . We conduct experiments on the Newman artificial networks [ 7 ] with different properties . The network consists 128 nodes divided into 4 classes , where each node has on average zin edges ( i.e. , intra-class edges ) connecting to nodes of the same class and zout edges ( i.e. , inter-class edges ) to nodes of other classes , and zin + zout = 16 . Note that here we utilize two indicators : ρin = zin/32 and ρout = zout/96 , to determine the network property , i.e. , ρin > ρout , ρin = ρout and ρin < ρout means the network with homophily , randomness and heterophily , respectively . For node attributes , we generate a 4h-dimensional binary attributes ( i.e. , xi ) for each node to form 4 attribute clusters , corresponding to the 4 classes [ 9 ] . To be specific , for every node in the m-th class , we use a binomial distribution with mean pin = hin/h to generate a h-dimensional binary vector as its ( ( m − 1 ) × h + 1 ) -th to ( m × h ) -th attributes , and generated the rest attributes using a binomial distribution with mean pout = hout/ ( 3h ) . In our experiments , we set 4h = 200 and hout = 4 ( hin+hout = 16 ) , so that pin > pout , the h-dimensional attributes are associated with the m-th class with a higher probability , whereas the rest 3h attributes are irrelevant . As shown in Figure 1 , for networks with strong homophily ( e.g. , ρout = 0.075 ) , it is easy to obtain high accuracy using 1-hop network neighbors . However , as the inter-class edges increase , the accuracy is rapidly reduced . This mainly due to the homophily assumption , preventing GCN from effectively fusing information . On the other hand , for networks with strong heterophily ( e.g. , ρout = 0.165 ) , it is surprising that , the accuracy of GCN of using 2-hop neighbors as neighborhoods ( i.e. , 83.15 % ) is much higher than that of using 1-hop network neighbors ( i.e. , 32.85 % ) . Since the homophily ratio of 2-hop neighbors may rise with the increase of inter-class edges , GCN of using 2-hop neighbors is more effective to some extent . Interestingly , we can find that GCN of utilizing kNN is easy to get the staple accuracy , i.e. , 71.46 % . In particular , it is much higher than those of using 1-hop and 2-hop neighbors on complete random network ( i.e. , ρout = 0.125 ) . Summary . This case study shows that the current propagation mechanism of GCN is not universal for general network data , but we can find that there are rules in several special situations ( i.e. , complete random network ) . This motivates us that networks with different structural properties may need adopt different propagation mechanisms . We conduct extra experiments on Newman networks [ 7 ] with complete homophily , randomness and complete heterophily utilizing GCN , so as to discover more appropriate propagation mechanism to select valuable nodes as neighborhoods , and thus improving the performance of GCNs for different networks . One straightforward strategy is to learn network embeddings with different GCNs using different types of nodes ( i.e. , 1-hop , 2-hop and kNN neighbors ) , and concatenate the embeddings into a single vector , so as to use the discriminative aggregation mechanism ( which will be introduced in Section 4.2 below ) to learn their importance for node classification . We show the attention values as a function of the number of training iterations in Figure 2 . Observation 1 : Network with complete homophily tends to obtain better performance utilizing 1-hop network neighbors for direct information propagation . For the network in Figure 2A , the edges exist only in nodes within the same class , or complete homophily . As shown , 1-hop network neighbors show great importance as the increase of training iterations . This partly validates that under the setting of complete homophily , 1-hop neighbors are more effective for direct information propagation compared to 2-hop and kNN neighbors . Observation 2 : Network with randomness tends to get better performance utilizing kNN for direct information propagation . The network in Figure 2B exhibits more randomness than Figure 2A , that is , complete random network . Obviously , with the increase of training iterations , the attention value of kNN is much higher than those of 1-hop and 2-hop neighbors . Since kNN typically constructed according to the similarity of node attributes , it can still realize the information fusion effectively , compared with the other two types of neighbors , in case that the network topology contains noise . Observation 3 : Network with complete heterophily tends to obtain better performance utilizing 2-hop neighbors for direct information propagation . For the network in Figure 2C , the edges exist only in nodes within different classes , or complete heterophily . While the learned attention values of these three types of neighbors differ slightly , the importance of 2-hop neighbors is relatively higher . This is mainly due to the fact that the homophily ratio of 2-hop neighbors becomes higher with the increase of inter-class edges ( which is often the real life in many network analysis tasks ) . While networks with different structural properties provide better performance utilize different propagation mechanisms , the real-world networks are complex , and may show diverse properties , e.g. , the network dominated by homophily may contain a small amount of heterophily . Therefore , it is imperative to explore a universal propagation mechanism for GCNs independent of the network structural assumption . Theorem 1 . The real-world networks can be approximately decomposed into a mixture of three kinds of simple networks , namely complete homophily , complete random and complete heterophily , in different proportions . Proof . Given a network G , where n and m denote the number of nodes and edges , respectively . Assuming that r edges ( 0 ≤ r ≤ m ) are generated randomly , i.e. , the probability of nodes connecting nodes within the same class or different classes is the same , which form a complete random network ( with n nodes and r edges ) . The remaining edges connecting two nodes within the same class can then be regarded as composing a complete homophily network , or a complete heterophily network . Summary . Now , we can conclude that a universal GCN model may not only consider the 1-hop ( Observation 1 ) , but also the 2-hop ( Observation 2 ) and kNN neighbors ( Observation 3 ) for direct information propagation . More importantly , considering different network properties can be more correlated with one of them or even their combinations , the model itself should adaptively learn their corresponding importance , so as to achieve feature fusion more effectively . This case study , although leveraging specific artificial networks , is representative because real-world networks can often be considered as the combination of these three simple network cases .

Summary: This paper is to improve GCN by considering different hops relationship of the network. It starts with the observation that for networks with complete homophily, complete heterophily and randomness, 1-hop network, 1-hop neighbors, 2-hop neighbors and k-nearest neighbor (kNN) are more suitable as neighborhoods for information propagation for GCN, respectively. Therefore it proposed a new algorithm: U-GCN, which first individually propagates node information under different hops of graphs and then combine their results using discriminative aggregation. It compares with traditional GCN methods in small scale datasets.