Towards Powerful Graph Neural Networks: Diversity Matters

1 INTRODUCTION . Graph neural networks ( GNNs ) have been shown to be effective at graph representation learning and many predictive tasks on graph-structured data , e.g. , node classification and graph classification ( Kipf & Welling , 2016 ; Xu et al. , 2018a ) . GNNs follow a neighborhood aggregation scheme , where the representation of a node is obtained by recursively aggregating and transforming representation of its neighboring nodes ( Gilmer et al. , 2017 ) . The success of GNNs is believed to be attributed to their high expressive power at learning representation of nodes and graphs ( Xu et al. , 2018a ) . Therefore , it is an important research problem to analyze and improve the expressive power of existing GNN models and design new GNNs with high expressive power . Several recent works focus on the expressive power of GNNs . Xu et al . pointed out that the expressive power of GNNs depends on the neighborhood aggregation function ( Xu et al. , 2018a ) . They develop a simple architecture , i.e. , leveraging multi-layer perceptron ( MLP ) and a sum pooling as a universal approximator defined on multi-set , to achieve injective neighborhood aggregation function . With injective aggregation function in each layer , the proposed graph isomorphism network ( GIN ) has the expressive power as high as the Weisfeiler-Lehman ( WL ) graph isomorphism test ( Weisfeiler & Lehman , 1968 ) . Similarly , Sato et al . implement a powerful GNN via consistent port numbering , i.e. , mapping edges to port numbering and neighbors are ordered by the port numbering ( Sato et al. , 2019 ) . However , port ordering of CPNGNNs is not unique , and not all orderings can distinguish the same set of graphs ( Garg et al. , 2020 ) . Principal neighborhood aggregation ( PNA ) defines multiple aggregation functions to improve the expressive power of GNNs ( Corso et al. , 2020 ) . However , the number of required aggregation functions to discriminate multi-sets depends on the size of multi-set , which is prohibitive for real world networks with skewed degree distribution . In sum , existing methods focus on designing an injective , often complex , aggregation function in each layer to achieve GNNs with high expressive power . However , injective functions are difficult to obtain and tricky to determine in practice . Indeed , layer-wise injective function is not always required and what we need is an injective function defined over rooted sub-graphs or graphs as a whole . In this paper , we propose a novel framework , namely diverse sampling , to improve the expressive power of GNNs . For a target node , diverse sampling offers it diverse neighborhoods , i.e. , rooted subgraphs , and the representation of target node is finally obtained via aggregating the representation of diverse neighborhoods obtained using any GNN model . High expressive power is guaranteed by the diversity of different neighborhoods . For convenience , we denote with DS-GNN the GNN implemented under the proposed diverse sampling framework . Input . Injective . Fig . 1 illustrates the main idea of the proposed DS-GNN , and compare it with two representative methods , i.e. , GIN and PNA . Fig . 1 ( a ) depicts the injective layer implemented via MLP or multiple aggregation functions , aggregating first-order neighboring nodes to obtain the representation of central node . Injective layer are stacked to achieve an overall injective function defined on rooted sub-graphs . On the contrary , DS-GNN does not follow the line of designing complicated aggregation functions in each layer . Instead , DS-GNN improve the expressive power of GNNs via obtaining diverse rooted sub-graphs for each node . Specifically , we sample nodes multiple times on the entire input graph based on diverse sampling , and obtain multiple sampled sub-graphs for each node . After diverse sampling , we leverage the shared GNN model to get the representation for the central node , including its high-order neighbors . In this way , each node is represented by a multi-set , consisting of the representations obtained from different sampled rooted sub-graphs . The final representation of central node is finally obtained via aggregating the representation of diverse neighborhoods . Finally , we use classical GNNs ( i.e. , GCN ( Kipf & Welling , 2016 ) and GAT ( Veličković et al. , 2017 ) ) as base models to evaluate the effectiveness of the proposed framework . Experiments are conducted at node-based multi-class classification task on three benchmark datasets and node-based multilabel classification task on a dataset collected in this paper . Extensive experiments demonstrate the proposed method consistently improve the performance of base GNN models . The proposed framework is applicable to any GNN models and thus is general for improving the expressive power of GNNs . 2 NOTATIONS AND PRELIMINARIES . We first introduce the general framework of GNNs . G = { V , A } denotes an undirected graph , where V is the set of nodes with |V | = n , and A is the adjacency matrix with Ai , j = Aj , i to define the connection between node i and node j. X ∈ Rn×p denotes the feature matrix and the i-th row in X represent the attributes of i-th node . DS-GNN Modern GNNs follow a neighborhood aggregation scheme , which iteratively update each node ’ s representation via aggregating the representation of its neighboring nodes . Formally , the k-th layer of GNN is a ( k ) v = AGGREGATE ( k ) ( h ( k−1 ) u : u ∈ N ( v ) ) , h ( k ) v = COMBINE ( k ) ( h ( k−1 ) v , a ( k ) v ) , ( 1 ) whereN ( v ) represents the neighbors of node v , h ( k ) v is the representation of node v in the k-th layer , and h ( 0 ) v = Xv . Additionally , we introduce two representative base models : Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2016 ) and Graph Attention Networks ( GAT ) ( Veličković et al. , 2017 ) in Appendix A . 3 DIVERSE SAMPLING BASED POWERFUL GRAPH NEURAL NETWORK . Considering that the injectivity of each node with its neighbors making GNNs be possessed with the most powerful ability for node-level task , i.e. , representations can be distinguishable for two nodes when they have dissimilar attributes or different neighbors , we devote to designing a powerful GNN to reach such injectivity , thus increase its expressive power . Specifically , we propose a novel framework , i.e. , Diverse Sampling ( DS-GNN ) , to increase the expressive power via constructing diverse rooted sub-graphs for each node . In this section , we first give the framework of our DSGNN . Then we will introduce the important parts of DS-GNN , including diverse sampling strategy that ensure the diversity among each sampling , as well as the part of model learning . We also theoretically analyze how the proposed DS-GNN can improve the expressive abilities when added to generic GNNs . 3.1 METHOD The framework of DS-GNN is illustrated in Fig . 2 . For the input graph , instead of running GNN on the entire graph , we first do node sampling with K-times and then obtain K sampled graphs . During sampling , we calculate the sampling probability for each node and randomly retain the node based on the probability . If one node is not sampled , it means that the node and all edges connected with it will be removed from the graph . The updated adjacency matrix and feature matrix of each sampling are fed into the base GNN model , and the GNN is shared over all samplings . Then we obtain one representation for each sampled node corresponded with one input sampled graph . Thus , if one node is sampled more than one time , we will obtain multiple representations . To get the final representation of each node , we need to integrate these representations . To hold the injectivity when integrating representations , we adopt a multi-layer perceptrons ( MLP ) followed by the sum aggregation to achieve the injective multi-set aggregation function . Once getting the final representation for each node , we calculate the loss function and optimize the model . Sampling Strategy Intuitively , with a large number of sampling numbers , the sampling results are expected to retain the information of entire graph . Thus previous methods , e.g. , Dropedge ( Rong et al. , 2019 ) which samples in each training epoch , do not need to carefully design the sampling strategy and they keep all edges when testing and validation . Meanwhile , they fail to achieve diverse sampled graphs in testing . Different from previous methods , we need to hold the K sampled graphs in testing and keep training and testing consistently . To solve this , we only sample the entire graph K-times , and leverage these sampled K graphs in all training epochs , as well as the validation and testing . To guarantee the performance of the proposed DS-GNN when even small K is chosen , we carefully design the important diverse sampling strategy . To ensure the reuse of sampling results , we propose node sampling to obtain K sampled graphs . In one sampling , the neighbors of each node are obtained from the same sampled graph . For the sampling probability of nodes , the most intuitive way is to set the same initial probability for each node , which is pinit . However , this method may require a large number of sampling numbers K. In order to further speed up or get an effective sampling plan , we also propose three intuitive guidelines for sampling strategy : 1 ) to ensure the coverage of samples , each node should appears at least in one sampled graph ; 2 ) to achieve the diversity among samples , we assume that the sampling probability of each node should decrease when its sampled times in previous is already large ; 3 ) to retain the important node in the original network and adopting the node degree as the importance indicator , we assume that the sampling probability of each node should increase when its degree in original network is large . Meanwhile , we also considering the previous sampled results , discounting the sampling probability of each node when the its degree in previous sampled graphs is already large . That ’ s to say , the sampling probability of each node increase with the increase of node degree in original network , and decrease with the increase of node degree in previous sampled graphs . To achieve the above three guidelines , we need to relate the sampling probability of each node to the historical sampling results . Next , we will introduce in detail how we do . Let K represent the total number of samples , pv , i represent the probability of node v in the i-th sampling , and Hv , i represent the number of times that the node v has been sampled before the ith sampling . To achieve guideline 1 ) , we should guarantee that the probability for a node , which has never been sampled in history K-1 samples , equals to 1 at the K-th sampling . In other words , pv , K = 1 when Hv , K = 0 . Additionally , to achieve the diversity guideline of 2 ) , the sampling probability pv , i should decrease with the increase of historical sampled numbers Hv , i . Together , the sampling probability of node v for the i-th sampling is defined as : pv , i = pinit + ( 1− pinit ) × i−Hv , i K . ( 2 ) To achieve guideline of 3 ) , a new sampling strategy is designed . Specifically , we define Dsamplev , j as the degree of node v in the j-th sampled graph , and Dv as the degree of node v in the original graph . To make the sampling probability increase with Dv , as well as decrease with D sample v , j as mentioned in guideline 3 ) , we define the coefficient Dgapv , i as : Dgapv , i = ∑i−1 j=1 ( Dv −D sample v , j ) ∑i−1 j=1Dv ( 3 ) To achieve our three guidelines in total , we integrate the Eq 2 and Eq 3 , and define the sampling probability for node v in the i-th sampling as : pv , i = pinit + ( 1− pinit ) × i−Hv , i K ×Dgapv , i . ( 4 ) Note that our sampling strategy is different from previous methods , i.e. , dropping features and dropping edges . The Dropout ( Srivastava et al. , 2014 ) is proposed to prevent over-fitting by randomly dropping features . Dropping edges can be regarded as a generation of Dropout from dropping features to dropping edges . However , dropping edges may cause that two nodes who are the same with identical neighbors obtain different representations , since dropping edges may cause these two nodes result in different neighbors after edge sampling . In contrast , we propose node sampling strategy , ensuring that the nodes with the same input features and neighbors will still have the same representation . We give the detailed introduction of some previous methods , included in dropping edges in Appendix B .

Summary: This paper claims that existing GNNs often suffers from the limited capability of the aggregation function. This paper proposes a new framework of a diverse sampling of the graph to solve this problem. Specifically, this paper first samples several different graphs and use GNN on each graph to generate features, and finally use a type of injective multi-set aggregation function to obtain the final representation. The experimental results show that adding this module to GCN and GAT can further boost node-based multi-class classification performance.