Towards Powerful Graph Neural Networks: Diversity Matters

1 INTRODUCTION . Graph neural networks ( GNNs ) have been shown to be effective at graph representation learning and many predictive tasks on graph-structured data , e.g. , node classification and graph classification ( Kipf & Welling , 2016 ; Xu et al. , 2018a ) . GNNs follow a neighborhood aggregation scheme , where the representation of a node is obtained by recursively aggregating and transforming representation of its neighboring nodes ( Gilmer et al. , 2017 ) . The success of GNNs is believed to be attributed to their high expressive power at learning representation of nodes and graphs ( Xu et al. , 2018a ) . Therefore , it is an important research problem to analyze and improve the expressive power of existing GNN models and design new GNNs with high expressive power . Several recent works focus on the expressive power of GNNs . Xu et al . pointed out that the expressive power of GNNs depends on the neighborhood aggregation function ( Xu et al. , 2018a ) . They develop a simple architecture , i.e. , leveraging multi-layer perceptron ( MLP ) and a sum pooling as a universal approximator defined on multi-set , to achieve injective neighborhood aggregation function . With injective aggregation function in each layer , the proposed graph isomorphism network ( GIN ) has the expressive power as high as the Weisfeiler-Lehman ( WL ) graph isomorphism test ( Weisfeiler & Lehman , 1968 ) . Similarly , Sato et al . implement a powerful GNN via consistent port numbering , i.e. , mapping edges to port numbering and neighbors are ordered by the port numbering ( Sato et al. , 2019 ) . However , port ordering of CPNGNNs is not unique , and not all orderings can distinguish the same set of graphs ( Garg et al. , 2020 ) . Principal neighborhood aggregation ( PNA ) defines multiple aggregation functions to improve the expressive power of GNNs ( Corso et al. , 2020 ) . However , the number of required aggregation functions to discriminate multi-sets depends on the size of multi-set , which is prohibitive for real world networks with skewed degree distribution . In sum , existing methods focus on designing an injective , often complex , aggregation function in each layer to achieve GNNs with high expressive power . However , injective functions are difficult to obtain and tricky to determine in practice . Indeed , layer-wise injective function is not always required and what we need is an injective function defined over rooted sub-graphs or graphs as a whole . In this paper , we propose a novel framework , namely diverse sampling , to improve the expressive power of GNNs . For a target node , diverse sampling offers it diverse neighborhoods , i.e. , rooted subgraphs , and the representation of target node is finally obtained via aggregating the representation of diverse neighborhoods obtained using any GNN model . High expressive power is guaranteed by the diversity of different neighborhoods . For convenience , we denote with DS-GNN the GNN implemented under the proposed diverse sampling framework . Input . Injective . Fig . 1 illustrates the main idea of the proposed DS-GNN , and compare it with two representative methods , i.e. , GIN and PNA . Fig . 1 ( a ) depicts the injective layer implemented via MLP or multiple aggregation functions , aggregating first-order neighboring nodes to obtain the representation of central node . Injective layer are stacked to achieve an overall injective function defined on rooted sub-graphs . On the contrary , DS-GNN does not follow the line of designing complicated aggregation functions in each layer . Instead , DS-GNN improve the expressive power of GNNs via obtaining diverse rooted sub-graphs for each node . Specifically , we sample nodes multiple times on the entire input graph based on diverse sampling , and obtain multiple sampled sub-graphs for each node . After diverse sampling , we leverage the shared GNN model to get the representation for the central node , including its high-order neighbors . In this way , each node is represented by a multi-set , consisting of the representations obtained from different sampled rooted sub-graphs . The final representation of central node is finally obtained via aggregating the representation of diverse neighborhoods . Finally , we use classical GNNs ( i.e. , GCN ( Kipf & Welling , 2016 ) and GAT ( Veličković et al. , 2017 ) ) as base models to evaluate the effectiveness of the proposed framework . Experiments are conducted at node-based multi-class classification task on three benchmark datasets and node-based multilabel classification task on a dataset collected in this paper . Extensive experiments demonstrate the proposed method consistently improve the performance of base GNN models . The proposed framework is applicable to any GNN models and thus is general for improving the expressive power of GNNs . 2 NOTATIONS AND PRELIMINARIES . We first introduce the general framework of GNNs . G = { V , A } denotes an undirected graph , where V is the set of nodes with |V | = n , and A is the adjacency matrix with Ai , j = Aj , i to define the connection between node i and node j. X ∈ Rn×p denotes the feature matrix and the i-th row in X represent the attributes of i-th node . DS-GNN Modern GNNs follow a neighborhood aggregation scheme , which iteratively update each node ’ s representation via aggregating the representation of its neighboring nodes . Formally , the k-th layer of GNN is a ( k ) v = AGGREGATE ( k ) ( h ( k−1 ) u : u ∈ N ( v ) ) , h ( k ) v = COMBINE ( k ) ( h ( k−1 ) v , a ( k ) v ) , ( 1 ) whereN ( v ) represents the neighbors of node v , h ( k ) v is the representation of node v in the k-th layer , and h ( 0 ) v = Xv . Additionally , we introduce two representative base models : Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2016 ) and Graph Attention Networks ( GAT ) ( Veličković et al. , 2017 ) in Appendix A . 3 DIVERSE SAMPLING BASED POWERFUL GRAPH NEURAL NETWORK . Considering that the injectivity of each node with its neighbors making GNNs be possessed with the most powerful ability for node-level task , i.e. , representations can be distinguishable for two nodes when they have dissimilar attributes or different neighbors , we devote to designing a powerful GNN to reach such injectivity , thus increase its expressive power . Specifically , we propose a novel framework , i.e. , Diverse Sampling ( DS-GNN ) , to increase the expressive power via constructing diverse rooted sub-graphs for each node . In this section , we first give the framework of our DSGNN . Then we will introduce the important parts of DS-GNN , including diverse sampling strategy that ensure the diversity among each sampling , as well as the part of model learning . We also theoretically analyze how the proposed DS-GNN can improve the expressive abilities when added to generic GNNs . 3.1 METHOD The framework of DS-GNN is illustrated in Fig . 2 . For the input graph , instead of running GNN on the entire graph , we first do node sampling with K-times and then obtain K sampled graphs . During sampling , we calculate the sampling probability for each node and randomly retain the node based on the probability . If one node is not sampled , it means that the node and all edges connected with it will be removed from the graph . The updated adjacency matrix and feature matrix of each sampling are fed into the base GNN model , and the GNN is shared over all samplings . Then we obtain one representation for each sampled node corresponded with one input sampled graph . Thus , if one node is sampled more than one time , we will obtain multiple representations . To get the final representation of each node , we need to integrate these representations . To hold the injectivity when integrating representations , we adopt a multi-layer perceptrons ( MLP ) followed by the sum aggregation to achieve the injective multi-set aggregation function . Once getting the final representation for each node , we calculate the loss function and optimize the model . Sampling Strategy Intuitively , with a large number of sampling numbers , the sampling results are expected to retain the information of entire graph . Thus previous methods , e.g. , Dropedge ( Rong et al. , 2019 ) which samples in each training epoch , do not need to carefully design the sampling strategy and they keep all edges when testing and validation . Meanwhile , they fail to achieve diverse sampled graphs in testing . Different from previous methods , we need to hold the K sampled graphs in testing and keep training and testing consistently . To solve this , we only sample the entire graph K-times , and leverage these sampled K graphs in all training epochs , as well as the validation and testing . To guarantee the performance of the proposed DS-GNN when even small K is chosen , we carefully design the important diverse sampling strategy . To ensure the reuse of sampling results , we propose node sampling to obtain K sampled graphs . In one sampling , the neighbors of each node are obtained from the same sampled graph . For the sampling probability of nodes , the most intuitive way is to set the same initial probability for each node , which is pinit . However , this method may require a large number of sampling numbers K. In order to further speed up or get an effective sampling plan , we also propose three intuitive guidelines for sampling strategy : 1 ) to ensure the coverage of samples , each node should appears at least in one sampled graph ; 2 ) to achieve the diversity among samples , we assume that the sampling probability of each node should decrease when its sampled times in previous is already large ; 3 ) to retain the important node in the original network and adopting the node degree as the importance indicator , we assume that the sampling probability of each node should increase when its degree in original network is large . Meanwhile , we also considering the previous sampled results , discounting the sampling probability of each node when the its degree in previous sampled graphs is already large . That ’ s to say , the sampling probability of each node increase with the increase of node degree in original network , and decrease with the increase of node degree in previous sampled graphs . To achieve the above three guidelines , we need to relate the sampling probability of each node to the historical sampling results . Next , we will introduce in detail how we do . Let K represent the total number of samples , pv , i represent the probability of node v in the i-th sampling , and Hv , i represent the number of times that the node v has been sampled before the ith sampling . To achieve guideline 1 ) , we should guarantee that the probability for a node , which has never been sampled in history K-1 samples , equals to 1 at the K-th sampling . In other words , pv , K = 1 when Hv , K = 0 . Additionally , to achieve the diversity guideline of 2 ) , the sampling probability pv , i should decrease with the increase of historical sampled numbers Hv , i . Together , the sampling probability of node v for the i-th sampling is defined as : pv , i = pinit + ( 1− pinit ) × i−Hv , i K . ( 2 ) To achieve guideline of 3 ) , a new sampling strategy is designed . Specifically , we define Dsamplev , j as the degree of node v in the j-th sampled graph , and Dv as the degree of node v in the original graph . To make the sampling probability increase with Dv , as well as decrease with D sample v , j as mentioned in guideline 3 ) , we define the coefficient Dgapv , i as : Dgapv , i = ∑i−1 j=1 ( Dv −D sample v , j ) ∑i−1 j=1Dv ( 3 ) To achieve our three guidelines in total , we integrate the Eq 2 and Eq 3 , and define the sampling probability for node v in the i-th sampling as : pv , i = pinit + ( 1− pinit ) × i−Hv , i K ×Dgapv , i . ( 4 ) Note that our sampling strategy is different from previous methods , i.e. , dropping features and dropping edges . The Dropout ( Srivastava et al. , 2014 ) is proposed to prevent over-fitting by randomly dropping features . Dropping edges can be regarded as a generation of Dropout from dropping features to dropping edges . However , dropping edges may cause that two nodes who are the same with identical neighbors obtain different representations , since dropping edges may cause these two nodes result in different neighbors after edge sampling . In contrast , we propose node sampling strategy , ensuring that the nodes with the same input features and neighbors will still have the same representation . We give the detailed introduction of some previous methods , included in dropping edges in Appendix B .

Summary: In this paper, the authors propose to sample nodes of a given graph multiple times to form a set of K sub-graphs. GNNs are then applied on each sampled graph for learning node representations. For each node, all representations are combined for the downstream tasks. The idea of doing multiple sampling is to increase the chance that nodes with different neighborhoods can be more different in the set of sampled graphs, than only considering the original single graph. In contrast, nodes with same neighborhood will remain the same over all sampled graphs. This mechanism helps discriminate node representations better.