Measuring and Improving the Use of Graph Information in Graph Neural Networks

1 INTRODUCTION . Graphs are powerful data structures that allow us to easily express various relationships ( i.e. , edges ) between objects ( i.e. , nodes ) . In recent years , extensive studies have been conducted on GNNs for tasks such as node classification and link predication . GNNs utilize the relationship information in graph data and significant improvements over traditional methods have been achieved on benchmark datasets ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Velickovic et al. , 2018 ; Xu et al. , 2019 ; Hou et al. , 2019 ) . Such breakthrough results have led to the exploration of using GNNs and their variants in different areas such as computer vision ( Satorras & Estrach , 2018 ; Marino et al. , 2017 ) , natural language processing ( Peng et al. , 2018 ; Yao et al. , 2019 ) , chemistry ( Duvenaud et al. , 2015 ) , biology ( Fout et al. , 2017 ) , and social networks ( Wang et al. , 2018 ) . Thus , understanding why GNNs can outperform traditional methods that are designed for Euclidean data is important . Such understanding can help us analyze the performance of existing GNN models and develop new GNN models for different types of graphs . In this paper , we make two main contributions : ( 1 ) two graph smoothness metrics to help understand the use of graph information in GNNs , and ( 2 ) a new GNN model that improves the use of graph information using the smoothness values . We elaborate the two contributions as follows . One main reason why GNNs outperform existing Euclidean-based methods is because rich information from the neighborhood of an object can be captured . GNNs collect neighborhood information with aggregators ( Zhou et al. , 2018 ) , such as the mean aggregator that takes the mean value of neighbors ’ feature vectors ( Hamilton et al. , 2017 ) , the sum aggregator that applies summation ( Duvenaud et al. , 2015 ) , and the attention aggregator that takes the weighted sum value ( Velickovic et al. , 2018 ) . Then , the aggregated vector and a node ’ s own feature vector are combined into a new feature vector . After some rounds , the feature vectors of nodes can be used for tasks such as node classification . Thus , the performance improvement brought by graph data is highly related to the quantity and quality of the neighborhood information . To this end , we propose two smoothness metrics on node features and labels to measure the quantity and quality of neighborhood information of nodes . The metrics are used to analyze the performance of existing GNNs on different types of graphs . In practice , not all neighbors of a node contain relevant information w.r.t . a specific task . Thus , neighborhood provides both positive information and negative disturbance for a given task . Simply aggregating the feature vectors of neighbors with manually-picked aggregators ( i.e. , users choose a type of aggregator for different graphs and tasks by trial or by experience ) often can not achieve optimal performance . To address this problem , we propose a new model , CS-GNN , which uses the smoothness metrics to selectively aggregate neighborhood information to amplify useful information and reduce negative disturbance . Our experiments validate the effectiveness of our two smoothness metrics and the performance improvements obtained by CS-GNN over existing methods . 2 MEASURING THE USEFULNESS OF NEIGHBORHOOD INFORMATION . We first introduce a general GNN framework and three representative GNN models , which show how existing GNNs aggregate neighborhood information . Then we propose two smoothness metrics to measure the quantity and quality of the information that nodes obtain from their neighbors . 2.1 GNN FRAMEWORK AND MODELS . The notations used in this paper , together with their descriptions , are listed in Appendix A . We use G = { V , E } to denote a graph , where V and E represent the set of nodes and edges of G. We use ev , v′ ∈ E to denote the edge that connects nodes v and v′ , and Nv = { v′ : ev , v′ ∈ E } to denote the set of neighbors of a node v ∈ V . Each node v ∈ V has a feature vector xv ∈ X with dimension d. Consider a node classification task , for each node v ∈ V with a class label yv , the goal is to learn a representation vector hv and a mapping function f ( · ) to predict the class label yv of node v , i.e. , ŷv = f ( hv ) where ŷv is the predicted label . GNNs are inspired by the Weisfeiler-Lehman test ( Weisfeiler & Lehman , 1968 ; Shervashidze et al. , 2011 ) , which is an effective method for graph isomorphism . Similarly , GNNs utilize a neighborhood aggregation scheme to learn a representation vector hv for each node v , and then use neural networks to learn a mapping function f ( · ) . Formally , consider the general GNN framework ( Hamilton et al. , 2017 ; Zhou et al. , 2018 ; Xu et al. , 2019 ) in Table 1 with K rounds of neighbor aggregation . In each round , only the features of 1-hop neighbors are aggregated , and the framework consists of two functions , AGGREGATE and COMBINE . We initialize h ( 0 ) v = xv . After K rounds of aggregation , each node v ∈ V obtains its representation vector h ( K ) v . We use h ( K ) v and a mapping function f ( · ) , e.g. , a fully connected layer , to obtain the final results for a specific task such as node classification . Many GNN models have been proposed . We introduce three representative ones : Graph Convolutional Networks ( GCN ) ( Kipf & Welling , 2017 ) , GraphSAGE ( Hamilton et al. , 2017 ) , and Graph Attention Networks ( GAT ) ( Velickovic et al. , 2018 ) . GCN merges the combination and aggregation functions , as shown in Table 1 , where A ( · ) represents the activation function and W is a learnable parameter matrix . Different from GCN , GraphSAGE uses concatenation ‘ || ’ as the combination function , which can better preserve a node ’ s own information . Different aggregators ( e.g. , mean , max pooling ) are provided in GraphSAGE . However , GraphSAGE requires users to choose an aggregator to use for different graphs and tasks , which may lead to sub-optimal performance . GAT addresses this problem by an attention mechanism that learns coefficients of neighbors for aggregation . With the learned coefficients a ( k−1 ) i , j on all the edges ( including self-loops ) , GAT aggregates neighbors with a weighted sum aggregator . The attention mechanism can learn coefficients of neighbors in different graphs and achieves significant improvements over prior GNN models . 2.2 GRAPH SMOOTHNESS METRICS . GNNs usually contain an aggregation step to collect neighboring information and a combination step that merges this information with node features . We consider the context cv of node v as the node ’ s own information , which is initialized as the feature vector xv of v. We use sv to denote the surrounding of v , which represents the aggregated feature vector computed from v ’ s neighbors . Since the neighborhood aggregation can be seen as a convolution operation on a graph ( Defferrard et al. , 2016 ) , we generalize the aggregator as weight linear combination , which can be used to express most existing aggregators . Then , we can re-formulate the general GNN framework as a context-surrounding framework with two mapping functions f1 ( · ) and f2 ( · ) in round k as : c ( k ) vi = f1 ( c ( k−1 ) vi , s ( k−1 ) vi ) , s ( k−1 ) vi = f2 ( ∑ vj∈Nvi a ( k−1 ) i , j · c ( k−1 ) vj ) . ( 1 ) From equation ( 1 ) , the key difference between GNNs and traditional neural-network-based methods for Euclidean data is that GNNs can integrate extra information from the surrounding of a node into its context . In graph signal processing ( Ortega et al. , 2018 ) , features on nodes are regarded as signals and it is common to assume that observations contain both noises and true signals in a standard signal processing problem ( Rabiner & Gold , 1975 ) . Thus , we can decompose a context vector into two parts as c ( k ) vi = c̆ ( k ) vi + n̆ ( k ) vi , where c̆ ( k ) vi is the true signal and n̆ ( k ) vi is the noise . Theorem 1 . Assume that the noise n̆ ( k ) vi follows the same distribution for all nodes . If the noise power of n̆ ( k ) vi is defined by its variance σ2 , then the noise power of the surrounding input∑ vj∈Nvi a ( k−1 ) i , j · c ( k−1 ) vj is ∑ vj∈Nvi ( a ( k−1 ) i , j ) 2 · σ2 . The proof can be found in Appendix B. Theorem 1 shows that the surrounding input has less noise power than the context when a proper aggregator ( i.e. , coefficient a ( k−1 ) i , j ) is used . Specifically , the mean aggregator has the best denoising performance and the pooling aggregator ( e.g. , max-pooling ) can not reduce the noise power . For the sum aggregator , where all coefficients are equal to 1 , the noise power of the surrounding input is larger than that of the context . 2.2.1 FEATURE SMOOTHNESS . We first analyze the information gain from the surrounding without considering the noise . In the extreme case when the context is the same as the surrounding input , the surrounding input contributes no extra information to the context . To quantify the information obtained from the surrounding , we present the following definition based on information theory . Definition 2 ( Information Gain from Surrounding ) . For normalized feature space Xk = [ 0 , 1 ] dk , if∑ vj∈Nvi a ( k ) i , j = 1 , the feature space of ∑ vj∈Nvi a ( k ) i , j · c̆ ( k ) vj is also in Xk = [ 0 , 1 ] dk . The probability density function ( PDF ) of c̆ ( k ) vj over Xk is defined as C ( k ) , which is the ground truth and can be estimated by nonparametric methods with a set of samples , where each sample point c̆ ( k ) vi is sampled with probability |Nvi |/2|E| . Correspondingly , the PDF of ∑ vj∈Nvi a ( k ) i , j · c̆ ( k ) vj is S ( k ) , which can be estimated with a set of samples { ∑ vj∈Nvi a ( k ) i , j · c̆ ( k ) vj } , where each point is sampled with probability |Nvi |/2|E| . The information gain from the surrounding in round k can be computed by Kullback–Leibler divergence ( Kullback & Leibler , 1951 ) as DKL ( S ( k ) ||C ( k ) ) = ∫ Xk S ( k ) ( x ) · log S ( k ) ( x ) C ( k ) ( x ) dx . The Kullback–Leibler divergence is a measure of information loss when the context distribution is used to approximate the surrounding distribution ( Kurt , 2017 ) . Thus , we can use the divergence to measure the information gain from the surrounding into the context of a node . When all the context vectors are equal to their surrounding inputs , the distribution of the context is totally the same with that of the surrounding . In this case , the divergence is equal to 0 , which means that there is no extra information that the context can obtain from the surrounding . On the other hand , if the context and the surrounding of a node have different distributions , the divergence value is strictly positive . Note that in practice , the ground-truth distributions of the context and surrounding signals are unknown . In addition , for learnable aggregators , e.g. , the attention aggregator , the coefficients are unknown . Thus , we propose a metric λf to estimate the divergence . Graph smoothness ( Zhou & Schölkopf , 2004 ) is an effective measure of the signal frequency in graph signal processing ( Rabiner & Gold , 1975 ) . Inspired by that , we define the feature smoothness on a graph . Definition 3 ( Feature Smoothness ) . Consider the condition of the first round , where c ( 0 ) v = xv , we define the feature smoothness λf over normalized space X = [ 0 , 1 ] d as λf = ∣∣∣∣∣∣∑v∈V ( ∑v′∈Nv ( xv − xv′ ) ) 2∣∣∣∣∣∣1 |E| · d , where || · ||1 is the Manhattan norm . According to Definition 3 , a larger λf indicates that the feature signal of a graph has higher frequency , meaning that the feature vectors xv and xv′ are more likely dissimilar for two connected nodes v and v′ in the graph . In other words , nodes with dissimilar features tend to be connected . Intuitively , for a graph whose feature sets have high frequency , the context of a node can obtain more information gain from its surrounding . This is because the PDFs ( given in Definition 2 ) of the context and the surrounding have the same probability but fall in different places in space X . Formally , we state the relation between λf and the information gain from the surrounding in the following theorem . For simplicity , we let X = X0 , d = d0 , C = C ( 0 ) and S = S ( 0 ) . Theorem 4 . For a graph G with the set of features X in space [ 0 , 1 ] d and using the mean aggregator , the information gain from the surrounding DKL ( S||C ) is positively correlated to its feature smoothness λf , i.e. , DKL ( S||C ) ∼ λf . In particular , DKL ( S||C ) = 0 when λf = 0 . The proof can be found in Appendix C. According to Theorem 4 , a large λf means that a GNN model can obtain much information from graph data . Note that DKL ( S||C ) here is under the condition when using the mean aggregator . Others aggregators , e.g. , pooling and weight could have different DKL ( S||C ) values , even if the feature smoothness λf is a constant .

Summary: The authors study how neighbor information on graphs can be used in Graph Neural Networks. It proposes measures on whether the data in neighboring nodes are useful in terms of labels or features. It also provides a new Graph Neural Network algorithm that is a modification of attention-based models incorporating the derived label and feature smoothness measures. The paper demonstrates the usefulness of these measures and algorithms with several different baselines from different families. The writing is mostly smooth, and the authors seem to provide enough detail of the experiments performed.