The Surprising Behavior Of Graph Neural Networks

1 INTRODUCTION . Convolutional Graph Neural Networks ( GNNs ) have produced state-of-the-art results in areas which utilize graph data ( Wu et al. , 2019 ) . Despite their widespread and rapid application across many fields , little research has been conducted on understanding the effect of topology on GNN behavior , especially in the context of attributed networks . Though GNNs use both graph modalities i.e. , the topology and nodal attributes , it is not clear whether they utilize topology to the same degree as nodal attributes or if they generalize across topological contexts . In this paper , we seek to underscore this lack of understanding . We present 4 empirical studies , which characterized and compared GNNs ’ utilization of topology , with surprisingly counter-intuitive results . Topology , Does it really matter ? : We empirically analyze GNN performance and show that it is , contrary to expectation , only loosely dependent on the nodal topological characteristics of a graph - particularly those of connectivity . The impact of connectivity is explored in the extreme case of a disconnected graph with multiple components ( Section 3 ) . Just Noisy Graphs : We analyze the impact of topological perturbations on the models ’ performance and dependence on topology . Given the weak dependence of GNN performance on topological features and the neighborhood aggregation mechanisms of GNNs , one would expect the graphs to be robust to topological noise . However , we see that GNN performance degrades considerably with noise ( Section 4 ) . Attributes & Topology , together or not ? : We show that instead of improving performance ( Fosdick & Hoff , 2015 ) , increased coupling between the modalities of topology and nodal attributes hampers it . We demonstrate a simple method to decouple the topological and attribute information which improves performance by acting as a regularization mechanism ( Section 5 ) . Attributes vs. Topology : We then create an experiment which questions whether the information in the graph structure and the attribute stack has any overlap and is inter-convertible by simple means ( Section 6 ) . The counter-intuitive results of these experiments highlight a gap in understanding of the behavior of GNNs in various common scenarios ( Du et al. , 2017 ) . This gap prohibits applying these powerful models to sensitive and diverse areas such as medicine ( Parisot et al. , 2018 ) , chemistry ( De Cao & Kipf , 2018 ) , and governance ( Li & Goldwasser , 2019 ) . The consequent recommendations from this study take the first step in shrinking that gap . 2 METHODS . In this work we compared the behavior of a set of GNNs across engineered graphs derived from a basket of benchmark datasets . Evaluation : The models were evaluated in a transductive node classification setting that closely follows the evaluation setup in Shchur et al . ( 2018 ) . For each model , we used a fixed set of hyperparameters reported in Shchur et al . ( 2018 ) , which are the best-performing configurations that have achieved the best average accuracy on Cora and Citeseer datasets ( averaged over 100 train/validation/test splits and 20 random initializations , using only the largest-connected-component . In reporting test accuracies , unless otherwise mentioned , we report the accuracy averaged over 20 train/validation/test splits and 2 random initializations for each model . Models : We consider the most prolific spectral and spatial GNNs , and compare against an attentionbased GNN . In this work , we study Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2016 ) , GraphSAGE ( GS ) ( Hamilton et al. , 2017 ) and Mixture Model Network ( MoNet ) ( Monti et al. , 2017 ) , and Graph Attention Network ( GAT ) ( Veličković et al. , 2017 ) from each category respectively . We also considered two additional non-GNN based methods : Label Propagation ( LabelProp ) and Label Propagation with Normalized Laplacian ( LabelProp NL ) . These methods only consider the graph structure and not its node attributes . Datasets : We use the following Citation , Co-author and Co-purchase networks ( Table 2 ) . Each network consists of an edge set and nodes that form the underlying topology . Attached to each node is an attribute stack ( or feature vector ) , which describes metadata associated with the node , and a label , which is the target of a node classification algorithm . Citation networks , used to evaluate GCNs and GATS : Cora , Citeseer ( Sen et al. , 2008 ) and Pubmed ( Namata et al. , 2012 ) , have documents as nodes , citation as edges and bag-of-words from the papers as node features . We consider a Co-author network , Coauthor Physics , with authors as nodes , paper co-authorship as edges , and keywords of the author ’ s papers as node features ( Shchur et al. , 2018 ) . We also consider Co-purchase networks : Amazon Computers and Amazon Photo ( Shchur et al. , 2018 ) , with goods as nodes , co-purchase as edges and product reviews as node features . 3 TOPOLOGY , DOES IT REALLY MATTER ? . While convolutional neural networks work with highly regular neighborhood structures ( 4- neighboring pixels ) , GNNs attempt to deal with situations where such regularity assumptions may not apply . They characterize nodes through the composition of their neighborhood either explicitly , as in spatial methods , or implicitly , as in approximate spectral methods . Thus , a denser neighborhood - one with more samples - would allow a more certain characterization of a node . In the following experiment , we investigate whether such an intuitive relationship with the underlying topological features , which proxy topological connectivity , exists . Question : Do topological features impact GNN performance ? In other words are nodal topological features good predictors of GNN performance ? Hypothesis/Expectation : Better connected nodes have larger neighborhoods and so should show less variance in GNN performance . Nodal topological characteristics should significantly impact performance . Counter-intuitive results : Nodal topology does not appear to strongly impact the performance of the GNN , even in the extreme case it appears to degrade performance only after a critical point . Methods : We selected features from Table 1 that can be defined at a node granularity . For each node , we computed these features and analyzed their correlations with the average classification accuracy for that node across test-train and initialization splits . Subsequently , we confirmed these results through a robust binning mechanism and a Mann-Whitney-U ( MWU ) test . We then analyzed the extreme case of completely disconnected components , and measured the impact of increasing number and size of disconnected components in the graph . For each dataset , we considered subgraphs with different types and number of components being retained as the input graph to a model : ( 1 ) only the largest connected component , ( 2 ) all components except isolated nodes , and ( 3 ) all components . Results : Surprisingly , the classification accuracy of node does not appear to depend strongly on its topological characteristics . As Fig . 1 shows , all topological features considered showed weak correlation ( |r| < 0.2 ) across both models and datasets with significance ( p < 0.05 ) , with most indicators of connectivity being mildly positive . Though all the datasets considered are relatively sparse ( D < 10−3 ) , ( Amazon Photograph and Computers ) show slightly better connectivity properties through long-tailed degree and coreness distributions . This increased connectivity did not affect the performance . These are supported by a visual analysis and MWU test which shows significance ( p < 0.001 ) in Coauthor Physics , Pubmed , and Coauthor Computers networks . In concordance with our intuition models that showed a positive mean-accuracy correlation with coreness , closeness and eccentricity also displayed moderate negative correlation ( 0.2 < |r| < 0.4 , p < 0.05 ) with the σ of the accuracy . We then drilled into the effect of extreme disconnectvity in the form of disconnected components . We chose 4 graphs that represent scenarios which represented a range multi-component properties , as seen in Table 2 . In all scenarios , GNNs ( except GCNs ) are able to handle graphs with multiple components better than Label Propagation baselines . We also observe that GNNs find multiplecomponent scenario presented in Citeseer the hardest to cope with ; on average , GNNs perform 20 % worse with the multi-component Citeseer . For the other three datasets , it may seem that the overall model accuracies change negligibly despite the addition of multiple components . However , the reported model accuracies are an average of test nodes ’ performance ; since most of the randomly selected test nodes are likely from the largest component , nodes located on the largest component would mask the poor node-level performance of other nodes . This is supported by Figure 2b , where we observe that nodes located in components other than the largest have a significantly worse performance than those located in the largest component . The sole exception is the Amazon comp . dataset , which shows a non-monotic irregular behavior amongst the smaller sized components . For GNNs across datasets , the average node accuracy on the largest component is 26.0 % higher than those on other components ; for LabelProp , this is even worse ( with a 51.6 % difference ) . Finally , given the discrepancies between the performance of GCNs and other models in handling multi-component graphs , we further analyzed the impact of the number of components in the graph on the performance of GCNs and GATs . In this experiment , for each graph , we input the subgraph retaining only the largest nc components , with nc ∈ { 1 , 2 , 10 , 100 , Nc−Ni , Nc } , Nc as total number of components and Ni as number of isolated nodes . Figure 2a reports the test accuracies achieved at these intervals . We find that as nc increases from 1 to Nc −Ni , there is only a slight decrease in accuracies suffered by GCNs and GATs which are within the statistical limit . Again , we see that there is only a small divergence in performance between GCNs and other GNNs as the graph includes isolated nodes . GATs , as well as other GCNNs , cope well with the presence of isolated nodes . A comprehensive comparison of the effect of isolated nodes can be seen in Table 5 . 4 JUST NOISY GRAPHS . Structural errors may originate during the acquisition of networks ( e.g . a malfunction in a cellular network ) or during the construction of networks from underlying data , particularly in dynamic networks where transient phenomenon are difficult to separate from noise . Thus , understanding the robustness of GNNs to errors of a topological nature is essential to applying them successfully in real world situations . Having established that GNNs do not strongly depend on topological features in Section 3 , it is reasonable to hypothesize that they should be robust , to all but specific , topological noise ( Zügner et al. , 2018 ) . We restrict our investigation to edge-based perturbations and measure the performance of GNNs on introducing specific controlled perturbations to the structure , allowing us to characterize the behavior of GNNs in noisy scenarios . Question : How does the addition of topological noise impact GNN performance ? Hypothesis/Expectation : As GNNs do not seem to depend on the topological features , they should remain fairly robust to noise and should continue to remain apathetic to topological features even in noisy circumstances . Counter-intuitive results : The addition of noise degrades performance significantly in some models and even more surprisingly , the GNN performance becomes more dependent on the global topological characteristics in noisy graphs . Methods : We constructed several perturbed versions of the Cora , Citeseer , Pubmed and Coauthor Physics graph datasets for this experiment - these span and represent different categories of connectivity and annotation ( attribute stack ) of networks . Cora has no isolated nodes and is one the best connected graphs ( Table 1 ) . Citeseer contrasts sharply in that it has the largest number of components.Pubmed and Coauthor Physics have the smallest and largest attribute stacks respectively . The perturbation method is derived from an Erdos-Renyi graph-based noise model , which can be shown to form the equivalent Gaussian noise in graphs . This allows use the equivalent of the central limit theorem in graphs ( Miettinen et al. , 2018 ) . The details of the algorithm and parameters can be seen in Appendix E. It perturbs an edge with probability P , and thresholds the resulting edges with respect to a parameter t. We picked a low P = 0.001 to stay true to the original graph and considered 5 perturbed versions using t ∈ { 0.3 , 0.45 , 0.6 , 0.75 , 0.9 } in our analysis . We further analyzed the behavior of GNNs in the case of smaller perturbations ( t ≥ 0.6 ) . As the perturbations are global rather than local , we used correlation analysis on the statistical characteristic of the global performance across graphs . Results : The noise model produced graph with comparable topological characteristics ( Table 4 ) which diverged from the original with increasing noise . Despite the topological similarity the addition of edge based noise degrades performance ( Figure 3a ) . All GNNs handled noise better than LabelProp methods . However all models , even GNNs , degrade considerably as the number of edges added exceeds its original edge count ; an exception is the performance of GraphSAGE ( mean and meanpool ) model on Coauthor Physics , which only saw an accuracy drop of 0.0006 at t = 0.3 , where more than 550,000 edges had been added ( i.e . adding about 2× the original number of edges ) . This tolerance to the addition of edges might be related to the high sparsity of the original Coauthor Physics graph , which has an edge density of 0.0005 . To analyze the how the topology effected GNNs in noisy environments . We considered the original and only high threshold ( least noise ) perturbed versions ( t ∈ { 0.6 , 0.75 , 0.9 } ) of 4 graph datasets . For each graph , we used the largest connected component as input graph to all GNN and LabelProp models . Figure 3b summarizes the significant correlations found between graph topological features and the average test accuracy achieved across all datasets per model . In stark contrast to the original graphs , there is a significant and strong positive correlation ( mean r = 0.74 , p = 0.006 ) with features which indicate better connectivity such as number of triangles , clustering ( max r = 0.766 ) , transitivity ( max r = 0.578 ) , coreness ( max r = 0.804 ) , and degree . There is also a significant negative correlation ( mean r = −0.54 , p = 0.03 ) with the minimum values of connectivity metrics . Correlations between these topological features with the standard deviation of accuracy across different train/validation/test splits also indicate that a higher transitivity ( r = −0.25 ) and clustering ( r = −0.22 ) correspond to a more consistent accuracy across splits . This clearly indicates that , in the presence of noise , the performance of GNNs is more related to the topological features as compared to baseline LabelProp methods where no significant correlations are found . The uniform polarity of the correlations suggests that these correlations are consistent across different models and are related to the same changes in performance ( albeit to varying magnitudes ) . Similar correlation patterns for validation F1-scores support this conclusion .

Summary: This paper analyzes the properties of graph neural networks (GNNs). It shows that several hypothesis that one might intuitively make about the behavior of GNNs, do not actually hold. In fact, some observations are even contradictory, indicating that GNNs' performance is not robust, and care needs to be taken when using them. In particular, the authors analyze what happens when topology is altered by computing correlations between topology metrics and accuracy, dropping connections, adding extraneous connections, etc. The authors examine performance is a function of graph connectedness, and find a weak correlation.