Title: A Metadata-Driven Approach to Understand Graph Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Networks

Metadata-Driven Analysis

Gini Coefficient of Degree Distribution
Abstract: Graph Neural Networks (GNNs) have achieved remarkable success in various applications, but their performance can be sensitive to specific data properties of the graph datasets they operate on. Current literature on understanding the limitations of GNNs has primarily employed a \emph{model-driven} approach that leverage heuristics and domain knowledge from network science or graph theory to model the GNN behaviors, which is time-consuming and highly subjective. In this work, we propose a \emph{metadata-driven} approach to analyze the sensitivity of GNNs to graph data properties, motivated by the increasing availability of graph learning benchmarks. We perform a multivariate sparse regression analysis on the metadata derived from benchmarking GNN performance across diverse datasets, yielding a set of salient data properties. To validate the effectiveness of our data-driven approach, we focus on one identified data property, the degree distribution, and investigate how this property influences GNN performance through theoretical analysis and controlled experiments. Our theoretical findings reveal that datasets with more balanced degree distribution exhibit better linear separability of node representations, thus leading to better GNN performance. We also conduct controlled experiments using synthetic datasets with varying degree distributions, and the results align well with our theoretical findings. Collectively, both the theoretical analysis and controlled experiments verify that the proposed metadata-driven approach is effective in identifying critical data properties for GNNs.

A Metadata-Driven Approach to Understand Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) have achieved remarkable success in various applications, but their performance can be sensitive to specific data properties of the graph datasets they operate on. Current literature on understanding the limitations of GNNs has primarily employed a model-driven approach that leverages heuristics and domain knowledge from network science or graph theory to model the GNN behaviors, which is time-consuming and highly subjective. In this work, we propose a metadata-driven approach to analyze the sensitivity of GNNs to graph data properties, motivated by the increasing availability of graph learning benchmarks. We perform a multivariate sparse regression analysis on the metadata derived from benchmarking GNN performance across diverse datasets, yielding a set of salient data properties. To validate the effectiveness of our data-driven approach, we focus on one identified data property, the degree distribution, and investigate how this property influences GNN performance through theoretical analysis and controlled experiments. Our theoretical findings reveal that datasets with a more balanced degree distribution exhibit better linear separability of node representations, thus leading to better GNN performance. We also conduct controlled experiments using synthetic datasets with varying degree distributions, and the results align well with our theoretical findings. Collectively, both the theoretical analysis and controlled experiments verify that the proposed metadata-driven approach is effective in identifying critical data properties for GNNs.

1 Introduction

Graph Neural Networks (GNNs), as a broad family of graph machine learning models, have gained increasing research interests in recent years. However, unlike the ResNet model [14] in computer vision or the Transformer model [36] in natural language processing, there has not been a dominant GNN architecture that is universally effective across a wide range of graph machine learning tasks. This may be attributed to the inherently diverse nature of graph-structured data, which results in the GNN performance being highly sensitive to specific properties of the graph datasets. Consequently, GNNs that demonstrate high performance on certain benchmark datasets often underperform on others with distinct properties. For example, early GNNs have been shown to exhibit degraded performance when applied to non-homophilous graph datasets, where nodes from different classes are highly interconnected and mixed [45, 46, 32, 11, 9].

However, it is non-trivial to identify and understand critical graph data properties that are highly influential on GNN performance. Current literature primarily employs what we term as a modeldriven approach, which attempts to model GNN performance using specific heuristics or domain knowledge derived from network science or graph theory [41, 45]. Although this approach can offer an in-depth understanding of GNN performance, it can also be time-consuming, subjective, and may not fully capture the entire spectrum of relevant data properties.

To address these limitations and complement the model-driven approach, we propose a metadatadriven approach to identify critical data properties affecting GNN performance. With the increasing availability of diverse benchmark datasets for graph machine learning [16, 27], we hypothesize that critical graph data properties can be inferred from the benchmarking performance of GNNs on these datasets, which can be viewed as the metadata of the datasets. More concretely, we carry out a multivariate sparse regression analysis on the metadata obtained from large-scale benchmark experiments [27] involving multiple GNN models and a variety of graph datasets. Through this regression analysis, we examine the correlation between GNN performance and the data properties of each dataset, thereby identifying a set of salient data properties that significantly influence GNN performance.

To validate the effectiveness of the proposed metadata-driven approach, we further focus on a specific salient data property, degree distribution, identified from the regression analysis, and investigate the mechanism by which this data property affects GNN performance. In particular, our regression analysis reveals a decline in GNN performance as the degree distribution becomes more imbalanced. We delve deeper into this phenomenon through a theoretical analysis and a controlled experiment.

We initiate our investigation with a theoretical analysis of the GNN performance under the assumption that the graph data is generated by a Degree-Corrected Contextual Stochastic Block Model (DCCSBM). Here, we define DC-CSBM by combining and generalizing the Contextual Stochastic Block Model [4] and the Degree-Corrected Stochastic Block Model [17]. Building upon the analysis by Baranwal et al. [3], we establish a novel theoretical result on how the degree distribution impacts the linear separability of the GNN representations and subsequently, the GNN performance. Within the DC-CSBM context, our theory suggests that more imbalanced degree distribution leads to few nodes being linearly separable in their GNN representations, thus negatively impacting GNN performance.

Complementing our theoretical analysis, we conduct a controlled experiment, evaluating GNN performance on synthetic graph datasets with varying degree distribution while holding other properties fixed. Remarkably, we observe a consistent decline in GNN performance correlating with the increase of the Gini coefficient of degree distribution, which reflects the imbalance of degree distribution. This observation further corroborates the findings of our metadata-driven regression analysis.

In summary, our contribution in this paper is two-fold. Firstly, we introduce a novel metadata-driven approach to identify critical graph data properties affecting GNN performance and demonstrate its effectiveness through a case study on a specific salient data property identified by our approach. Secondly, we develop an in-depth understanding of how the degree distribution of graph data influences GNN performance through both a novel theoretical analysis and a carefully controlled experiment, which is of interest to the graph machine learning community in its own right.

2 Related Work

2.1 Analysis on the Limitations of GNNs

There has been a wealth of existing literature investigating the limitations of GNNs. However, most of the previous works employ the model-driven approach. Below we summarize a few wellknown limitations of GNNs while acknowledging that an exhaustive review of the literature is impractical. Among the limitations identified, GNNs have been shown to be sensitive to the extent of homophily in graph data, and applying GNNs to non-homophilous data often has degraded performance [1, 9, 23, 46, 45]. In addition, over-smoothing, a phenomenon where GNNs lose their discriminative power with deeper layers [20, 34, 6], is a primary concern particularly for node-level prediction tasks where distinguishing the nodes within the graph is critical. Further, when applied to graph-level prediction tasks, GNNs are limited by their ability to represent and model specific functions or patterns on graph-structured data, an issue often referred to as the expressiveness problem of GNNs. [41, 30, 25, 43]. Most of these limitations are understood through a model-driven approach, which offers in-depth insights but is time-consuming and highly subjective. In contrast, this paper presents a metadata-driven approach, leveraging metadata from benchmark datasets to efficiently screen through a vast array of data properties.

2.2 Data-Driven Analysis in Graph Machine Learning

With the increasing availability of graph learning benchmarks, there have been several recent studies that leverage diverse benchmarks for data-driven analysis. For example, Liu et al. [24] presents a principled pipeline to taxonomize benchmark datasets. Specifically, by applying a number of different perturbation methods on each dataset and obtaining the sensitivity profile of the resulting GNN performance on perturbed datasets, they perform hierarchical clustering on these sensitivity profiles to cluster statistically similar datasets. However, this study only aims to categorize datasets instead of identifying salient data properties that influence GNN performance. Ma et al. [27] establish a Graph Learning Indexer (GLI) library that curates a large collection of graph learning benchmarks and GNN models and conducts a large-scale benchmark study. We obtain our metadata from their benchmarks. Palowitch et al. [31] introduce a GraphWorld library that can generate diverse synthetic graph datasets with various properties. These synthetic datasets can be used to test GNN models through controlled experiments. In this paper, we have used this library to verify the effectiveness of the identified critical data properties.

2.3 Impact of Node Degrees on GNN Performance

There have also been a few studies investigating the impact of node degrees on GNNs. In particular, it has been observed that within a single graph dataset, there tends to be an accuracy discrepancy among nodes with varying degrees [35, 22, 44, 39]. Typically, GNN predictions on nodes with lower degrees tend to have lower accuracy. However, the finding of the Gini coefficient of the degree distribution as a strong indicator of GNN performance is novel. Furthermore, this indicator describes the dataset-level characteristics, allowing comparing GNN performance across different graph datasets. In addition, this paper presents a novel theoretical analysis, directly relating the degree distribution to the generalization performance of GNNs.

3 A Metadata-Driven Analysis on GNNs

3.1 Understanding GNNs with Metadata

Motivation. Real-world graph data are heterogeneous and incredibly diverse, contrasting with images or texts that often possess common structures or vocabularies. The inherent diversity of graph data makes it particularly challenging, if not unfeasible, to have one model to rule all tasks and datasets in the graph machine learning domain. Indeed, specific types of GNN models often only perform well on a selected set of graph learning datasets. For example, the expressive power of GNNs [41] is primarily relevant to graph-level prediction tasks rather than node-level tasks higher-order GNNs with improved expressive power are predominantly evaluated on graph-level prediction tasks [30, 41]. As another example, several early GNNs such as Graph Convolution Networks (GCN) [19] or Graph Attention Networks (GAT) [37] only work well when the graphs exhibit homophily [45]. Consequently, it becomes crucial to identify and understand the critical data properties that influence the performance of different GNNs, allowing for more effective model design and selection.

The increasing availability of graph learning benchmarks that offer a wide range of structural and feature variations [16, 27] presents a valuable opportunity: one can possibly infer critical data properties from the performance of GNNs on these datasets. To systematically identify these critical data properties, we propose to conduct a regression analysis on the metadata of the benchmarks.

Regression Analysis on Metadata. In the regression analysis, the performance metrics of various GNN models on each dataset serve as the dependent variables, while the extracted data properties from each dataset act as the independent variables. Formally, we denote the number of datasets as n , the number of GNN models as q , and the number of data properties as p . Define the response variables { y i } i ∈ [ q ] to be GNN model performance operated on each dataset and the covariate variables { x j } j ∈ [ p ] to be properties of each dataset. Note that y i ∈ R n , ∀ i ∈ [ q ] and x j ∈ R n , ∀ j ∈ [ p ] . For ease of notation, we define Y = ( y 1 , ..., y q ) ∈ R n × q to be the response matrix of n samples and q variables, and X = ( x 1 , ..., x p ) ∈ R n × p to be the covariate matrix of n samples and p variables.

Given these data matrices, we establish the following multivariate linear model to analyze the relationship between response matrix Y and covariate matrix X , which is characterized by the coefficient matrix B .

Definition 3.1 (Multivariate Linear Model) .

where B ∈ R p × q is the coefficient matrix and W = ( w 1 , ..., w q ) ∈ R n × q is the matrix of error terms.

Our goal is to find the most salient data properties that correlate with the performance of GNN models given a number of samples. To this end, we introduce two sparse regularizers for feature selections, which leads to the following Multivariate Sparse Group Lasso problem.

Definition 3.2 (Multivariate Sparse Group Lasso Problem) .

where ∥ B ∥ 1 = ∑ p i =1 ∑ q j =1 | B ij | is the L 1 norm of B , ∥ B ∥ 2 , 1 = ∑ p i =1 √ ∑ q j =1 B 2 ij is the L 2 , 1 group norm of B , and λ 1 , λ g > 0 are the corresponding penalty parameters.

In particular, the L 1 penalty encourages the coefficient matrix B to be sparse, only selecting salient data properties. The L 2 , 1 penalty further leverages the structure of the dependent variables and tries to make only a small set of the GNN models' performance depends on each data property, thus differentiating the impacts on different GNNs.

To solve for the coefficient matrix B in Equation 2, we employ an R package, MSGLasso [21], using matrices Y and X as input. To ensure proper input for the MSGLasso solver [21], we have preprocessed the data by standardizing the columns of both Y and X .

3.2 Data Properties and Model Performance

Next, we introduce the metadata used for the regression analysis. We obtain both the benchmark datasets and the model performance using the Graph Learning Indexer (GLI) library [27].

Data Properties. Weinclude the following benchmark datasets in our regression analysis: cora [42], citeseer [42], pubmed [42], texas [33], cornell [33], wisconsin [33], actor [33], squirrel [33], chameleon [33], arxiv-year [23], snap-patents [23], penn94 [23], pokec [23], genius [23], and twitch-gamers [23]. For each graph dataset, we calculate 15 data properties, which can be categorized into the following six groups:

Basic : Edge Density, Average Degree, Degree Assortativity;

Distance : Pseudo Diameter;

Connectivity : Relative Size of Largest Connected Component (RSLCC);

Clustering : Average Clustering Coefficient (ACC), Transitivity, Degeneracy;

Degree Distribution : Gini Coefficient of Degree Distribution (Gini-Degree);

Attribute : Edge Homogeneity, In-Feature Similarity, Out-Feature Similarity, Feature Angular SNR, Homophily Measure, Attribute Assortativity.

The formal definition of these graph properties can be found in Appendix A.

Model Performance. For GNN models, we include GCN [19], GAT [37], GraphSAGE [13], MoNet [29], MixHop [1], and LINKX [23] into our regression analysis. We also include a non-graph model, Multi-Layer Perceptron (MLP). The complete experimental setup for these models can be found in Appendix B.

3.3 Analysis Results

The estimated coefficient matrix B is presented in Table 1. As can be seen, the estimated coefficient matrix is fairly sparse, allowing us to identify salient data properties. Next, we will discuss the six most salient data properties that correlate to some or all of the GNN models' performance. For the data properties that have an impact on all GNNs' performance, we call them Widely Influential Factors ; for the data properties that have an impact on over one-half of GNNs' performance, we call them Narrowly Influential Factors . Notice that the (+ , -) sign after the name of the factors indicates whether this data property has a positive or negative correlation with the GNN performance.

Widely Influential Factors. We discover that the Gini coefficient of the degree distribution (GiniDegree), Edge Homogeneity, and In-Feature Similarity impact all GNNs' model performance consistently.

Gini-Degree ( -) measures how the graph's degree distribution deviates from the perfectly equal distribution, i.e., a regular graph. This is a crucial data property that dramatically influences GNNs' performance but remains under-explored in prior literature.

Edge Homogeneity (+) is a salient indicator for all GNN models' performance. This phenomenon coincides with the fact that various GNNs assume strong homophily condition [28] to obtain improvements on node classification tasks [13, 19, 37].

In-feature Similarity (+) calculates the average of feature similarity within each class. Under the homophily assumption, GNNs work better when nodes with the same labels additionally have similar node features, which also aligns with existing findings in the literature [15].

Narrowly Influential Factors. Wefind that Average Degree, Pseudo Diameter, and Feature Angular SNRare salient factors for a subset of GNN models, although we do not yet have a good understanding on the mechanism of how these data properties impact model performance.

Average Degree (+) is more significant for GCN, GraphSAGE, MoNet, and LINKX.

Pseudo Diameter ( -) is more significant for GAT, GraphSAGE, MixHop, LINKX, and MLP.

Feature Angular SNR (+) is more significant for GCN, GraphSAGE, MixHop, LINKX, and MLP.

We note that the regression analysis only indicates associative relationships between data properties and the model performance. While our analysis has successfully identified well-known influential

data properties, e.g., Edge Homogeneity, the mechanism for most identified data properties through which they impact the GNN performance remains under-explored.

To further verify the effectiveness of the proposed metadata-driven approach in identifying critical data properties, we perform an in-depth analysis for Gini-Degree , one of the most widely influential factors. In the following Section 4 and 5, we conduct theoretical analysis and controlled experiments to understand how Gini-Degree influences GNNs' performance.

4 Theoretical Analysis on the Impact of Degree Distribution

In this section, we present a theoretical analysis on influence of graph data's degree distribution on the performance of GNNs. Specifically, our analysis investigates the linear separability of node representations produced by applying graph convolution to the node features. In the case that the graph data comes from a Degree-Corrected Stochastic Block Model, we show that nodes from different classes are more separable when their degree exceeds a threshold. This separability result relates the graph data's degree distribution to the GNN performance. Finally, we discuss the role of Gini-Degree on the GNN performance using implications of our theory.

4.1 Notations and Sketch of Analysis

The Graph Data. Let G = {V , E} be an undirected graph, where V is the set of nodes and E is the set of edges. The information regarding the connections within the graph can also be summarized as an adjacency matrix A ∈ { 0 , 1 } |V|×|V| , where |V| is the number of nodes in the graph G . Each node i ∈ V possesses a d -dimensional feature vector x i ∈ R d . The features for all nodes in G can be stacked and represented as a feature matrix X ∈ R |V|× d . In the context of node classification, each node i is associated with a class label y i ∈ C , where C is the set of labels.

Graph Convolutional Network [19]. In our analysis, we consider a single-layer graph convolution, which can be defined as an operation on the adjacency matrix and feature matrix of a graph G to produce a new feature matrix ˜ X . Formally, the output of a single-layer graph convolution operation can be represented as ˜ X = D -1 ˜ AX , where ˜ A = A + I is the augmented adjacency matrix with added self-loops, and D is the diagonal degree matrix with D ii = deg( i ) = ∑ j ∈ [ n ] ˜ A ij . Hence, for each node i ∈ V , the new node representation will become ˜ x i ∈ R d , which is the i th row of the output matrix ˜ X .

Sketch of Our Analysis. Our analysis builds upon and generalizes the theoretical framework introduced by Baranwal et al. [3], where they demonstrate that in comparison to raw node features, the graph convolution representations of nodes have better linear separability if the graph data comes from Contextual Stochastic Block Model (CSBM) [4, 8]. However, in CSBM, the nodes within the same class all have similar degrees with high probability, which prevents us to draw meaningful conclusions about the impact of degree distribution.

To better understand the role of degree distribution in the GNN performance, we develop a non-trivial generalization of the theory by Baranwal et al. [3]. Specifically, we first coin a new graph data generation model, Degree-Corrected Contextual Stochastic Block Model (DC-CSBM) that combines and generalizes Degree-Corrected SBM (DC-SBM) [17] and CSBM, and leverages heterogeneity in node degrees into consideration. Under DC-CSBM, we find that node degrees play a crucial role in the statistical properties of the node representations, and the node degrees have to exceed a certain threshold in order for the node representations to sufficiently leverage the neighborhood information and become reliably separable. Notably, the incorporation of the node degree heterogeneity into the analysis requires a non-trivial adaptation of the analysis by Baranwal et al. [3].

4.2 Degree-Corrected Contextual Stochastic Block Model (DC-CSBM)

In this section, we introduce the DC-CSBM that models the generation of graph data. Specifically, we assume the graph data is randomly sampled from a DC-CSBM with 2 classes.

DC-CSBM With 2 Classes. Let us define the class assignments ( ϵ i ) i ∈ [ n ] as independent and identically distributed (i.i.d.) Bernoulli random variables coming from Ber ( 1 2 ) , where n = |V| is the number of nodes in the graph G . These class assignments divide n nodes into 2 classes: C 0 = { i ∈ [ n ] : ϵ i = 0 } and C 1 = { i ∈ [ n ] : ϵ i = 1 } . Assume that inter-class edge probability is q and intra-class edge probability is p , and no self-loops are allowed. For each node i , we additionally introduce a degree-correction parameter θ i ∈ (0 , n ] , which can be interpreted as the propensity of node i to connect with others. Note that to keep the DC-SBM identifiable and easier to analyze, we adopt a normalization rule to enforce the following constraint: ∑ i ∈ C 0 θ i = | C 0 | , ∑ i ∈ C 1 θ i = | C 1 | and thus ∑ i ∈V θ i = n .

Assumptions on Adjacency Matrix and Feature Matrix. Conditioning on ( ϵ i ) i ∈ [ n ] , each entry of the adjacency matrix A is a Poisson random variable with A ij ∼ Poi ( θ i θ j p ) if i, j are in the same class and A ij ∼ Poi ( θ i θ j q ) if i, j are in different classes. On top of this, let X ∈ R n × d be the feature matrix where each row x i represents the node feature of node i . Assume each x i is an independent d -dimensional Gaussian random vector with x i ∼ N ( µ , 1 d I ) if i ∈ C 0 and x i ∼ N ( ν , 1 d I ) if i ∈ C 1 . We let µ , ν ∈ R d to be fixed d -dimensional vectors with ∥ µ ∥ 2 , ∥ ν ∥ 2 ≤ 1 , which serve as the Gaussian mean for the two classes.

Given a particular choice of n, µ , ν , p, q and θ = ( θ i ) i ∈ [ n ] , we can define a class of random graphs generated by these parameters and sample a graph from such DC-CSBM as G = ( A , X ) ∼ DCCSBM ( n, µ , ν , p, q, θ ) .

4.3 Linear Separability After Graph Convolution

Linear Separability. Linear separability refers to the ability to linearly differentiate nodes in the two classes based on their feature vectors. Formally, for any V s ⊆ V , we say that { ˜ x i : i ∈ V s } is linearly separable if there exists some unit vector v ∈ R d and a scalar b such that v ⊤ ˜ x i + b < 0 , ∀ i ∈ C 0 ∩V s and v ⊤ ˜ x i + b > 0 , ∀ i ∈ C 1 ∩V s . Note that linear separability is closely related to GNN performance. Intuitively, more nodes being linearly separable will lead to better GNN performance.

Degree-Thresholded Subgroups of C 0 and C 1 . To better control the behavior of graph convolution operation, we will focus on particular subgroups of C 0 and C 1 where the member nodes having degree-corrected factor larger or equal to a pre-defined threshold α > 0 . Slightly abusing the notations, we denote these subgroups as C 0 ( α ) and C 1 ( α ) , which are formally defined below.

Definition 4.1 ( α -Subgroups) . Given any α ∈ (0 , n ] , define α -subgroups of C 0 and C 1 as follows:

Let V α := C 0 ( α ) ∪ C 1 ( α ) , we are interested in analyzing the linear separability of the node representations after the graph convolution operation, namely { ˜ x i : i ∈ V α } . Recall that for each node i , ˜ x i = 1 deg ( i ) ∑ j ∈N ( i ) x j , where N ( i ) is the set of neighbors of node i .

Relationship Between α and Linear Separability. We first make the following assumptions about the DC-CSBM, closely following the assumptions made by Baranwal et al. [3].

Assumption 4.2 (Graph Size) . Assume the relationship between the graph size n and the feature dimension d follows ω ( d log d ) ≤ n ≤ O ( poly ( d )) .

Assumption 4.3 (Edge Probabilities) . Define Γ( p, q ) := p -q p + q . Assume the edge probabilities p, q satisfy p, q = ω (log 2 ( n ) /n ) and Γ( p, q ) = Ω(1) .

Theorem 4.4 asserts that if the threshold α is not too small, then the set V α = C 0 ( α ) ∪ C 1 ( α ) can be linear separated with high probability. The proof of Theorem 4.4 can be found in Appendix C.

Theorem 4.4 (Linear Separability of α -Subgroups) . Suppose that Assumption 4.2 and 4.3 hold. For any ( X , A ) ∼ DC-CSBM( n, µ , ν , p, q, θ ), if α = ω ( max ( 1 log n , log n dn ( p + q ) ∥ µ -ν ∥ 2 2 )) , then

where o d (1) is a quantity that converges to 0 as d approaches infinity.

Note that Theorem 4.4 suggests that, when the heterogeneity of node degrees is taken into consideration, the nodes with degrees exceeding a threshold α are more likely to be linearly separable. And the requirement for the threshold α depends on the DC-CSBM parameters: n, p, q, µ , ν .

Remark 4.5. If we let p, q ∈ Θ( log 3 n n ) and ∥ µ -ν ∥ 2 be fixed constant, then the requirement can be reduced to α ∈ ω ( 1 log n ) , which is not too large. Given this particular setting and reasonable selection of p, q , the regime of acceptable α is broad and thus demonstrates the generalizability of Theorem 4.4.

4.4 Implications on Gini-Degree

Finally, we qualitatively discuss the relationship between Gini-Degree and GNNs' performance using the results from Theorem 4.4. For any α > 0 that meets the criteria in the statement, we can consider,

Negative correlation between Gini-Degree and the size of V α : If the number of nodes and edges is fixed, a higher Gini-Degree implies more high-degree nodes in the network and thus the majority of nodes are receiving lower degrees. Clearly, if most of the nodes have lower degrees, then there will be fewer nodes having degrees exceeding a certain threshold proportional to α 1 and being placed in V α . Hence, a dataset with a higher (or lower) Gini-Degree will lead to a smaller (or larger) size of V α .

Positive correlation between the size of V α and model performance : Intuitively, the GNN performance tends to be better if there are more nodes that can be linearly separable after graph convolution. Consequently, the GNN performance is positively relevant to the size of V α corresponding to the minimum possible α .

Combining the two factors above, our analysis suggests that Gini-Degree tends to have a negative correlation with GNNs' performance.

5 Controlled Experiment on Gini-Degree

To further verify whether there is a causal relationship between the degree distribution of graph data (in particular, measured by Gini-Degree) and the GNN performance, we conduct a controlled experiment using synthetic graph datasets.

Experiment Setup. We first generate a series of synthetic graph datasets using the GraphWorld library [31]. To investigate the causal effect of Gini-Degree, we manipulate the data generation parameters to obtain datasets with varying Gini-Degree while keeping a bunch of other properties fixed. Specifically, we use the SBM generator in GraphWorld library and set the number of nodes n = 5000 , the average degree as 30 , the number of clusters as 4 , cluster size slope as 0 . 5 , feature center distance as 0 . 5 , the edge probability ratio p/q = 4 . 0 , feature dimension as 16 , feature cluster variance as 0 . 05 . The parameters above are fixed throughout our experiments, and their complete definition can be found in the Appendix. By manipulating the power law exponent parameter of the generator, we obtain five synthetic datasets with Gini-Degree as 0 . 906 , 0 . 761 , 0 . 526 , 0 . 354 , and 0 . 075 , respectively.

Then we train the same set of GNN models and MLP model as mentioned in Table 1 on each dataset. We randomly split the nodes into training, validation, and test sets with a ratio of 3:1:1. We closely follow the hyperparameters and the training protocol in the GLI library [27], which is where we obtain the metadata in Section 3. We run five independent trials with different random seeds.

Experiment Results. The experiment results are shown in Table 2. We observe an evident monotonically decreasing trend for the performance of the graph-based models, GCN, GAT, GraphSAGE, MoNet, MixHop, and LINKX, as Gini-Degree increases. However, there is no clear pattern for the non-graph model, MLP. This result suggests that these widely-used GNN models are indeed sensitive to Gini-Degree, which validates our result of sparse regression analysis. Note that MLP does not take the graph structure into consideration, and hence the degree distribution has less influence

on the performance of MLP. The result on MLP also indicates that we have done a reasonably well-controlled experiment.

6 Conclusion

In this work, we propose a novel metadata-driven approach that can efficiently identify critical graph data properties influencing the performance of GNNs. This is a significant contribution given the diverse nature of graph-structured data and the sensitivity of GNN performance to these specific properties. We also verify the effectiveness of the proposed approach through an in-depth case study around one identified salient graph data property.

As a side product, this paper also highlights the considerable impact of the degree distribution, a salient data property identified through our metadata-driven regression analysis, on the GNN performance. We present a novel theoretical analysis and a carefully controlled experiment to demonstrate this impact.

References

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning , pages 21-29. PMLR, 2019.

Robert J Adler, Jonathan E Taylor, et al. Random fields and geometry , volume 80. Springer, 2007.

Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. Graph convolution for semisupervised classification: Improved linear separability and out-of-distribution generalization. arXiv preprint arXiv:2102.06966 , 2021.

Norbert Binkiewicz, Joshua T Vogelstein, and Karl Rohe. Covariate-assisted spectral clustering. Biometrika , 104(2):361-377, 2017.

Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491 , 2021.

Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318 , 2020.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning , pages 1725-1735. PMLR, 2020.

Yash Deshpande, Subhabrata Sen, Andrea Montanari, and Elchanan Mossel. Contextual stochastic block models. Advances in Neural Information Processing Systems , 31, 2018.

Yingtong Dou, Zhiwei Liu, Li Sun, Yutong Deng, Hao Peng, and Philip S Yu. Enhancing graph neural network-based fraud detectors against camouflaged fraudsters. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 315-324, 2020.

Jian Du, Shanghang Zhang, Guanhang Wu, José MF Moura, and Soummya Kar. Topology adaptive graph convolutional networks. arXiv preprint arXiv:1710.10370 , 2017.

Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. Protein interface prediction using graph convolutional networks. Advances in neural information processing systems , 30, 2017.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997 , 2018.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Yifan Hou, Jian Zhang, James Cheng, Kaili Ma, Richard TB Ma, Hongzhi Chen, and MingChang Yang. Measuring and improving the use of graph information in graph neural networks. In International Conference on Learning Representations , 2019.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Brian Karrer and Mark EJ Newman. Stochastic blockmodels and community structure in networks. Physical review E , 83(1):016107, 2011.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Yanming Li, Bin Nan, and Ji Zhu. Multivariate sparse group lasso for the multivariate multiple linear regression with an arbitrary group structure. Biometrics , 71(2):354-363, 2015.

Langzhang Liang, Zenglin Xu, Zixing Song, Irwin King, and Jieping Ye. Resnorm: Tackling long-tailed degree distribution issue in graph neural networks via normalization. arXiv preprint arXiv:2206.08181 , 2022.

Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. Advances in Neural Information Processing Systems , 34:20887-20902, 2021.

Renming Liu, Semih Cantürk, Frederik Wenkel, Sarah McGuire, Xinyi Wang, Anna Little, Leslie O'Bray, Michael Perlmutter, Bastian Rieck, Matthew Hirn, et al. Taxonomy of benchmarks in graph representation learning. In Learning on Graphs Conference , pages 6-1. PMLR, 2022.

Xiao Liu, Lijun Zhang, and Hui Guan. Uplifting message passing neural network with graph original information, 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Jiaqi Ma, Xingjian Zhang, Hezheng Fan, Jin Huang, Tianyue Li, Ting Wei Li, Yiwen Tu, Chenshu Zhu, and Qiaozhu Mei. Graph learning indexer: A contributor-friendly and metadatarich platform for graph learning benchmarks. arXiv preprint arXiv:2212.04537 , 2022.

Miller McPherson, Lynn Smith-Lovin, and James M Cook. Birds of a feather: Homophily in social networks. Annual review of sociology , 27(1):415-444, 2001.

Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5115-5124, 2017.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 4602-4609, 2019.

John Palowitch, Anton Tsitsulin, Brandon Mayer, and Bryan Perozzi. Graphworld: Fake graphs bring real insights for gnns. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 3691-3701, 2022.

Shashank Pandit, Duen Horng Chau, Samuel Wang, and Christos Faloutsos. Netprobe: a fast and scalable system for fraud detection in online auction networks. In Proceedings of the 16th international conference on World Wide Web , pages 201-210, 2007.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. arXiv preprint arXiv:2002.05287 , 2020.

T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303.10993 , 2023.

Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Yiqi Wang, Jiliang Tang, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. Investigating and mitigating degree-related biases in graph convoltuional networks. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 1435-1444, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Quanmin Wei, Jinyan Wang, Xingcheng Fu, Jun Hu, and Xianxian Li. Aic-gnn: Adversarial information completion for graph neural networks. Information Sciences , 2023.

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning , pages 6861-6871. PMLR, 2019.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 10737-10745, 2021.

Sukwon Yun, Kibum Kim, Kanghoon Yoon, and Chanyoung Park. Lte4g: Long-tail experts for graph neural networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 2434-2443, 2022.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems , 33:7793-7804, 2020.

Jiong Zhu, Ryan A Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K Ahmed, and Danai Koutra. Graph neural networks with heterophily. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11168-11176, 2021.

A Definitions of Dataset Properties

We introduce the formal definitions of the dataset properties mentioned in Section 3.2. Following the definitions in Section 4.1, we further define n = |V| and m = |E| to denote the number of nodes and edges of graph G . Also, in the context of the node classification task, we define Y ∈ R n as the vector of node labels and C as the number of classes.

A.1 Basic

Edge Density: The edge density for an undirected graph is calculated as 2 m n ( n -1) , while for a directed graph, it is computed as m n ( n -1) .

Average Degree: The average degree for an undirected graph is defined as 2 m n , while for a directed graph, it is defined as m n .

Degree Assortativity: The degree assortativity is the average Pearson correlation coefficient of all pairs of connected nodes. It quantifies the tendency of nodes in a network to be connected to nodes with similar or dissimilar degrees and ranges between -1 and 1.

A.2 Distance

Pseudo Diameter: The pseudo diameter is an approximation of the diameter of a graph and provides a lower bound estimation of its exact value.

A.3 Connectivity

Relative Size of Largest Connected Component (RSLCC): The relative size of the largest connected component is determined by calculating the ratio between the size of the largest connected component and n .

A.4 Clustering

Average Clustering Coefficient (ACC): First define T ( u ) as the number of triangles including node u , then the local clustering coefficient for node u is calculated as 2 deg( u )(deg( u ) -1) T ( u ) for undirected graph, where deg( u ) is the degree of node u ; and is calculated as 2 deg tot ( u )(deg tot ( u ) -1) -2 deg ↔ ( u ) T ( u ) for directed graph, where deg tot ( u ) is the sum of in-degree and out-degree of node u and deg ↔ ( u ) is the reciprocal degree of u . The average clustering coefficient is then defined as the average local clustering coefficient of all the nodes in the graph.

Transitivity: The transitivity is defined as the fraction of all possible triangles present in the graph. Formally, it can be written as 3 #triangles #triads , where a triad is a pair of two edges with a shared vertex.

Degeneracy: The degeneracy is determined as the least integer k such that every induced subgraph of the graph contains a vertex with its degree smaller or equal to k .

A.5 Degree Distribution

Gini Coefficient of Degree Distribution (Gini-Degree): The Gini coefficient of the node degrees of the graph.

A.6 Attribute

Edge Homogeneity [31]: The edge homogeneity is defined as the ratio of edges whose endpoints have the same node labels.

In-Feature Similarity [31]: First define within-class angular feature similarity as 1 -angular_distance ( x i , x j ) for an edge ( i, j ) with its endpoints i and j have the same node labels. In-Feature Similarity is the average within-class angular feature similarity of all such edges in the graph.

Out-Feature Similarity [31]: First define between-class angular feature similarity as 1 -angular_distance ( x i , x j ) for an edge ( i, j ) with its endpoints i and j have different node labels. Out-Feature Similarity is the average between-class angular feature similarity of all such edges in the graph.

Feature Angular SNR [31]: The feature angular SNR is computed as the ratio between in-feature similarity and out-feature similarity.

Homophily Measure [23]: The homophily measure is defined as

where [ a ] + = max( a, 0) , | C k | is the total number of nodes having their label k and h k is the class-wise homophily metric defined below,

where d u is the number of neighbors of node u and d ( Y u ) u is the number of neighbors of node u having the same node label.

Attribute Assortativity: The attribute assortativity is the average Pearson correlation coefficient of all pairs of connected nodes. It quantifies the tendency of nodes in a network to be connected to nodes with the same or different attributes (here node label) and ranges between -1 and 1.

B Experiment Setup for Obtaining Metadata

In this section, we describe more details of the experimental setup to obtain GNNs' performance that we use in Section 3.2, mostly following Ma et al. [27]. For completeness, we list down the model setting used by them in the following paragraphs.

GCN [19], GAT [37], GraphSAGE [13], MoNet [29], MLP, and MixHop [1] are set to have two layers with hidden dimension equals to 8. For LINKX [23], MLP A , MLP X are set to be a one-layer network and MLP f to be a two-layers network, following the setting in Lim et al. [23].

For the rest of the training settings, we adopt the same configuration for all experiments. Specifically, we set learning rate = 0.01, weight decay = 0.001, dropout rate = 0.6, max epoch = 10000, and batch size = 256. We use Adam [18] as an optimizer for all models except LINKX. AdamW [26] is used with LINKX in order to comply with Lim et al. [23]. For datasets with binary labels (i.e., penn94, pokec, genius, and twitch-gamers), we choose the ROC AUC score as the evaluation metric; while for other datasets, we use test accuracy instead.

We also let all the detailed model settings remain consistent with the same with Ma et al. [27]. Namely,

GAT: Number of heads in multi-head attention = 8. LeakyReLU angle of negative slope = 0.2. No residual is applied. The dropout rate on attention weight is the same as the overall dropout.

GraphSAGE: Aggregator type is GCN. No norm is applied.

MoNet: Number of kernels = 3. Dimension of pseudo-coordinte = 2. Aggregator type = sum.

MixHop: List of powers of adjacency matrix = [1 , 2 , 3] . No norm is applied. Layer Dropout rate = 0 . 9 .

LINKX: No inner activation.

C Proof of Theorem 4.4

Proof Sketch. To prove Theorem 4.4, we first show that degree and the neighborhood distribution of each node concentrate with high probability. Then we claim that the node features after the

convolution operation will be centered around specific mean values, depending on the node classes. Finally, we demonstrate that the nodes in different classes can be linearly separated by the hyperplane passing through the mid-point of the two mean values µ , ν with high probability.

We prove the intermediate results in Lemma C.4 (degree and neighborhood distribution concentration inequalities) by utilizing Lemma C.5 (Chernoff bound for Poisson random variable) and in Lemma C.6 (convoluted feature concentration) by making use of Lemma C.7 (Borell's inequality). Finally, given the requirement of α stated in Theorem 4.4, we argue that the convoluted node features in two classes can be linearly separated with a high level of confidence.

Novelty of our Proof. The general structure of our proof follows that of Baranwal et al. [3]. However, our analysis requires non-trivial adaptation of the proof by Baranwal et al. [3]. This is because we have a more general data model, DC-CSBM, where the CSBM assumed by Baranwal et al. [3] is a restricted special case of ours. In particular, we assume each edge is generated by the Poisson random variable following DC-SBM, instead of the Bernoulli random variable assumed by CSBM; we also incorporate the degree-corrected factor in our analysis to model node degree heterogeneity within communities, which gives us the flexibility to discuss linear separability for subgraphs with different levels of sparsity.

Before we state and prove Lemma C.4, let us first define the following events that we will work on.

Definition C.1 (Class Size Concentration) . For any δ > 0 , define

Definition C.2 (Degree Concentration) . For any δ ′ > 0 and for each node i ∈ [ n ] , define

Definition C.3 (Neighborhood Distribution Concentration) . For any δ ′ > 0 and for each node i ∈ [ n ] , define

where N i denotes the set of nodes connected to node i .

Then in Lemma C.4, we argue that for nodes in the α -subgroup defined in 4.1 for some appropriately chosen α > 0 , the above events will happen simultaneously with high probability.

Lemma C.4 (Concentration Inequalities) . Given α ∈ ( 1 log n , n ] , C 0 ( α ) , C 1 ( α ) defined by Definition 4.1, and V α = C 0 ( α ) ∪ C 1 ( α ) . Let δ = n -1 / 2+ ϵ and δ ′ = ( α log n ) -1 / 2+ ϵ , then for ϵ > 0 small enough, we have for any c > 0 , there is some C > 0 such that

Proof. Firstly, we consider the event I 1 ( δ ) . Since ( ϵ i ) i ∈ [ n ] ∼ Ber ( 1 2 ) , by the Hoeffding's inequality for independent Bernoulli random variables [38, Theorem 2.2.6], we have for any δ > 0 that

Notice that ∑ n i =1 ϵ i = | C 1 | and | C 0 | + | C 1 | = n , we can conclude that for any δ > 0 , the probability that the number of nodes in each class concentrates will satisfy

for some constant C, c > 0 .

We now turn to the events { I 2 ,i ( δ ′ ) } i ∈ [ n ] . Notice that the node degrees are sums of independent Poisson random variables. It is known that sums of independent Poisson random variables will be another Poisson random variable. Hence, conditioning on θ = ( θ i ) i ∈ [ n ] , for each node i ∈ [ n ] , we have

where D ii is the degree of node i , and

To prove that I 2 ,i ( δ ′ ) will occur with high probability for each i ∈ [ n ] , we introduce the following result [38, Corollary 2.3.7] whose proof can be found in the referred literature:

Lemma C.5 (Corollary 2.3.7 [38]) . If X ∼ Poi ( λ ) , then for t ∈ (0 , λ ] , we have

Here, we can let t = δ ′ λ where δ ′ ∈ (0 , 1] and get a tail bound as follows:

It follows that for each i ∈ [ n ] and any δ ′ ∈ (0 , 1] , we have

for some C, c > 0 .

We next consider the events { I 3 ,i ( δ ′ ) } i ∈ [ n ] . Observe that for each node i , we can decompose node degree as D ii = D intra ii + D inter ii , where and

̸

Obviously, D intra ii = | C ϵ i ∩ N i | and D inter ii = | C 1 -ϵ i ∩ N i | will concentrate around npθ i 2 and nqθ i 2 , correspondingly. And given the tail bound for { I 2 ,i ( δ ′ ) } i ∈ [ n ] , by a similar argument, we have for each i ∈ [ n ] and any δ ′ ∈ (0 , 1] ,

for some C, c > 0 .

Define the union event U ( δ, δ ′ ) = I 1 ( δ ) ⋂ i ∈V α I 2 ,i ( δ ′ ) ⋂ i ∈V α I 3 ,i ( δ ′ ) . Recall that ∀ i ∈ V α , we have θ i ≥ α . Thus, we can then choose δ = n -1 / 2+ ϵ and δ ′ = ( α log n ) -1 / 2+ ϵ . Since p, q = ω ( log 2 n n ) from Assumption 4.3, by a simple union bound, we have for ϵ > 0 small enough, for any c > 0 there is C > 0 such that

Finally, we establish the lower bound for α indicated in the statement, which is 1 log n . The reason why we need this lower bound is that if α is too small, then the subgroups: C 0 ( α ) , C 1 ( α ) will be too sparse that their member nodes' degree is too small to assure the concentration inequalities.

By the definition of the event: U ( δ, δ ′ ) and union bound, we have

and

We want to ensure the last term stays in O (1 /n β ) for some β > 0 . Notice that if α < log -1 n , we cannot find suitable ϵ > 0 for some small c to satisfy 1 -c · ( α log n ) 2 ϵ < 0 . Hence, we can conclude that a natural lower bound for α should be 1 log n , i.e., α > 1 log n .

Thus, combining Equation 3with this fact, we complete the proof.

Next, in Lemma C.6, we claim that given the adjacency matrix A , class memberships ( ϵ i ) i ∈ [ n ] , degree-corrected factors ( θ i ) i ∈ [ n ] and a pre-defined threshold α > 0 , then with high probability, the convoluted node features ˜ x i ≈ p µ + q ν p + q for i ∈ C 0 ( α ) and ˜ x i ≈ q µ + p ν p + q for i ∈ C 1 ( α ) .

Lemma C.6 (Convoluted Feature Concentration) . Given α ∈ ( 1 log n , n ] , C 0 ( α ) , C 1 ( α ) defined by Definition 4.1, and V α = C 0 ( α ) ∪ C 1 ( α ) . Conditionally on A , ( ϵ i ) i ∈ [ n ] and ( θ i ) i ∈ [ n ] , we have that for any c > 0 and some C > 0 , with probability at least 1 -C n c , for every node i ∈ V α and any unit vector w ,

Proof. Since ( X , A ) is sampled from DC-CSBM( µ , ν , p, q, θ ), when conditioning on ( ϵ i ) i ∈ [ n ] , we have node i 's node feature x i ∼ N ( m i , 1 d I ) where m i = µ if i ∈ C 0 and m i = ν if i ∈ C 1 . We can also write

where g i ∼ N ( 0 , I ) is standard normal vector.

On the other hand, conditioning on the adjacency matrix A and class memberships ϵ = ( ϵ i ) i ∈ [ n ] , the mean of the convoluted feature of node i can be written as

by the definition of the graph convolution operation ( ˜ x i = [ D -1 ˜ AX ] i ).

Thus, for any unit vector w , we have

Let us define F i = 1 D ii √ d ∑ j ∈ [ n ] ˜ A ij · ⟨ g j , w ⟩ and observe that ⟨ g j , w ⟩ is a standard Gaussian random variable for all j ∈ [ n ] . Thus, we have that F i ∼ N (0 , 1 dD ii ) , conditioning on the adjacency matrix A . Now we introduce Borell's inequality [2] to give a high-probability bound of | F i | for all i ∈ V α .

Lemma C.7 (Borell's Inequality, Theorem 2.1.1 in Adler et al. [2]) . Let F i ∼ N (0 , σ 2 F i ) for each i ∈ V α . Then for any K > 0 , we have

We can further define the event Q α = Q α ( t ) = { max i ∈V α | F i | ≤ t } . Observe that

If we let the union event U := U ( n -1 / 2+ ϵ , ( α log n ) -1 / 2+ ϵ ) defined the same as in Lemma C.4, then by Lemma C.7 and the definition of V α ,

for any c > 0 and some c ′ > 0 .

By the definition of event U ( δ, δ ′ ) , we can derive the upper bound of max i ∈V α σ F i as follows:

Since δ ′ is determined, we have for some constant k ′ , k > 0 ,

By choosing t = C ′ √ log n dn ( p + q ) α for some large constant C ′ > k > 0 , we can obtain

Thus, we have

Recall that when conditioning on the event U , we have

Thus, on the event U ∩ Q α , we have for each node i ∈ V α ,

which completes the proof.

Now we are ready to prove Theorem 4.4.

Proof. Recall the definition of linear separability, we need to find some unit vector v ∈ R d and b ∈ R such that

We fix ˜ v = 1 2 γ ( ν -µ ) and ˜ b = -⟨ µ + ν , ˜ v ⟩ 2 , where γ = 1 2 ∥ µ -ν ∥ 2 . By Assumption 4.3, Lemma C.4 and C.6, with probability at least 1 -O ( n -c ) for any c > 0 , for all i ∈ C 0 ( α ) , we have

Similarly, for all i ∈ C 1 ( α ) , we have

The above two inequalities imply the linear separability of { ˜ x i , i ∈ V α } , which completes the proof.

D Controlled Experiments of Identified Salient Factors

From Section 3.3, we discover six prominent dataset properties that correlate with some or all of the GNN models' performance. In Section 5, we have presented controlled experiments for GiniDegree to verify its relationship to GNNs' performance (Table 2). In this section, we further conduct controlled experiments for all the remaining identified salient factors, except for Pseudo Diameter , which is hard to control via manipulating explicit parameters provided by GraphWorld.

Across all experiments, we fix the number of nodes n = 5000 , cluster size slope as 0 . 0 , the number of clusters as 4 , feature dimension as 16, and feature center distance as 0 . 05 . For each of the experiments, we will keep most of the remaining GraphWorld parameters the same and only vary one of the parameters. The remaining parameters that we will manipulate are the p/q ratio, average degree, power exponent, and feature cluster variance. We give a short description of all the parameters in Table 3. For completeness, we summarize the value of the remaining parameters used in all four experiments in Table 4.

Table 5, 6, and 7 show the results of the four controlled experiments, correspondingly. Note that varying feature cluster variance can manipulate In-Feature Similarity and Feature Angular SNR simultaneously (Table 7). In general, all the results closely follow the regression results indicated in Table 1 and the discussion in Section 3.3.

E Robustness of the Sparse Regression Analysis

To demonstrate the robustness of the identified salient factors (defined in Section 3.3), we expand our sparse regression analysis to include five additional models that are more recent and popular. The models are TAGCN [10], GATv2 [5], SGC [40], APPNP [12] and GCNII [7]. For the additional experiments, we adopt the same configuration as in Appendix B. The updated analysis result is presented in Table 8.

We can observe that the widely influential factors and the narrowly influential factors all remain salient after incorporating the five additional models. The results show that our proposed multivariate regression analysis is robust with respect to our diverse choice of GNN models.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEdit: Based on the new experiments, I am increasng my score slightly\n\nThey conduct a multivariate regression on the metadata of datasets and performance of multiple GNN models and examine the correlation between GNN performance and data properties. \n\nThey show that Gini-Degree, Edge Homogeneity, and In-feature similarity are salient features for all of the models they consider and that average degree, pseudo-diameter, and feature angular snr are salient for some of the models. \n\nThey also perform some analysis on how the gini-coefficient of the degree distribution affects model performances on a novel SBM.\n\nOverall, I think this is a novel approach to an important problem and a good paper.\n\nSTRENGTHS:\nThis is a novel approach to an important problem. Advances in ML are based on empirical performance on benchmarks as much, if not more than, as they are on theoretical guarantees. Therefore, understanding the benchmark-datasets, and what makes a network preform well on them is of paramount importance. This paper which focuses on the metadata of the datasets can play a complementary role to other model driven analyses. \n\nThe DC-SBM model is interesting as is the concept of separate the $C(\\alpha)'s$ rather than the entire classes.\n\nAs a disclaimer: I did not read the proof of Theorem 4.4 as thoroughly as I would like to due to time constraints. At a glance, it appears to be a well-organized application of concentration of measure techniques. Would be curious if any of the reviewers have checked it more closely. (This is obviously not the authors fault and I do not hold it against them.)\n\nWEAKNESSES:\nThe characterization of ``universal factors\" is somewhat misleading as they are only universal amongst the several GNN models considered. Indeed, a potential direction, building off of this work would be to find GNN models which are not affected by these `universal' factors. For instance, there are GNNs designed to alleviate the oversmoothing issue and they might be more robust to datasets with low edge homogeneity or in-feature similarity. \n\nSome of the equations in the proof are spilling off of the page.\n\nQUESTIONS:\nThere are GNNs which use non-local aggregations such [1] and [2]. Do you think that they would be able to avoid the dependence on the Pseudo Diameters?\n\n[1] https://arxiv.org/abs/2305.13084\n[2] https://arxiv.org/abs/2201.08932\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper offers an intriguing analysis aimed at analyzing the performance of Graph Neural Networks (GNNs). The analysis explores various graph-theoretic properties of the graph datasets, which may impact the performance of GNNs. The authors investigate 15 graph properties across 15 datasets and examine the coefficient matrix of a multivariate linear regression model using the obtained metadata. The analysis reveals an intriguing observation: a negative correlation between Gini-Degree and the performance of GNNs. The authors support this observation with theoretical analysis and controlled experiments, providing compelling evidence to support their claim.

STRENGTHS:
1 - the hypothesis considered makes sense and properly evaluated
2 - the paper is well-written and well-organized
3 - the investigation reveal clear results and are supported by theoretical analysis and controlled experiments

WEAKNESSES:
1 - I appreciate the thorough investigation of the proposed hypothesis on generated synthetic datasets for node classification. However, it would be intriguing to expand the analysis by exploring a few real-world datasets that demonstrate both high and low Gini-degree. This would provide valuable insights into the applicability and generalizability of the observed relationship in real-world scenarios.

QUESTIONS:
1- I am curious to know if this observation holds true for the graph classification task as well.

2- Table 2 indicates that all GNNs exhibit improved performance with a decrease in Gini-degree. However, is this consistently the case, or is there a relationship with the type of convolution used? It would be intriguing to explore the connection between different graph convolution operators such as MPNN and GIN, and their relationship with Gini-Degree.

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGraph neural networks (GNNs) have demonstrated significant success across various problem areas. In this study, the authors propose a metadata-driven approach to examine the sensitivity of GNNs to graph data properties, spurred by the growing availability of graph learning benchmarks. A multivariate sparse regression analysis is performed on GNN performance across a range of datasets, identifying a set of significant data properties. The authors then focus on the degree distribution, investigating its influence on GNN performance through theoretical analysis and controlled experiments.\n\nSTRENGTHS:\n1. The paper is well-written and easy to comprehend.\n2. The concept of metadata-driven analysis is intriguing and holds potential for further refinement in future research.\n3. The paper includes a theoretical contribution that enhances our understanding of GNNs' success.\n\nWEAKNESSES:\n1. Given the comprehensive nature of this study, the authors should justify their selection of the GNN models used in Section 3, given the hundreds of models available in this field. There should be a principled approach to choosing models that are (1) representative, (2) popular, and (3) have diverse operational mechanisms, ensuring that the study is reliable and generalizable.\n2. The metadata-driven analysis presented in Section 3 is somewhat incomplete. As mentioned by the authors in Lines 201 - 202, Section 3 only investigates the correlation between GNN performances and data properties. Causality is explored later and exclusively for Gini-Degree. The insights related to edge homogeneity and in-feature similarity are primarily replications of existing work findings. Regarding the other three properties, the authors note that \"we do not yet have a good understanding of the mechanism of how these data properties impact model performance,\" indicating that the study is not fully complete.\n3. There should be a more comprehensive discussion relating to node degrees, as this is a popular property used to understand graphs and networks. The authors claim that \"the finding of the Gini coefficient of the degree distribution\" is novel, but how is it distinctly novel compared to existing work? I suggest the authors connect the findings in this work more closely to those in previous studies to strengthen their theoretical contribution.\n\nQUESTIONS:\nThe overall contribution of the study seems weak, mainly because of Section 3. Please refer to the weaknesses listed above.\n\nLIMITATIONS:\nThe authors have acknowledged the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors analyze the sensitivity of GNNs to graph properties such as degree distribution, homophily, etc. Using a multivariate sparse regression analysis (based on Lasso), they find a set of properties that are universal in that they effect all kinds of GNNs and some properties that are local in that they effect only a few of the GNNs. Among the global properties, one that is prominent is degree distribution that has an inverse effect on node classification accuracy. In the second half of the paper, the authors analyze the reasons behind this behavior using a theoretical model based on a Degree-Corrected Contextual Stochastic Block Model. The results of the model are validated by a controlled experiment.

STRENGTHS:
The paper presents a good analysis of the effect of degree distribution through empirical and theoretical analysis.

WEAKNESSES:
Translating the theoretical results into empirical outcomes requires the total degree to be bounded so that there are many nodes with similar degrees. Is this true for the datasets the authors considered?

At the same time, it is not surprising that it becomes harder to classify nodes that have similar degrees since one of the important features is not useful anymore. 

Can the theoretical model explain the other universal features such as edge heterogeneity and in-feature similarity?

Lasso requires the errors to be Gaussian. Did the authors verify this?

It is interesting to see that MLP also suffers from the effect of degree distribution.

Overall, the impact of the paper is not likely to be high since it only considers the problem of node classification.

QUESTIONS:
Please see above.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06