Title: Theoretical and Empirical Insights into the Origins of Degree Bias in Graph Neural Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: graph learning

fairness

degree
Abstract: Graph Neural Networks (GNNs) often perform better for high-degree nodes than low-degree nodes on node classification tasks. This degree bias can reinforce social marginalization by, e.g., privileging celebrities and other high-degree actors in social networks during social and content recommendation. While researchers have proposed numerous hypotheses for why GNN degree bias occurs, we find via a survey of 38 degree bias papers that these hypotheses are often not rigorously validated, and can even be contradictory. Thus, we provide an analysis of the origins of degree bias in message-passing GNNs with different graph filters. We prove that high-degree test nodes tend to have a lower probability of misclassification regardless of how GNNs are trained. Moreover, we show that degree bias arises from a variety of factors that are associated with a node's degree (e.g., homophily of neighbors, diversity of neighbors). Furthermore, we show that during training, some GNNs may adjust their loss on low-degree nodes more slowly than on high-degree nodes; however, with sufficiently many epochs of training, message-passing GNNs can achieve their maximum possible training accuracy, which is not significantly limited by their expressive power. Throughout our analysis, we connect our findings to previously-proposed hypotheses for the origins of degree bias, supporting and unifying some while drawing doubt to others. We validate our theoretical findings on 8 common real-world networks, and based on our theoretical and empirical insights, describe a roadmap to alleviate degree bias.

Theoretical and Empirical Insights into the Origins of Degree Bias in Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) often perform better for high-degree nodes than low-degree nodes on node classification tasks. This degree bias can reinforce social marginalization by, e.g., privileging celebrities and other high-degree actors in social networks during social and content recommendation. While researchers have proposed numerous hypotheses for why GNN degree bias occurs, we find via a survey of 38 degree bias papers that these hypotheses are often not rigorously validated, and can even be contradictory. Thus, we provide an analysis of the origins of degree bias in message-passing GNNs with different graph filters. We prove that high-degree test nodes tend to have a lower probability of misclassification regardless of how GNNs are trained. Moreover, we show that degree bias arises from a variety of factors that are associated with a node's degree (e.g., homophily of neighbors, diversity of neighbors). Furthermore, we show that during training, some GNNs may adjust their loss on low-degree nodes more slowly than on high-degree nodes; however, with sufficiently many epochs of training, message-passing GNNs can achieve their maximum possible training accuracy, which is not significantly limited by their expressive power. Throughout our analysis, we connect our findings to previouslyproposed hypotheses for the origins of degree bias, supporting and unifying some while drawing doubt to others. We validate our theoretical findings on 8 common real-world networks, and based on our theoretical and empirical insights, describe a roadmap to alleviate degree bias. Our code can be found at: github.com/ArjunSubramonian/degree-bias-exploration .

1 Introduction

Graph neural networks (GNNs) have been applied to node classification tasks such as document topic prediction [4] and content moderation [41]. However, in recent years, researchers have shown that GNNs exhibit better performance for high-degree nodes on node classification tasks. This has significant social implications, such as the marginalization of: (1) authors of less-cited papers when predicting the topic of papers in citation networks; (2) junior researchers when predicting the suitability of prospective collaborators in academic collaboration networks; (3) creators of newer or niche products when predicting the category of products in online product networks; and (4) authors of short or standalone websites when predicting the topic of websites in hyperlink networks.

To illustrate this phenomenon, Figure 1 shows that across different message-passing GNNs (see §D for details about architectures) applied to the CiteSeer dataset (where nodes represent documents and the classification task is to predict their topic), high-degree nodes generally incur a lower test loss than low-degree nodes. In practice, if such GNNs are applied to predict the topic of documents in social scientific studies, less-cited documents will be misclassified, which can lead to the con-

0

10

20

30

40

Degree

0.0

0.2

0.4

0.6

0.8

1.0

Test loss

RW

0

10

20

30

40

Degree

0.0

0.2

0.4

0.6

0.8

1.0

Test loss

SYM

0

10

20

30

40

Degree

0.0

0.2

0.4

0.6

0.8

1.0

Test loss

ATT

CITESEER

tributions of their authors not being appropriately recognized and erroneous scientific results. We present additional evidence of degree bias across different GNNs and datasets in §E.

Researchers have proposed various hypotheses for why GNN degree bias occurs in node classification tasks. However, we find via a survey of 38 degree bias papers that these hypotheses are often not rigorously validated, and can even be contradictory (§2). Furthermore, almost no prior works on degree bias provide a comprehensive theoretical analysis of the origins of degree bias that explicitly links a node's degree to its test and training error in the semi-supervised learning setting (§2).

Hence, we theoretically analyze the origins of degree bias in node classification during test and training time for general message-passing GNNs, with separate parameters for source and target nodes and residual connections. Our analysis spans different graph filter choices: RW (random walk-normalized filter), SYM (symmetric-normalized filter), and ATT (attention-based filter) (see §D for formal definitions). In particular, we prove that high-degree test nodes tend to have a lower probability of misclassification regardless of how GNNs are trained. Moreover, we show that degree bias arises from a variety of factors that are associated with a node's degree (e.g., homophily of neighbors, diversity of neighbors). Furthermore, we show that during training, SYM (compared to RW) may adjust its loss on low-degree nodes more slowly than on high-degree nodes; however, with sufficiently many epochs of training, message-passing GNNs can achieve their maximum possible training accuracy, which is only trivially curtailed by their expressive power. Throughout our analysis, we connect our findings to previously-proposed hypotheses for the origins of degree bias, supporting and unifying some while drawing doubt to others. We validate our theoretical findings on 8 real-world datasets (see §C)that are commonly used in degree bias papers (see Figure 2, §F). Based on our theoretical and empirical insights, we describe a principled roadmap to alleviate degree bias.

2 Background and Related Work

Numerous prior works have proposed hypotheses for why GNN degree bias occurs in node classification tasks. We summarize these hypotheses in Table 2 based on a survey of 38 non-review papers about degree bias in node classification that cite [45], a seminal work on degree bias.

While many of these papers have contributed solutions to degree bias (see §A for a thorough overview), we find that their hypotheses for the origins of degree bias are often not rigorously validated, and can even be contradictory. For example, some hypotheses locate the source of degree bias in the training stage, while others cite interactions between training and test-time factors or purely test-time issues. Moreover, hypothesis (H5) in Table 2, which posits that high-degree node representations cluster more strongly, conflicts with and (H10) , which argues that high-degree node representations have a larger variance. In our theoretical analysis of the origins of degree bias, we connect our findings to these hypotheses.

We further find that almost no prior works on degree bias provide a comprehensive theoretical analysis of the origins of GNN degree bias that explicitly links a node's degree to its test and training

error in the semi-supervised learning setting (see Table 3). For example, most works prove that: (a) high-degree nodes have a larger influence on GNN node representations or parameter gradients, or (b) high-degree nodes cluster more strongly around their class centers or are more likely to be linearly separable; however, these works do not directly bound the probability of misclassifying a node during training vs. test time in terms of its degree.

The few works that do provide a theoretical analysis of degree bias: (A1) perform this analysis with overly strong assumptions, e.g., that graphs are sampled from a Contextual Stochastic Block Model (CSBM) [10], or (A2) posit that GNNs do not have sufficient expressive power to map nodes with different degrees to distinct representations. However, in the case of (1), for CSBM graphs, as the number of nodes n → ∞ , the degrees of nodes in each class concentrate around a constant value, which is contradictory to real-world graphs, making CSBM an inappropriate model to theoretically analyze degree bias. Moreover, many real-world social networks exhibit a power-law degree distribution [3], which is not captured by a CSBM. In the case of (2), §I shows that the accuracy of GNNs on real-world networks is not significantly limited by the Weisfeiler-Leman (WL) test, which draws doubt to hypothesis (H7) .

Ultimately, previously-proposed hypotheses for why GNN degree bias occurs lack rigorous validation, and can even be contradictory. To unify and distill these hypotheses, we provide an analysis of the origins of degree bias in message-passing GNNs with different graph filters.

3 Preliminaries

Throughout our theoretical analysis, we connect our findings to previously-proposed hypotheses for the origins of degree bias, supporting and unifying some while drawing doubt to others. We further validate our findings on 8 real-world datasets (see §C) that are commonly used in degree bias papers (see Figure 2, §F). In all figures (except the PCA plots), error bars are reported over 10 random seeds. The factors of variability include model parameter initialization and training dynamics. All error bars are 1-sigma and represent the standard deviation (not standard error) of the mean. We

implicitly assume that errors are normally-distributed. Error bars are computed using PyTorch's std function [40]. We relegate all proofs to §B.

We first introduce relevant notation and assumptions. Suppose we have a C -class node classification problem defined over an undirected connected graph G = ( V , E ) with N = |V| nodes. We assume that our graph structure A ∈ { 0 , 1 } N × N and node labels Y ∈ N N ≤ C are fixed, but our node features X ∈ R N × d (0) are independently sampled from class-specific feature distributions, i.e., ∀ i ∈ V , X i ∼ D Y i . We further have a model M that maps X , A to predictions ̂ Y ∈ R N × C . We use a cross-entropy loss function ℓ ( M| i, c ) = -log ̂ Y i,c that computes the loss for node i ∈ V with respect to class c for M . Per the semi-supervised learning paradigm [24], we train M with the full graph X , A but only a labeled subset of nodes S ⊂ V .

4 Test-Time Degree Bias

The test-time degree bias of models is important to study, as it can yield disparate performance for low-degree nodes when models are deployed in the real world. We prove that high-degree test nodes tend to have a lower probability of misclassification. Moreover, we show that GNN degree bias arises from a variety of factors that are associated with a node's degree (e.g., homophily of neighbors, diversity of neighbors). We first present a theorem that bounds the probability of a test node i ∈ V \ S being misclassified. We suppose M is a neural network that has L layers. It takes as input X , A and generates node representations Z ( L ) ∈ R N × C ; these representations are then passed through a softmax activation function to get ˆ Y = H ( L ) = softmax ( Z ( L ) ) . At this point, we make few assumptions about the architecture of M ; M could be a graph neural network (GNN), or even an MLP or logistic regression model.

̸

Theorem 1. Consider a test node i ∈ V \ S , with Y i = c . Furthermore, consider a label c ′ = c . Let P ( ℓ ( M| i, c ) > ℓ ( M| i, c ′ )) be the probability of misclassifying i . Then, if E [ Z ( L ) i,c ′ -Z ( L ) i,c ] < 0 (i.e., M generalizes in expectation):

where the squared inverse coefficient of variation R i,c ′ = ( E [ Z ( L ) i,c ′ -Z ( L ) i,c ]) 2 Var [ Z ( L ) i,c ′ -Z ( L ) i,c ] .

The assumption that M generalizes in expectation is required for the application of Cantelli's inequality in the proof. Notably, it is not possible to prove a similar lower bound without making assumptions about the higher-order moments of Z ( L ) i,c ′ -Z ( L ) i,c . The coefficient of variation

Std [ Z ( L ) i,c ′ -Z ( L ) i,c ] E [ Z ( L ) i,c ′ -Z ( L ) i,c ] is a normalized measure of dispersion that is often used in economics to quantify inequality [13]. Thus, R i,c ′ captures how little Z i varies relative to its expected value. In summary, the probability of misclassification P ( ℓ ( M| i, c ) > ℓ ( M| i, c ′ )) can be minimized when R i,c ′ is maximized. Intuitively, the probability of misclassification is reduced when Z i is farther away, in expectation, from the decision boundary that separates classes c and c ′ , and has low dispersion. The following subsections reveal why R i,c ′ is large when i is high-degree.

4.1 Random Walk Graph Filter

So far, we have made few assumptions about M . Now, we suppose M is a general message-passing GNN [16]. In particular, for layer l :

where H ( l ) ∈ R N × d ( l ) are the l -th layer node representations (with H (0) = X and d ( L ) = C ), σ ( l ) is an instance-wise non-linearity (with σ ( L ) being softmax), P ( l ) ∈ R N × N is a graph filter, and W ( l ) 1 , W ( l ) 2 , W ( l ) 3 ∈ R d ( l -1) × d ( l ) are the l -th layer model parameters.

CITESEER

0.15

0.10

0.05

0.00

0.05

0.10

0.15

PC1

0.05

0.00

0.05

0.10

0.15

PC2

RW

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

250

Trace of sample covariance

of test representations

RW

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

Training loss

RW

Low-degree nodes

High-degree nodes

450

475

500

0.15

0.10

0.05

0.00

0.05

0.10

0.15

PC1

0.05

0.00

0.05

0.10

0.15

PC2

SYM

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

250

Trace of sample covariance

of test representations

SYM

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

Training loss

SYM

Low-degree nodes

High-degree nodes

450

475

500

0.15

0.10

0.05

0.00

0.05

0.10

0.15

PC1

0.05

0.00

0.05

0.10

0.15

PC2

ATT

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

250

Trace of sample covariance

of test representations

ATT

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

Training loss

ATT

Low-degree nodes

High-degree nodes

450

475

500

We first consider the special case that ∀ l ∈ N ≤ L , P ( l ) = P rw = D -1 A (i.e., the uniform random walk transition matrix), where D is the diagonal degree matrix with entries D ii = ∑ j ∈V A ij . We further simplify the model by choosing all σ ( l ) ( l < L ) to be the identity function (e.g., as in [51]). By doing so, we get the following linear jumping knowledge model RW [58]:

where ∀ l ∈ N ≤ L , W ( l ) ∈ R d (0) × C . W ( l ) is the sum of all the weight terms that correspond to P l rw in Eqn. 2; for simplicity, we collapse each sum of weight terms into a single weight matrix. It is still reasonable to have a different weight matrix W ( l ) for each term P l rw X , as we may need to extract different information from features aggregated from neighborhoods at different hops. For each model M , M denotes the linearized version of the model that we theoretically analyze. Linearizing GNNs is a common practice in the literature [51, 7, 39].

We now prove a lower bound for R i,c ′ . By identifying nodes for which this lower bound is larger, we can indirectly figure out which nodes have a lower probability of misclassification. In particular, we find that the bound is generally larger for high-degree nodes, which sheds light on the origins of degree bias. For simplicity of notation, we denote the weights corresponding to the decision boundary of the l -th term that separates classes c and c ′ by w ( l ) c ′ -c = W ( l ) .,c ′ -W ( l ) .,c , and N ( l ) ( i ) to

be the distribution over the terminal nodes of lengthl uniform random walks starting from node i . We further define:

as the l -hop prediction homogeneity of i with respect to c ′ when Y i = c . In essence, E x ∼D Y j [ x T w ( l ) c ′ -c ] captures the expected prediction score of w ( l ) c ′ -c for a node j whose features X j ∼ D Y j ; when E x ∼D Y j [ x T w ( l ) c ′ -c ] is more negative on average, w ( l ) c ′ -c predicts j to belong to class c with higher likelihood. Thus, β ( l ) i,c ′ measures the expected prediction score for nodes j , weighted by their probability of being reached by a lengthl random walk starting from i .

From a topological perspective, because β ( l ) i,c ′ depends on the distribution of random walks from i , it is intimately related to local graph structure. Indeed, β ( l ) i,c ′ can be interpreted as a 'local subgraph difference' and is highly influenced by the local homophily of i . However, β ( l ) i,c ′ is also influenced by the presence of l -hop neighbors contained in the training set, as the model is more likely to correctly classify these nodes by a large margin; hence, β ( l ) i,c ′ does not only boil down to local homophily. We discuss other connections between prediction homogeneity, homophily, and separability in §A.4.

In addition to the l -hop prediction homogeneity, we denote the l -hop collision probability by:

which quantifies the probability of two lengthl random walks starting from i colliding at the same end node j . When the collision probability is lower, random walks starting from i have a higher likelihood of ending at distinct nodes; in effect, the random walks can be considered to be more diverse.

Theorem 2. Assume that ∀ l ∈ N ≤ L , ∀ j ∈ V , Var x ∼D Y j [ x T w ( l ) c ′ -c ] ≤ M . Then:

We observe that to make R i,c ′ larger, and thus minimize the probability of misclassification, it is sufficient (although not necessary) that the inverse collision probability 1 ∑ L α ( l ) is larger. When

L = 1 , 1 ∑ L l =0 α ( l ) i = 1 1 D ii +1 , which is larger for high-degree nodes. We find empirically that the inverse collision probability is positively associated with node degree (see Figures 3, 13, 14). (We elaborate on connections between the inverse collision probability and node degree in §K.) Furthermore, disparities in the inverse collision probability across nodes with different degrees is reduced by residual connections and increased by self-loops. Intuitively, random walks starting from highdegree nodes diffuse more quickly, maximizing the probability of any two random walks not colliding at the same end node; in this way, a higher inverse collision probability indicates a more diverse and possibly informative L -hop neighborhood. This finding supports hypothesis (H1) (see Table 2).

l

=0

i

Additionally, to make R i,c ′ larger, it is sufficient that for all l ∈ N ≤ L , β ( l ) i,c ′ is more negative, e.g., when most nodes in the l -hop neighborhood of i are predicted to belong to class c . Thus, β ( l ) i,c ′ can be more negative when nodes in the l -hop neighborhood of i also are in class c (i.e., node i has high local homophily) and were part of the training set S , leading to them being correctly classified. This finding supports hypotheses (H4) and (H6) . Notably, we cannot make ∑ L l =0 β l ) i,c ′ more positive to increase R i,c ′ ; this would violate the assumption of Theorem 2 that the model generalizes in expectation, which is necessary to make a mathematically rigorous statement about degree bias via tail bounds. Intuitively, it also would not make sense that RW and SYM reduce the misclassification error for a node by predicting its neighbors to be of a different class, since message passing smooths the representations of adjacent nodes. Moreover, distribution shifts in local homophily from train to

0

10

20

30

40

Degree

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Inverse collision probability

RW

0

10

20

30

40

Degree

0

5

10

15

20

Inverse collision probability

SYM

0

10

20

30

40

Degree

0.3

0.4

0.5

0.6

0.7

0.8

Inverse collision probability

ATT

CITESEER

test time can reduce test-time prediction performance, bringing β ( l ) i,c ′ closer to 0; this can increase R i,c ′ , thereby not inducing as much degree bias at the expense of overall test performance.

Furthermore, our proof of Theorem 2 (Eqn. 21) reveals that in expectation, the linearized model RW produces similar representations for low and high-degree nodes with similar L -hop neighborhood homophily levels. However, low-degree nodes (specifically nodes with a lower inverse collision probability) tend to have a higher variance in RW's representation space than high-degree nodes do (Eqn. 24). This entails that factors beyond homophily (e.g., diversity of neighbors) induce degree bias. We validate these findings empirically in Figure 2 and §F. In Figure 2, we see in the left plot in the RW row (first row) that low-degree test nodes have representations that are similarly centered but more spread out in the first two principal components of all the test representations than high-degree nodes; we confirm that low-degree node representations have a larger variance in the middle plot in the RW row. Thus, regardless of how RW is trained, low-degree nodes have a higher probability of being on the wrong side of RW's decision boundaries. Indeed, the left plot in the RW row shows that low-degree nodes of a certain class end up closer to nodes of a different class at a higher rate. Notably, this occurs even when RW is relatively shallow (i.e., 3 layers). Thus, this finding supports hypothesis (H5) , as well as draws doubt to hypotheses (H3) and (H10) . Our results for RW may also hold for ATT when low-degree nodes are generally less attended to since like random walk transition matrices, attention matrices are row-stochastic.

4.2 Symmetric Graph Filter

We now consider the special case that ∀ l ∈ N ≤ L , P ( l ) = P sym = D -1 2 AD -1 2 . We once again simplify M by making all σ ( l ) the identity function, getting SYM:

We define:

as the degree-discounted l -hop prediction homogeneity. Similar to β ( l ) i,c ′ , ˜ β ( l ) i,c ′ measures the expected prediction score for nodes j , but weighted by the inverse square root of their degree in addition to their probability of being reached by a lengthl random walk starting from i . In effect, ˜ β ( l ) i,c ′ more heavily discounts the prediction scores for high-degree nodes. We also denote the degree-discounted sum of collision probabilities by:

where each summation term [ ( P l rw ) ij ] 2 quantifies the probability of two lengthl random walks starting from i ending at j and is discounted by the degree of j . Compared to the random walk setting, the degree-discounted prediction homogeneity and sum of collision probabilities suppress the contributions of high-degree nodes. We now prove a lower bound for R i,c ′ for SYM.

Theorem 3. Assume that ∀ l ∈ N ≤ L , ∀ j ∈ V , Var x ∼D Y j [ x T w ( l ) c ′ -c ] ≤ M

Once again, we observe that R i,c ′ is larger, and thus the probability of misclassification is minimized, when the inverse (degree-discounted) sum of collision probabilities 1 ∑ L l =0 ˜ α ( l ) i is larger and for all l ∈ N ≤ L , the (degree-discounted) l -hop prediction homogeneity ˜ β ( l ) i,c ′ is more negative. Like for RW, these findings support hypotheses (H1) , (H4) , and (H6) (see Table 2).

Furthermore, our proof of Theorem 3 (Eqn. 33) reveals that in expectation, SYM often produces representations for low-degree nodes that lie closer to SYM's decision boundary than representations of high-degree nodes with similar L -hop neighborhood homophily levels. This is because SYM produces node representations that are approximately scaled by the square root of the node's degree. However, for the same reason, unlike for RW, low-degree nodes tend to have a lower variance in SYM's representation space than high-degree nodes do (Eqn. 36); this corroborates the findings of [12]. We validate this empirically in Figure 2 and §F on the homophilic datasets (i.e., all datasets except chameleon and squirrel). In Figure 2, we see in the left plot in the SYM row (second row) that low-degree test nodes (particularly low-degree nodes with many high-degree nodes in their L -hop neighborhood) have representations that are closer to SYM's decision boundaries but less spread out in the first two principal components of all the test representations than high-degree nodes; we confirm that low-degree node representations have a smaller or comparable variance in the middle plot in the SYM row. We emphasize that while SYM representations of high-degree nodes have a higher variance, this itself is not the cause of degree bias; since the standard deviation and expectation of SYM node representations are approximately scaled by the same factor, by Theorem 1, the variance of SYM representations of high-degree nodes does not enlarge R i,c ′ noticeably more than in the RW case.

Notably, our theoretical findings do extend to heterophilic graphs. In particular, high-degree nodes in heterophilic networks (e.g., chameleon and squirrel) do not have higher negative L -hop prediction homogeneity levels due to higher local heterophily (see §F), and hence we do not necessarily observe better test performance for them (see Figure 5). None of our theoretical analysis assumes homophilic networks.

Ultimately, like for RW, low-degree nodes (specifically nodes with a lower inverse collision probability) have a larger probability of being on the wrong side of SYM's decision boundaries (regardless of how SYM is trained). Indeed, low-degree nodes of a certain class end up closer to nodes of a different class at a higher rate. Notably, this occurs even when SYM is relatively shallow (i.e., 3 layers). Thus, this finding supports hypothesis (H5) , and draws doubt to hypotheses (H3) , (H7) , and (H10) .

5 Training-Time Degree Bias

We show that during training, SYM (compared to RW) may adjust its loss on low-degree nodes more slowly than on high-degree nodes. This finding is important because as GNNs are applied to increasingly large networks, only a few epochs of training may be possible due to limited compute; as such, we must ask: which nodes receive superior utility from limited training? Even though we know the labels for training nodes, GNNs may serve as an efficient lookup mechanism for training nodes in deployed systems; thus, if partially-trained, GNNs can perform poorly for low-degree training nodes. We also empirically demonstrate that despite learning at different rates for low vs. high-degree nodes, message-passing GNNs (even those with static filters) can achieve their maximum possible training accuracy, which is not significantly curtailed by their expressive power.

We first demonstrate that during each step of training of SYM with gradient descent, the loss of low-degree nodes is adjusted more slowly than high-degree nodes. We consider the setting that, for all l ∈ N ≤ L , at each training step t :

where W ( l ) [ t ] is W ( l ) at training step t , η is the learning rate, ℓ [ t ] is the model's loss at t , and B [ t ] ⊆ S (where S ⊆ V is the labeled subset of nodes) is the batch used at step t .

Consider a node i ∈ V , with Y i = c . We define Z ( L ) i [ t ] to be Z ( L ) i at timestep t . We begin by proving the following lemma, which states that for any model M , ℓ [ t ]( M| i, c ) (for all t ) is λ -Lipschitz continuous with respect to Z ( L ) i [ t ] .

Lemma 1. For all t , ℓ [ t ]( M| i, c ) is λ -Lipschitz continuous with respect to Z ( L ) i [ t ] with constant λ = √ 2 , that is:

Now, we move to the main theorem where we bound the change in loss i after an arbitrary training step t (regardless of batching paradigm) in terms of its degree. We denote the residual of the predictions of SYM at step t by ϵ [ t ] = H ( L ) [ t ] -onehot ( Y [ t ]) , where H ( L ) [ t ] and onehot ( Y [ t ]) are the submatrices formed from the rows of H ( L ) and onehot ( Y ) , respectively, that correspond to the nodes in B [ t ] . Furthermore, we denote ∀ l ∈ N ≤ L , the expected similarity of the neighborhoods of i and B [ t ] by ˜ χ ( l ) i ∈ R | B [ t ] | , where for m ∈ B [ t ] , ( ˜ χ ( l ) i [ t ] ) m =

√ D mm E j ∼N ( l ) ( i ) ,k ∼N ( l ) ( m ) [ 1 √ D jj D kk X j X T k ] . Specifically, ( ˜ χ ( l ) i [ t ] ) m captures the degreediscounted expected similarity between the raw features of nodes j and k with respect to the l -hop random walk distributions of i ∈ V and m ∈ B [ t ] . Notably, our matrix is prefeature aggregation (e.g., unlike [37]).

Theorem 4. The change in loss for i after an arbitrary training step t obeys:

As observed, the change (either increase or decrease) in loss for i after an arbitrary training step has a smaller magnitude if i is low-degree. Thus, when ℓ [ t + 1]( SYM | i, c ) < ℓ [ t ]( SYM | i, c ) (e.g., if i ∈ B [ t ] ), the loss for i decreases more slowly when i is low-degree. In effect, because the magnitude of SYM node representations is positively associated with node degree while the magnitude of each gradient descent step is the same across nodes, the representations of low-degree nodes experience a smaller change during each step. We additionally notice that the loss for i changes more slowly when the features of nodes in its L -hop neighborhood are not similar to the features in the L -hop neighborhoods of the nodes in each training batch (i.e., ∑ L l =0 ∥ ∥ ∥˜ χ ( l ) i [ t ] ∥ ∥ ∥ 2 is small). Because the L -hop neighborhoods of low-degree nodes tend to be smaller than those of high-degree nodes, their neighborhoods often have less overlap with the neighborhoods of training nodes, which can further constrain the rate at which the loss for i changes. Notably, while node degree highly affects the rate of learning, differences in χ across nodes due to factors other than degree are also influential.

˜

We confirm these findings empirically in Figure 2 and §F. For all the datasets, when training SYM, the blue curve (i.e., the loss for low-degree nodes) has a less steep rate of decrease than the orange curve (i.e., the loss for high-degree nodes) as the number of epochs increases. For example, in Figure 2, in the case of RW and ATT, the training loss curves for low and high-degree nodes (including error bars) overlap during the first ∼ 20 epochs of training. However, for SYM, the loss curve for highdegree nodes descends more rapidly than the curve for low-degree nodes. These findings support hypothesis (H2) (c.f. Table 2).

In §H, we demonstrate that during each step of training RW with gradient descent, compared to SYM, the loss of low-degree nodes in S is not necessarily adjusted more slowly. Furthermore,

in §I, we empirically show that SYM (despite learning at different rates for low vs. high-degree nodes), RW, and ATT can achieve their maximum possible training accuracy, which is often close to 100%; this indicates that expressive power does not significantly limit the accuracy of these models in practice and draws doubt to hypothesis (H7) .

6 Principled Roadmap to Address Degree Bias

The primary aim of this work is to explore and explain the origins of GNN degree bias, which lacks a principled understanding. Future research can build on the strong theoretical and empirical foundation laid by this paper to propose alleviation strategies for degree bias. In particular, our findings reveal that any alleviation strategies should target the following theoretically-justified criteria, which we have empirically validated on 8 real-world datasets:

Maximizing the inverse collision probability of low-degree nodes (e.g., via edge augmentation for low-degree nodes). Figure 3 and the plots in Section G show strong positive associations between inverse collision probability and degree for the RW, SYM, and ATT filters, and Figure 1 and the plots in Section E show strong negative associations between degree and test loss for the homophilic datasets. Hence, we validate that a higher inverse collision probability is associated with lower test loss, as our theory predicts.

Increasing the L -hop prediction homogeneity of low-degree nodes (e.g., by ensuring similar label densities in the neighborhoods of low and high-degree nodes). The lack of degree bias observed in Figure 5 for chameleon and squirrel (which are heterophilic networks), compared to Figure 1 and the plots in Section E, confirms our theoretical finding that under heterophily, the prediction homogeneity for high-degree nodes is closer to 0, so high-degree nodes do not necessarily experience better performance.

Minimizing distributional differences (e.g., differences in expectation, variance) in the representations of low and high-degree nodes. Figures 2 and 6-10 empirically confirm our theoretical finding that disparities in the expectation and variance of node representations are responsible for performance disparities. Figures 11 and 12 suggest that smaller distributional differences among representations (due to heterophily) can alleviate degree bias.

Reducing training discrepancies with regards to the rate at which GNNs learn for low vs. high-degree nodes. Figure 2 and the plots in Section F validate our theoretical finding that SYM adjusts its loss on low-degree nodes more slowly than on high-degree nodes (see 5).

These criteria are important because they reflect (to a large extent) inherent fairness issues with the graph filters that are popular in graph learning. For instance, the random walk and symmetric filters disadvantage low-degree nodes by yielding representations with high variance and low magnitude, respectively. It is valuable for graph learning practitioners to investigate filters that are adaptive or not restricted to the graph topology in a way that ensures that low-degree nodes are not marginalized through disparate representational distributions or poor neighborhood diversity.

7 Conclusion

Our theoretical analysis aims to unify and distill previously-proposed hypotheses for the origins of GNN degree bias. We prove that high-degree test nodes tend to have a lower probability of misclassification and that degree bias arises from a variety of factors associated with a node's degree (e.g., homophily of neighbors, diversity of neighbors). Furthermore, we show that during training, some GNNs may adjust their loss on low-degree nodes more slowly; however, GNNs often achieve their maximum possible training accuracy and are trivially limited by their expressive power. We validate our theoretical findings on 8 real-world networks. Finally, based on our theoretical and empirical insights, we describe a roadmap to alleviate degree bias. More broadly, we encourage research efforts that unveil forms of inequality reinforced by GNNs. We detail the limitations and possible future directions of our work in §L, including our survey, theoretical analysis (e.g., focusing on linearized GNNs, node classification), empirical validation (e.g., exploring degree bias in the inductive learning setting and heterogeneous and directed networks), and roadmap. We additionally discuss broader impacts in §M.

Acknowledgments and Disclosure of Funding

This work was partially supported by NSF 2211557, NSF 1937599, NSF 2119643, NSF 2303037, NSF 2312501, NASA, SRC JUMP 2.0 Center, Amazon Research Awards, and Snapchat Gifts.

References

Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In Hal Daum´ e III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 74-84. PMLR, 13-18 Jul 2020.

Chirag Agarwal, Himabindu Lakkaraju*, and Marinka Zitnik*. Towards a unified framework for fair and stable graph representation learning. 2021.

Albert-L´ aszl´ o Barab´ asi. Network science. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 371, 2016.

Aleksandar Bojchevski and Stephan G¨ unnemann. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. In International Conference on Learning Representations , 2018.

Stephen Bonner, Ufuk Kirik, Ola Engkvist, Jian Tang, and Ian P Barrett. Implications of topological imbalance for representation learning on biomedical knowledge graphs. Briefings in Bioinformatics , 23(5):bbac279, 07 2022.

Jiajia Chen, Jiancan Wu, Jiawei Chen, Xin Xin, Yong Li, and Xiangnan He. How graph convolutions amplify popularity bias for recommendation? Frontiers of Computer Science , 18(5), December 2023.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Hal Daum´ e III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1725-1735. PMLR, 13-18 Jul 2020.

Renyao Chen, Junye Lei, Hong Yao, Tailong Li, and Shengwen Li. Anchor-enhanced geographical entity representation learning. IEEE Transactions on Neural Networks and Learning Systems , pages 1-15, 2023.

Zhengyu Chen, Teng Xiao, and Kun Kuang. Ba-gnn: On learning bias-aware graph neural network. In 2022 IEEE 38th International Conference on Data Engineering (ICDE) , pages 3012-3024, 2022.

Yash Deshpande, Subhabrata Sen, Andrea Montanari, and Elchanan Mossel. Contextual stochastic block models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Wei Ding, Jiawei Sun, Jie Li, and Chentao Wu. Inductive dummy-based homogeneous neighborhood augmentation for graph collaborative filtering. In 2023 International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2023.

Claire Donnat and So Won Jeong. Studying the effect of GNN spatial convolutions on the embedding space's geometry. In Robin J. Evans and Ilya Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence , volume 216 of Proceedings of Machine Learning Research , pages 539-548. PMLR, 31 Jul-04 Aug 2023.

Brian Everitt. The Cambridge dictionary of statistics . Cambridge University Press, Cambridge, UK; New York, 2002.

Fuli Feng, Weiran Huang, Xiangnan He, Xin Xin, Qifan Wang, and Tat-Seng Chua. Should graph convolution trust neighbors? a simple causal inference method. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '21, page 1208-1218, New York, NY, USA, 2021. Association for Computing Machinery.

Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Francesco Di Giovanni, James Rowbottom, Benjamin Paul Chamberlain, Thomas Markovich, and Michael M. Bronstein. Understanding convolution on graphs via energies. Transactions on Machine Learning Research , 2023.

William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 1025-1035, Red Hook, NY, USA, 2017. Curran Associates Inc.

Haoyu Han, Xiaorui Liu, Feng Shi, MohamadAli Torkamani, Charu C. Aggarwal, and Jiliang Tang. Towards label position bias in graph neural networks. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Xiaotian Han, Kaixiong Zhou, Ting-Hsiang Wang, Jundong Li, Fei Wang, and Na Zou. Marginal nodes matter: Towards structure fairness in graphs, 2023.

Van Thuy Hoang and O-Joun Lee. Mitigating degree biases in message passing mechanism by utilizing community structures, 2023.

Mingxuan Ju, Tong Zhao, Wenhao Yu, Neil Shah, and Yanfang Ye. Graphpatcher: Mitigating degree bias for graph neural networks via test-time augmentation. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Jian Kang, Yan Zhu, Yinglong Xia, Jiebo Luo, and Hanghang Tong. Rawlsgcn: Towards rawlsian difference principle on graph convolutional network. In Proceedings of the ACM Web Conference 2022 , WWW'22, page 1214-1225, New York, NY, USA, 2022. Association for Computing Machinery.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Peizhao Li, Yifei Wang, Han Zhao, Pengyu Hong, and Hongfu Liu. On dyadic fairness: Exploring and mitigating bias in graph connections. In International Conference on Learning Representations , 2021.

Ting Wei Li, Qiaozhu Mei, and Jiaqi Ma. A metadata-driven approach to understand graph neural networks. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Xueqi Li, Guoqing Xiao, Yuedan Chen, Zhuo Tang, Wenjun Jiang, and Kenli Li. An explicitly weighted gcn aggregator based on temporal and popularity features for recommendation. ACM Trans. Recomm. Syst. , 1(2), apr 2023.

Langzhang Liang, Zenglin Xu, Zixing Song, Irwin King, Yuan Qi, and Jieping Ye. Tackling long-tailed distribution issue in graph neural networks via normalization. IEEE Transactions on Knowledge and Data Engineering , pages 1-11, 2023.

Jie Liao, Jintang Li, Liang Chen, Bingzhe Wu, Yatao Bian, and Zibin Zheng. Sailor: Structural augmentation based tail node representation learning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , CIKM '23, page 1389-1399, New York, NY, USA, 2023. Association for Computing Machinery.

Ningyi Liao, Haoyu Liu, Zulun Zhu, Siqiang Luo, and Laks V. S. Lakshmanan. Benchmarking spectral graph neural networks: A comprehensive study on effectiveness and efficiency, 2024.

Songtao Liu, Rex Ying, Hanze Dong, Lanqing Li, Tingyang Xu, Yu Rong, Peilin Zhao, Junzhou Huang, and Dinghao Wu. Local augmentation for graph neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 14054-14072. PMLR, 17-23 Jul 2022.

Yazheng Liu, Xi Zhang, and Sihong Xie. Trade less accuracy for fairness and trade-off explanation for gnn. In 2022 IEEE International Conference on Big Data (Big Data) , pages 4681-4690, 2022.

Zemin Liu, Yuan Fang, Wentao Zhang, Xinming Zhang, and Steven C.H. Hoi. Locality-aware tail node embeddings on homogeneous and heterogeneous networks. IEEE Transactions on Knowledge and Data Engineering , pages 1-16, 2023.

Zemin Liu, Trung-Kien Nguyen, and Yuan Fang. Tail-gnn: Tail-node graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , KDD '21, page 1109-1119, New York, NY, USA, 2021. Association for Computing Machinery.

Zemin Liu, Trung-Kien Nguyen, and Yuan Fang. On generalized degree fairness in graph neural networks. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence , AAAI'23/IAAI'23/EAAI'23. AAAI Press, 2023.

L. Lov´ asz. Random walks on graphs: A survey. In D. Mikl´ os, V. T. S´ os, and T. Sz˝ onyi, editors, Combinatorics, Paul Erd˝ os is Eighty , volume 2, pages 353-398. J´ anos Bolyai Mathematical Society, Budapest, 1996.

Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, XiaoWen Chang, and Doina Precup. Revisiting heterophily for graph neural networks. Advances in neural information processing systems , 35:1362-1375, 2022.

Sitao Luan, Chenqing Hua, Minkai Xu, Qincheng Lu, Jiaqi Zhu, Xiao-Wen Chang, Jie Fu, Jure Leskovec, and Doina Precup. When do graph neural networks help with node classification? investigating the homophily principle on node distinguishability. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily a necessity for graph neural networks? In International Conference on Learning Representations , 2022.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K¨ opf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library . Curran Associates Inc., Red Hook, NY, USA, 2019.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-Scale attributed node embedding. Journal of Complex Networks , 9(2):cnab014, 05 2021.

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan G¨ unnemann. Pitfalls of graph neural network evaluation. ArXiv , abs/1811.05868, 2018.

Harry Shomer, Wei Jin, Wentao Wang, and Jiliang Tang. Toward degree bias in embeddingbased knowledge graph completion. In Proceedings of the ACM Web Conference 2023 , WWW '23, page 705-715, New York, NY, USA, 2023. Association for Computing Machinery.

Arjun Subramonian, Levent Sagun, and Yizhou Sun. Networked inequality: Preferential attachment bias in graph neural network link prediction. In NeurIPS 2023 Workshop: New Frontiers in Graph Learning , 2023.

Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Yiqi Wang, Jiliang Tang, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. Investigating and mitigating degree-related biases in graph convoltuional networks. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , CIKM '20, page 1435-1444, New York, NY, USA, 2020. Association for Computing Machinery.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li` o, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Srinivas Virinchi and Anoop Saladi. Blade: Biased neighborhood sampling based graph neural network for directed graphs. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining , WSDM'23, page 42-50, New York, NY, USA, 2023. Association for Computing Machinery.

Junfu Wang, Yuanfang Guo, Liang Yang, and Yunhong Wang. Understanding heterophily for graph neural networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 50489-50529. PMLR, 21-27 Jul 2024.

Ruijia Wang, Xiao Wang, Chuan Shi, and Le Song. Uncovering the structural fairness in graph contrastive learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Chunyu Wei, Jian Liang, Di Liu, Zehui Dai, Mang Li, and Fei Wang. Meta graph learning for long-tail recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 2512-2522, New York, NY, USA, 2023. Association for Computing Machinery.

Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr. au2, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks, 2019.

Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self-supervised graph learning for recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '21, page 726-735, New York, NY, USA, 2021. Association for Computing Machinery.

Jun Wu, Jingrui He, and Jiejun Xu. Demo-net: Degree-specific graph neural networks for node and graph classification. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD '19, page 406-415, New York, NY, USA, 2019. Association for Computing Machinery.

Teng Xiao, Zhengyu Chen, Donglin Wang, and Suhang Wang. Learning how to propagate messages in graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , KDD '21, page 1894-1903, New York, NY, USA, 2021. Association for Computing Machinery.

Hui Xu, Liyao Xiang, Femke Huang, Yuting Weng, Ruijie Xu, Xinbing Wang, and Chenghu Zhou. Grace: Graph self-distillation and completion to mitigate degree-related biases. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 2813-2824, New York, NY, USA, 2023. Association for Computing Machinery.

Jianan Xu, Jiajin Huang, Jianwei Zhao, and Jian Yang. Hyncf: A hybrid normalization strategy via feature statistics for collaborative filtering. Expert Systems with Applications , 238:121875, 2024.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2019.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 54535462. PMLR, 10-15 Jul 2018.

Yujun Yan, Milad Hashemi, Kevin Swersky, Yaoqing Yang, and Danai Koutra. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 1287-1292, 2022.

Sukwon Yun, Kibum Kim, Kanghoon Yoon, and Chanyoung Park. Lte4g: Long-tail experts for graph neural networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , CIKM '22, page 2434-2443, New York, NY, USA, 2022. Association for Computing Machinery.

An Zhang, Wenchang Ma, Xiang Wang, and Tat-Seng Chua. Incorporating bias-aware margins into contrastive loss for collaborative filtering. Advances in Neural Information Processing Systems , 35:7866-7878, 2022.

Guixian Zhang, Debo Cheng, Guan Yuan, and Shichao Zhang. Learning fair representations via rebalancing graph structure. Information Processing & Management , 61(1):103570, 2024.

Guixian Zhang, Rongjiao Liang, Zhongyi Yu, and Shichao Zhang. Rumour detection on social media with long-tail strategy. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2022.

Guixian Zhang, Shichao Zhang, and Guan Yuan. Bayesian graph local extrema convolution with long-tail strategy for misinformation detection. ACM Trans. Knowl. Discov. Data , jan 2024. Just Accepted.

Minghao Zhao, Qilin Deng, Kai Wang, Runze Wu, Jianrong Tao, Changjie Fan, Liang Chen, and Peng Cui. Bilateral filtering graph convolutional network for multi-relational social recommendation in the power-law networks. ACM Trans. Inf. Syst. , 40(2), sep 2021.

Mingxia Zhao and Adele Lu Jia. Dahgn: Degree-aware heterogeneous graph neural network. Knowledge-Based Systems , 285:111355, 2024.

Jiangqiang Zhu, Kai Li, Jinjia Peng, and Jing Qi. Self-supervised graph attention collaborative filtering for recommendation. Electronics , 12(4), 2023.

Markus Zopf. 1-wl expressiveness is (almost) all you need. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2022.

Appendix

Table of Contents

A Overview of Hypotheses for, Theoretical Analyses of, and Proposed Solutions to Degree Bias

A.1 Hypotheses for Degree Bias

A.2 Theoretical Analyses of Degree Bias

A.3 Proposed Solutions to Degree Bias

One line of research has produced neighborhood augmentation strategies. For example, [34] perform feature-adaptive neighborhood translation from high-degree nodes to structurally-limited lowdegree nodes to enhance their representations. [52] generate multiple views of node neighborhoods (e.g., via node and edge dropping) and learn to maximize the similarity of representations of different views, towards improving the robustness of low-degree node representations. [21] patch the ego-graphs of low-degree nodes by generating virtual neighbors. [55] self-distill graphs and complete the neighborhoods of low-degree nodes with more homophilic links. Other methods include:

[67] and [66] generate multiple views of node neighborhoods (e.g., via node and edge dropping) and learn to maximize the similarity of representations of different views, towards improving the robustness of low-degree node representations.

[49] interpolate additional links for low-degree nodes and purify links for high-degree nodes to balance neighborhood information across nodes with different degrees.

[31] generate more samples in the local neighborhoods of low-degree nodes, to augment the information in these neighborhoods.

[63] leverage a Transformer architecture and contrastive learning with augmentations (e.g., via node and edge dropping) to bolster the influence of low-degree nodes during training.

[43] augment the neighborhoods of low-degree nodes in knowledge graphs with synthetic triples.

[29] augment the local structure of low-degree nodes by adding homophilic edges.

[47] generate contextually-dependent neighborhoods based on a node's degree.

[50] contribute a meta-learning strategy to generate additional edges for low-degree nodes.

[11] introduce dummy nodes connected to all the nodes in the graph to improve message passing to low-degree nodes.

[20] introduce a learnable graph augmentation strategy to connect low-degree nodes via more within-community edges, as well as an improved self-attention mechanism.

Another line of research has produced algorithms that normalize graph filters or node representations to minimize distributional differences between the representations of nodes with different degrees. [22] propose pre-processing and in-processing approaches that re-normalize the graph filter to be doubly stochastic. [56] calculate node representation statistics via a hybrid strategy and use these statistics to normalize representations. [28] normalize the representations of low-degree and highdegree nodes to have similar distributions.

A different line of research has produced algorithms that separate the learning process for high and low-degree nodes. [53] embed degree-specific weights and hashing functions within the layers of GNNs that guarantee degree-aware node representations. [45] propose a degree-specific GNN layer, to avoid parameter sharing across nodes with different degrees, and a Bayesian teacher network that generates pseudo-labeled neighbors for low-degree nodes to increase their proximity to labels. [60] learn separate expert models for high and low-degree nodes with class and degree-balanced data

subsets, and distill the knowledge of these experts into two student models that are tailored to low and high-degree nodes.

There exist yet other solutions (e.g., based on adversarial learning, attention) that have been produced. [35] learn debiasing functions that distill and complement the GNN encodings of high and low-degree nodes, respectively. [33] leverage meta-learning to learn to learn representations for low-degree nodes in a locality-aware manner. [18] propose a label proximity score, which they find to be more strongly associated with performance than degree, and learn a new graph structure that reduces discrepancies in label proximity scores across nodes. [64] leverage an attention mechanism to enhance focus on low-degree nodes. Other methods include:

[65] modify the aggregation weights for neighboring node representations according to node degrees.

[54] learn custom message passing strategies for nodes with different degrees.

[14] causally determine if low-degree nodes should 'trust' messages from their neighbors.

[9] learn node representations that are invariant to shifts in local neighborhood distributions.

[32] optimize the graph's adjacency matrix to reduce an upper bound on degree bias with a minimal decrease in overall accuracy.

[8] locate anchor nodes that through the introduction of links with them can improve the representations of low-degree nodes.

[27] propose a popularity-weighted aggregator for Graph Convolutional Networks.

[19] leverage hop-aware attentive aggregation to attend differently to information at different distances.

[6] estimate the effect of influential high-degree nodes on the representations of low-degree nodes and remove this effect after each graph convolution.

[62] use adversarial learning to boost the influence of low-degree nodes.

A.4 Degree Bias from the Perspectives of Homophily and Topology

Prior research has connected degree bias to homophily and graph topology:

[59] provides a complementary perspective on the possible performance issues of GNNs that arise from degree disparities in graphs (e.g., low-degree nodes induce oversmoothing in homophilic graphs, while high-degree nodes induce oversmoothing in heterophilic networks). Oversmoothing is related to prediction homogeneity ( ∑ L l =0 β ( l ) i,c ′ ) 2 ; for homophilic networks, as the number of layers in a GNN increases (i.e., as oversmoothing occurs), ( ∑ L l =0 β ( l ) i,c ′ ) 2 gets closer to 0 (i.e., does not increase R i,c ′ ), thereby not inducing as much degree bias. In contrast, our theoretical analysis demonstrates that degree bias occurs without oversmoothing and is amplified by high local homophily.

[38] connects node distinguishability to node degree and homophily by analyzing the intra-class vs. inter-class embedding distance. We discuss similar quantities in §4.1 and §4.2. However, with the exception of Section 3.5, [38] considers the CSBM-H model in its theoretical analysis, which has pitfalls (as we discuss in §2). Moreover, unlike our work, [38] does not explicitly link the misclassification error of a node to its degree in a more general data and model setting.

[48] analyzes the effect of heterophily on GNNs via class separability, which it characterizes through neighborhood distributions and average node degree. Similar to [38], [48] only considers the HSBM model in its theoretical analysis.

[30] observes that GNN performance is lower for high-degree nodes under heterophily. We likewise observe this in Figure 5 for chameleon and squirrel (which are heterophilic networks). Moreover, our theoretical analysis explains why degree bias is not observed for heterophilic graphs. In §4.2, we explain that high-degree nodes in heterophilic networks do not have lower l -hop prediction homogeneity levels due to higher local heterophily; hence, we do not necessarily observe better performance for them compared to low-degree nodes.

B Proofs

B.1 Theorem 1

Proof. Misclassification occurs when ℓ ( M| i, c ) > ℓ ( M| i, c ′ ) .

If E [ Z ( L ) i,c ′ -Z ( L ) i,c ] < 0 (i.e., M generalizes in expectation), by Cantelli's inequality:

We use Cantelli's inequality, rather than Chebyshev's inequality, because Cantelli's inequality is sharper for one-sided bounds.

B.2 Theorem 2

Proof. Denoting the l -th term in the summation T ( l ) = P l rw XW ( l ) , T ( l ) i,c = ∑ j ∈V ( P l rw ) ij X j W ( l ) .,c . It follows by the linearity of expectation that:

Furthermore, by the linearity of variance:

Then, once again by the linearity of expectation and variance:

Consequently:

B.3 Theorem 3

Proof. Re-expressing the l -th term T ( l ) = P l sym XW ( l ) in the summation:

It follows by the linearity of expectation that:

Furthermore, by the linearity of variance:

Then, once again, by the linearity of expectation and variance:

Consequently:

B.4 Lemma 1

Proof. Define g ( Z ( L ) i [ t ]) = ∇ Z ( L ) i [ t ] ℓ [ t ]( M| i, c ) . For simplicity of notation, let x = Z ( L ) i [ t ] . It is sufficient to show that ∥ g ( x ) ∥ 2 ≤ λ . By simple derivation, ( g ( x )) i = -∑ a = i e x a ∑ b e x b , and for j = i , ( g ( x )) j = -e x j ∑ b e x b . Then, by H¨ older's inequality:

̸

̸

̸

̸

Thus, λ = 2

̸

B.5 Theorem 4

Proof. By the Lipschitz continuity of ℓ [ t ]( SYM | i, c ) (Lemma 1) and the triangle inequality:

By simple derivation, we see that ∂ℓ [ t ] ∂ W ( l ) [ t ] ( B [ t ]) = ( P l sym [ t ] X ) T ϵ [ t ] , where P l sym [ t ] ∈ R | B [ t ] |× N is the submatrix formed from the rows of P l sym that correspond to the nodes in B [ t ] . Then, by the sub-multiplicativity of the L 2 norm:

Similarly to the proof of Theorem 3:

Hence, ∥ ∥ ∥ ( P l sym ) i XX T ( P l sym [ t ] ) T ∥ ∥ ∥ 2 = √ D ii · ∥ ∥ ∥˜ χ ( l ) i [ t ] ∥ ∥ ∥ 2 , and:

B.6 Theorem 5

Proof. By the Lipschitz continuity of ℓ [ t ]( RW | i, c ) (Lemma 1) and the triangle inequality:

By simple derivation, we see that ∂ℓ [ t ] ∂ W ( l ) [ t ] ( B [ t ]) = ( P l rw [ t ] X ) T ϵ [ t ] , where P l rw [ t ] ∈ R | B [ t ] |× N is the submatrix formed from the rows of P l rw that correspond to the nodes in B [ t ] . Then, by the sub-multiplicativity of the L 2 norm:

Similarly to the proof of Theorem 2:

Hence, ∥ ∥ ∥ ( P l rw ) i XX T ( P l rw [ t ] ) T ∥ ∥ ∥ 2 = ∥ ∥ ∥ χ ( l ) i [ t ] ∥ ∥ ∥ 2 , and:

C Datasets

In our experiments, we use 8 real-world network datasets from [4], [42], and [41], covering diverse domains (e.g., citation networks, collaboration networks, online product networks, Wikipedia networks). We provide a description and statistics of each dataset in Table 4. All the datasets have node features and are undirected. For each node, we normalize its features to sum to 1, following [15] 1 . We were unable to find the exact class names and their label correspondence from the dataset documentation.

In all the citation network datasets, nodes represent documents, edges represent citation links, and features are a binary bag-of-words representation of documents. The classification task is to predict the topic of documents.

In the collaboration network datasets, nodes represent authors, edges represent coauthorships, and features are a binary bag-of-word representation of keywords from the authors' papers. The classification task is to predict the most active field of study for authors.

In the online product network datasets, nodes represent products, edges represent that two products are often purchased together, and features are a binary bag-of-word representation of product reviews. The classification task is to predict the category of products.

In the Wikipedia network datasets, nodes represent Wikipedia websites, edges represent hyperlinks between them, and features are a binary bag-of-word representation of informative nouns from the pages. The classification task is to predict the level of average daily traffic for pages.

We use PyTorch and PyTorch Geometric to load and process all datasets [40, 15]. Our usage of these libraries and datasets complies with their license. PyTorch and PyTorch Geometric are available under a torch-specific license 2 and MIT license 3 , respectively. Cora ML and CiteSeer are available under an MIT License 4 and can be found here: https://github.com/ abojchevski/graph2gauss/tree/master/data . CS, Physics, Amazon Photo, and Amazon Computers are available under an MIT License 5 and can be found here: https://github.com/ shchur/gnn-benchmark/tree/master/data/npz . chameleon and squirrel are available under a GLP-3.0 license 6 and can be found here: https://graphmining.ai/datasets/ptg/wiki .

While these datasets are widely used, we did not obtain explicit consent from any data subjects whose data the datasets may contain. To the best of our knowledge (via manual sampling and inspection), the datasets do not contain any personally identifiable information or offensive content.

D Models

In our experiments, we transductively learn and compute node representations using encoders based on Graph Convolutional Networks (GCNs) [24], GraphSAGE [17], and Graph Attention Networks (GATs) [46].

In all cases, we use general message-passing GNNs M [16], which include separate parameters for source and target nodes and residual connections; in particular, for layer l :

where H ( l ) ∈ R N × d ( l ) are the l -th layer node representations (with H (0) = X and d ( L ) = C ), σ ( l ) is an instance-wise non-linearity (with σ ( L ) being softmax), P ( l ) ∈ R N × N is a graph filter, and W ( l ) 1 , W ( l ) 2 , W ( l ) 3 ∈ R d ( l -1) × d ( l ) are the l -th layer model parameters.

We consider the following special cases of M which vary with respect to their graph filter:

RW: ∀ l ∈ N ≤ L , P ( l ) = P rw = D -1 A (i.e., the uniform random walk transition matrix), where D is the diagonal degree matrix with entries D ii = ∑ j ∈V A ij .

SYM: ∀ l ∈ N ≤ L , P ( l ) = P sym = D -1 2 AD -1 2 .

ATT: ∀ l ∈ N ≤ L , P ( l ) is a graph attentional operator with default hyperparameters and a single head [46].

Each encoder has three layers (64-dimensional hidden layers), with a ReLU nonlinearity in between layers. We do not use any regularization (e.g., Dropout, BatchNorm). The encoders are explicitly trained for node classification with the cross-entropy loss and the Adam optimizer [23] with fullbatch gradient descent on the training set. We use a learning rate of 5e-3. We further use a random node split of 1000-500-rest for test-val-train. We train all encoders until they reach the training accuracy of MAJWL and select the model parameters with the highest validation accuracy. Although we do not do any hyperparameter tuning, the test accuracy values indicate that the encoders are welltrained.

We use PyTorch [40] and PyTorch Geometric [15] 7 to train all the encoders on a single NVIDIA GeForce GTX Titan Xp Graphic Card with 12196MiB of space on an internal cluster. On average (with respect to the datasets), the median time per training epoch was 0.05 seconds. Thus, the estimated total compute is:

These experiments were run a few times (e.g., due to the discovery of bugs), but the full research project did not involve other experiments not reported in the paper. We used a single CPU worker to load datasets and plot results. The datasets take up 0.2975 MB of disk space.

E Additional Degree Bias Plots

Unlike the other datasets, we do not observe degree bias for chameleon and squirrel because these datasets are heterophilic. We intentionally include these datasets to draw contrast to the other, homophilic datasets and validate our theory. For example, in §4.2, we explain that high-degree nodes in heterophilic networks do not have more negative l -hop prediction homogeneity levels due to higher local heterophily levels; hence, we do not necessarily observe better performance for them compared to low-degree nodes.

0

20

40

60

Degree

0

5

10

15

20

Test loss

RW

0

20

40

60

Degree

0

5

10

15

20

Test loss

SYM

0

20

40

60

Degree

0

5

10

15

20

Test loss

ATT

CORA_ML

0

25

50

75

100

125

Degree

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Test loss

RW

0

25

50

75

100

125

Degree

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Test loss

SYM

0

25

50

75

100

125

Degree

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Test loss

ATT

CS

0

50

100

150

Degree

0

1

2

3

4

Test loss

RW

0

50

100

150

Degree

0

1

2

3

4

Test loss

SYM

0

50

100

150

Degree

0

1

2

3

4

Test loss

ATT

PHYSICS

0

200

400

600

800

Degree

0

2

4

6

8

10

12

Test loss

RW

0

200

400

600

800

Degree

0

2

4

6

8

10

12

Test loss

SYM

0

200

400

600

800

Degree

0

2

4

6

8

10

12

Test loss

ATT

PHOTO

0

200

400

600

Degree

0

5

10

15

20

Test loss

RW

0

200

400

600

Degree

0

5

10

15

20

Test loss

SYM

0

200

400

600

Degree

0

5

10

15

20

Test loss

ATT

COMPUTERS

0

20

40

60

80

Degree

0

5

10

15

20

Test loss

RW

0

20

40

60

80

Degree

0

5

10

15

20

Test loss

SYM

0

20

40

60

80

Degree

0

5

10

15

20

Test loss

ATT

CHAMELEON

0

50

100

150

200

250

Degree

0

10

20

30

40

50

Test loss

RW

0

50

100

150

200

250

Degree

0

10

20

30

40

50

Test loss

SYM

0

50

100

150

200

250

Degree

0

10

20

30

40

50

Test loss

ATT

F Additional Visual Summaries of Theoretical Results

In the plots below, we consider low-degree nodes to be the 100 nodes with the smallest degrees and high-degree nodes to be the 100 nodes with the largest degrees. Each point in the plots in the left column corresponds to a test node representation and its color represents the node's class. The plots in the left column are based on a single random seed, while the plots in the middle and right columns are based on 10 random seeds. RW representations of low-degree nodes often have a larger variance than high-degree node representations, while SYM representations of low-degree nodes often have a smaller variance. Furthermore, SYM generally adjusts its training loss on low-degree nodes less rapidly.

The training loss curves in Figure 6 still support our theoretical analysis. Theorem 4 reveals that for SY M , node degree and the (degree-discounted) expected feature similarity ˜ χ i affects the rate of learning. On the other hand, Theorem 5 indicates that for RW , while we do not expect node degree to impact the rate of learning, the expected feature similarity χ i is still influential. Hence, interpreting Theorems 4 and 5 jointly, we expect and accordingly observe that the orange curve for SYM has a steeper rate of decrease relative to the orange curve for RW as the number of epochs increases.

0.15

0.10

0.05

0.00

0.05

PC1

0.10

0.05

0.00

0.05

0.10

PC2

RW

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

250

300

Trace of sample covariance

of test representations

RW

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

Training loss

RW

Low-degree nodes

High-degree nodes

450

475

500

0.15

0.10

0.05

0.00

0.05

PC1

0.10

0.05

0.00

0.05

0.10

PC2

SYM

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

250

300

Trace of sample covariance

of test representations

SYM

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

Training loss

SYM

Low-degree nodes

High-degree nodes

450

475

500

0.15

0.10

0.05

0.00

0.05

PC1

0.10

0.05

0.00

0.05

0.10

PC2

ATT

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

250

300

Trace of sample covariance

of test representations

ATT

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

Training loss

ATT

Low-degree nodes

High-degree nodes

450

475

500

CORA_ML

CS

0.100

0.075

0.050

0.025 0.000

0.025

PC1

0.05

0.00

0.05

0.10

PC2

RW

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

100

200

300

400

Trace of sample covariance

of test representations

RW

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training loss

RW

Low-degree nodes

High-degree nodes

450

475

500

0.100

0.075

0.050

0.025 0.000

0.025

PC1

0.05

0.00

0.05

0.10

PC2

SYM

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

100

200

300

400

Trace of sample covariance

of test representations

SYM

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training loss

SYM

Low-degree nodes

High-degree nodes

450

475

500

0.100

0.075

0.050

0.025 0.000

0.025

PC1

0.05

0.00

0.05

0.10

PC2

ATT

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

100

200

300

400

Trace of sample covariance

of test representations

ATT

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training loss

ATT

Low-degree nodes

High-degree nodes

450

475

500

PHYSICS

0.100 0.075 0.050 0.0250.000 0.025 0.050

PC1

0.05

0.00

0.05

0.10

PC2

RW

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

Trace of sample covariance

of test representations

RW

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

Training loss

RW

Low-degree nodes

High-degree nodes

450

475

500

0.100 0.075 0.050 0.0250.000 0.025 0.050

PC1

0.05

0.00

0.05

0.10

PC2

SYM

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

Trace of sample covariance

of test representations

SYM

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

Training loss

SYM

Low-degree nodes

High-degree nodes

450

475

500

0.100 0.075 0.050 0.0250.000 0.025 0.050

PC1

0.05

0.00

0.05

0.10

PC2

ATT

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

Trace of sample covariance

of test representations

ATT

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

Training loss

ATT

Low-degree nodes

High-degree nodes

450

475

500

PHOTO

0.10

0.05

0.00

0.05

0.10

PC1

0.075

0.050

0.025

0.000

0.025

0.050

0.075

PC2

RW

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

250

300

Trace of sample covariance

of test representations

RW

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

Training loss

RW

Low-degree nodes

High-degree nodes

450

475

500

0.10

0.05

0.00

0.05

0.10

PC1

0.075

0.050

0.025

0.000

0.025

0.050

0.075

PC2

SYM

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

250

300

Trace of sample covariance

of test representations

SYM

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

Training loss

SYM

Low-degree nodes

High-degree nodes

450

475

500

0.10

0.05

0.00

0.05

0.10

PC1

0.075

0.050

0.025

0.000

0.025

0.050

0.075

PC2

ATT

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

200

250

300

Trace of sample covariance

of test representations

ATT

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.0

0.5

1.0

1.5

2.0

Training loss

ATT

Low-degree nodes

High-degree nodes

450

475

500

COMPUTERS

0.10

0.05

0.00

0.05

PC1

0.100

0.075

0.050

0.025

0.000

0.025

0.050

0.075

PC2

RW

Low-degree nodes

High-degree nodes

0

500

1000

1500

2000

2500

3000

Epochs of training

0

500

1000

1500

2000

2500

Trace of sample covariance

of test representations

RW

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

5.0

2.5

0.0

2.5

5.0

7.5

10.0

Training loss

RW

Low-degree nodes

High-degree nodes

2950

2975

3000

0.10

0.05

0.00

0.05

PC1

0.100

0.075

0.050

0.025

0.000

0.025

0.050

0.075

PC2

SYM

Low-degree nodes

High-degree nodes

0

500

1000

1500

2000

2500

3000

Epochs of training

0

500

1000

1500

2000

2500

Trace of sample covariance

of test representations

SYM

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

5.0

2.5

0.0

2.5

5.0

7.5

10.0

Training loss

SYM

Low-degree nodes

High-degree nodes

2950

2975

3000

0.10

0.05

0.00

0.05

PC1

0.100

0.075

0.050

0.025

0.000

0.025

0.050

0.075

PC2

ATT

Low-degree nodes

High-degree nodes

0

500

1000

1500

2000

2500

3000

Epochs of training

0

500

1000

1500

2000

2500

Trace of sample covariance

of test representations

ATT

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

5.0

2.5

0.0

2.5

5.0

7.5

10.0

Training loss

ATT

Low-degree nodes

High-degree nodes

2950

2975

3000

CHAMELEON

0.15

0.10

0.05

0.00

0.05

0.10

PC1

0.15

0.10

0.05

0.00

0.05

0.10

PC2

RW

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

Trace of sample covariance

of test representations

RW

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Training loss

RW

Low-degree nodes

High-degree nodes

450

475

500

0.15

0.10

0.05

0.00

0.05

0.10

PC1

0.15

0.10

0.05

0.00

0.05

0.10

PC2

SYM

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

Trace of sample covariance

of test representations

SYM

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Training loss

SYM

Low-degree nodes

High-degree nodes

450

475

500

0.15

0.10

0.05

0.00

0.05

0.10

PC1

0.15

0.10

0.05

0.00

0.05

0.10

PC2

ATT

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

50

100

150

Trace of sample covariance

of test representations

ATT

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Training loss

ATT

Low-degree nodes

High-degree nodes

450

475

500

0.10

0.05

0.00

0.05

0.10

PC1

0.10

0.05

0.00

0.05

0.10

PC2

RW

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

100

200

300

400

500

600

Trace of sample covariance

of test representations

RW

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Training loss

RW

Low-degree nodes

High-degree nodes

450

475

500

0.10

0.05

0.00

0.05

0.10

PC1

0.10

0.05

0.00

0.05

0.10

PC2

SYM

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

100

200

300

400

500

600

Trace of sample covariance

of test representations

SYM

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Training loss

SYM

Low-degree nodes

High-degree nodes

450

475

500

0.10

0.05

0.00

0.05

0.10

PC1

0.10

0.05

0.00

0.05

0.10

PC2

ATT

Low-degree nodes

High-degree nodes

0

100

200

300

400

500

Epochs of training

0

100

200

300

400

500

600

Trace of sample covariance

of test representations

ATT

Low-degree nodes

High-degree nodes

0

20

40

60

80

100

Epochs of training

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Training loss

ATT

Low-degree nodes

High-degree nodes

450

475

500

G Additional Inverse Collision Probability Plots

0

20

40

60

Degree

0.4

0.6

0.8

Inverse collision probability

RW

0

20

40

60

Degree

0

10

20

30

40

Inverse collision probability

SYM

0

20

40

60

Degree

0.4

0.6

0.8

Inverse collision probability

ATT

CORA_ML

0

50

100

Degree

0.4

0.6

0.8

1.0

Inverse collision probability

RW

0

50

100

Degree

0

20

40

60

80

100

Inverse collision probability

SYM

0

50

100

Degree

0.4

0.6

0.8

1.0

Inverse collision probability

ATT

CS

0

50

100

150

Degree

0.4

0.6

0.8

1.0

Inverse collision probability

RW

0

50

100

150

Degree

0

50

100

150

Inverse collision probability

SYM

0

50

100

150

Degree

0.4

0.6

0.8

1.0

Inverse collision probability

ATT

PHYSICS

0

250

500

750

Degree

0.4

0.6

0.8

1.0

Inverse collision probability

RW

0

250

500

750

Degree

0

200

400

600

800

Inverse collision probability

SYM

0

250

500

750

Degree

0.4

0.6

0.8

1.0

Inverse collision probability

ATT

PHOTO

0

200

400

600

Degree

0.4

0.6

0.8

1.0

Inverse collision probability

RW

0

200

400

600

Degree

0

200

400

600

Inverse collision probability

SYM

0

200

400

600

Degree

0.4

0.6

0.8

1.0

Inverse collision probability

ATT

COMPUTERS

0

20

40

60

80

Degree

0.0

0.2

0.4

0.6

0.8

1.0

Inverse collision probability

RW

0

20

40

60

80

Degree

0

20

40

60

80

Inverse collision probability

SYM

0

20

40

60

80

Degree

0.4

0.6

0.8

1.0

Inverse collision probability

ATT

CHAMELEON

0

100

200

Degree

0.0

0.2

0.4

0.6

0.8

1.0

Inverse collision probability

RW

0

100

200

Degree

0

50

100

150

200

250

Inverse collision probability

SYM

0

100

200

Degree

0.4

0.6

0.8

1.0

Inverse collision probability

ATT

H Training-Time Degree Bias: Random Walk Graph Filter

We now demonstrate that during each step of training RW with gradient descent, compared to SYM, the loss of low-degree nodes in S is not necessarily adjusted more slowly. We define ∀ l ∈ N ≤ L , χ ( l ) i ∈ R | B [ t ] | , where for m ∈ B [ t ] , ( χ ( l ) i [ t ] ) m = E j ∼N ( l ) ( i ) ,k ∼N ( l ) ( m ) [ X j X T k ] . In effect, ( χ ( l ) i [ t ] ) m captures the expected similarity between the raw features of nodes j and k with respect to the l -hop random walk distributions of i ∈ V and m ∈ B [ t ] .

Theorem 5. The change in loss for i after an arbitrary training step t obeys:

For RW, the change (either increase or decrease) in loss for i after an arbitrary training step does not necessarily have a smaller magnitude if i is low-degree. However, the L -hop neighborhoods of low-degree nodes still often have less overlap with the neighborhoods of training nodes, which can constrain the rate at which the loss for i changes. We confirm these findings empirically in Figure 2 and §F. For all the datasets, the blue curve for RW has a less steep rate of decrease than the blue curve for SYM as the number of epochs increases. However, for RW itself, the orange curve generally descends more quickly than the blue curve, with the exception of the heterophilic chameleon and squirrel datasets, for which the features of nodes in the neighborhoods of highdegree nodes and training nodes are dissimilar. Therefore, models do not learn more rapidly for high-degree nodes under heterophily. These findings support hypothesis (H2) . Our results for RW may also apply to ATT when low-degree nodes are generally attended to less.

I Achieving Maximum Training Accuracy

CITESEER

400

425

450

475

500

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Mean absolute gradient

1e

7

RW

SYM

ATT

400

425

450

475

500

Epoch

0.96

0.98

1.00

1.02

1.04

Training accuracy

RW

SYM

ATT

MAJWL

We now empirically show that SYM (despite learning at different rates for low vs. high-degree nodes), RW, and ATT can achieve their maximum possible training accuracy (i.e., the accuracy of a majority voting-classifier MAJWL). Furthermore, per our experiments, the accuracy of MAJWL is often close to 100%, indicating that the WL test does not significantly limit the accuracy of SYM, RW, and ATT in practice.

Per [57], because the expressive power of SYM, RW, and ATT are limited by the WL test, an upper bound on their training accuracy is the accuracy of a majority voting-classifier MAJWL applied to WL node colorings (for details on how to compute colorings, see §II.A and §VI.B of [68]). In particular, if the WL test produces the colors c ∈ K | S | for nodes in S , MAJWL predicts node i to have the label ̂ Y i = MAJWL ( i, X , A ) = mode { Y j | j ∈ V , c j = c i } . However, Figure 15 and §J reveal that as the number of training epochs increases, the training accuracy of SYM, RW, and ATT reach the accuracy of MAJWL. Because the accuracy of MAJWL is often close to 100%, our experiments suggest that insufficient expressive power likely does not contribute to degree bias, drawing doubt to hypothesis (H7) .

Empirically inspecting the model gradients, compared to ATT, as the number of training epochs increases, the mean absolute gradients of SYM and RW are comparably small but often decrease more slowly or fluctuate. To understand this, we can analytically inspect the gradients of RW:

( P l rw [ t ] ) T (for l > 0 ) often has numerous eigenvalues around 0, which can yield gradients ∂ℓ [ t ] ∂ W ( l ) [ t ] ( B [ t ]) with a small magnitude even when ∥ ϵ [ t ] ∥ is not small. The same analysis holds for SYM. As such, SYM and RW may get trapped in suboptimal minima during training, yielding slower or unstable convergence; in contrast, because ATT has a dynamic filter, its training loss rarely exhibits slow or unstable convergence.

J Additional Training Loss Plots

400

425

450

475

500

Epoch

0.0

0.5

1.0

1.5

Mean absolute gradient

1e

8

RW

SYM

ATT

400

425

450

475

500

Epoch

0.96

0.98

1.00

1.02

1.04

Training accuracy

RW

SYM

ATT

MAJWL

CORA_ML

400

425

450

475

500

Epoch

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Mean absolute gradient

1e

8

RW

SYM

ATT

400

425

450

475

500

Epoch

0.96

0.98

1.00

1.02

1.04

Training accuracy

RW

SYM

ATT

MAJWL

CS

400

425

450

475

500

Epoch

0

1

2

3

4

5

6

Mean absolute gradient

1e

9

RW

SYM

ATT

400

425

450

475

500

Epoch

0.96

0.98

1.00

1.02

1.04

Training accuracy

RW

SYM

ATT

MAJWL

PHYSICS

400

425

450

475

500

Epoch

0

1

2

3

4

5

6

Mean absolute gradient

1e

6

RW

SYM

ATT

400

425

450

475

500

Epoch

0.96

0.98

1.00

1.02

1.04

Training accuracy

RW

SYM

ATT

MAJWL

PHOTO

2900

2925

2950

2975

3000

Epoch

0.0

0.5

1.0

1.5

2.0

2.5

Mean absolute gradient

1e

6

RW

SYM

ATT

2900

2925

2950

2975

3000

Epoch

0.96

0.98

1.00

1.02

1.04

Training accuracy

RW

SYM

ATT

MAJWL

COMPUTERS

CHAMELEON

400

425

450

475

500

Epoch

0

1

2

3

4

5

Mean absolute gradient

1e

6

RW

SYM

ATT

400

425

450

475

500

Epoch

0.92

0.94

0.96

0.98

1.00

Training accuracy

RW

SYM

ATT

MAJWL

400

425

450

475

500

Epoch

0

2

4

6

8

Mean absolute gradient

1e

6

RW

SYM

ATT

400

425

450

475

500

Epoch

0.96

0.98

1.00

1.02

1.04

Training accuracy

RW

SYM

ATT

MAJWL

SQUIRREL

K Connecting Inverse Collision Probability to Node Degree

Our theoretical analysis may be improved upon by establishing a rigorous connection, or a lack thereof, between the inverse collision probability of a node and its degree.

Via some preliminary analysis, we find that it is possible to express the inverse collision probability of i (i.e., equivalently express the sum of l -hop collision probabilities) in terms of D ii . In particular, we can show: ∑ L l =0 α ( l ) i = α (0) i + ∑ L l =1 ∑ j ∈V [( P l rw ) ij ] 2 = 1 + 1 D 2 ii ∑ L l =1 ∑ j ∈V [ ∑ k ∈N ( i ) ( P l -1 rw ) kj ] 2 . As before, we can see that the inverse collision probability is larger (and thus R i,c ′ is larger) when D ii is larger and ∑ L l =1 ∑ j ∈V [ ∑ k ∈N ( i ) ( P l -1 rw ) kj ] 2 ∈ o ( D 2 ii ) . However, because ∑ k ∈N ( i ) depends on D ii , this expression does not completely isolate the impact of D ii on the inverse collision probability. A similar expression can be derived for SYM by expressing: P sym = D 1 2 P rw D -1 2 .

Alternatively, one may consider ∑ L l =0 α ( l ) i ≤ ∑ ∞ l =0 α ( l ) i , and then plug in the steady-state probabilities of uniform random walks on graphs. However, as l → ∞ , ( P l rw ) ij = D jj 2 |E| only depends on D jj (not D ii , as desired). It is also possible to upper bound the inverse collision probability as: 1 / ∑ l =0 α ( l ) i ≤ 1 / ( α (0) i + α (1) i ) = 1 / (1 / D ii +1) , which is in terms of D ii ; however, we cannot use this upper bound to lower bound R i,c ′ . The collision probabilities themselves are intimately related to more global properties of graphs; for instance, via eigendecomposition, [( P l rw ) ij ] 2 ≤ λ 2 l , where λ < 1 is the eigenvalue of P rw with the largest magnitude [36]. Then, the inverse collision probability is strictly greater than 1 ∑ L l =0 |V| λ 2 l .

This being said, our paper argues that the inverse collision probability is a more fundamental quantity that influences what the fair graph learning community has termed 'degree bias' than degree alone, which we have shown is positively associated with the inverse collision probability.

L Limitations and Future Directions

Survey While we aimed to be thorough in our survey of prior papers on degree bias, it is inevitable that we missed some relevant work. In addition, we extract hypotheses for the origins of degree bias from the main bodies of papers; it is possible that the hypotheses do not fully or accurately reflect the current perspectives of the papers' authors.

Theoretical analysis Our theoretical analysis is limited to linearized message-passing GNNs. While this is a common practice in the literature [51, 7, 39], it is a strong simplifying assumption. We empirically validate our theoretical findings on GNNs with non-linear activation functions, but our paper does not address possible sources of degree bias related to these non-linearities, which would be interesting to investigate in future work. Towards this, a possible option is to assume that node features are drawn from a Gaussian distribution and derive precise high-dimensional asymptotics for degree bias in non-linear GNNs using the Gaussian equivalence theorem, as in [1]. Our assumptions that GNNs generalize in expectation (Theorem 1) and the variance of node representations is finite (Theorems 2 and 3) are not overly strong assumptions in practice.

Furthermore, our paper focuses on node classification. However, our findings readily lend insight into the origins of degree bias in link prediction. For example, if one uses node representations and an inner-product decoder to predict links between nodes, our results indicate that:

In the random walk filter case, link prediction scores between low-degree nodes will suffer from higher variance because low-degree node representations have higher variance (Theorem 2). Hence, Theorem 1 suggests that predictions for links between low-degree nodes will have a higher misclassification error.

In the symmetric filter case, our proof of Theorem 3 suggests that the link prediction scores between high-degree nodes will be over-calibrated (i.e., disproportionately large) because highdegree node representations have a larger magnitude (i.e., approximately proportional to the square root of their degree). Hence, over-optimistic and possibly inaccurate links will be predicted between high-degree nodes.

The labels and evaluation for link prediction can confound intuition. Unlike node classification, the labels for link prediction (i.e., the existence or not of a link) make the task naturally imbalanced with respect to node degree; high-degree nodes have a much higher rate of positive links than lowdegree nodes. This association between degree and positive labels can influence the misclassification error. Ultimately, more rigorous theoretical analysis and experimentation are needed to confirm the hypothesized implications of node degree for link prediction performance. Similarly, more research is required to understand the implications of our findings for degree bias in the context of graph classification.

Furthermore, our theoretical analysis does not encompass heterogeneous graphs. In our literature survey, we cover works that establish the issue of degree bias for knowledge graph predictions and embeddings (e.g., [5, 43]). Our theoretical analysis is general and covers diverse message-passing GNNs, and can be extended to heterogeneous networks if messages aggregated from different edge types are subsequently linearly combined. In this setting, R i,c ′ can be computed as the sum of the prediction homogeneity quantities ( ∑ L l =0 β ( l ) i,c ′ ) 2 for each edge type divided by the sum of the collision probability quantities ∑ j ∈V [( P l rw ) ij ] 2 for each edge type.

Empirical validation We sought to be transparent throughout our paper regarding misalignments between empirical and theoretical findings. Our experiments focus on the transductive learning setting; it would be valuable to validate our theoretical findings in the inductive learning setting as well. Furthermore, while we aimed to cover diverse domains (e.g., citation networks, collaboration networks, online product networks, Wikipedia networks), as well as homophilic and heterophilic networks, it remains to identify the shortcomings of our theoretical findings for heterogeneous and directed networks.

Principled roadmap To do justice to studying the origins of degree bias in GNNs, which our paper reveals has various and sometimes conflicting understandings, we limit the scope of our paper to understanding the root causes of degree bias, not providing a concrete algorithm to alleviate it.

Instead, we offer a principled roadmap based on our theoretical findings to address degree bias in the future. We further comment on the limitations of algorithmic solutions to degree bias in our Broader Impacts section below.

M Broader Impacts

Our paper touches upon issues of discrimination, bias, and fairness, with the goal of advancing justice in graph learning. In particular, our analysis of the origins of degree bias in GNNs seeks to inform principled approaches to mitigate unfair performance disparities faced by low-degree nodes in networks (e.g., lowly-cited authors, junior researchers, niche product and content creators). Despite our focus on fairness, our work can still have negative societal impacts in malicious contexts. For example, alleviating the degree bias of GNNs that are intended to surveil individuals can further violate the privacy of low-degree individuals. Ultimately, performance disparities should only be mitigated when the task is aligned with the interests and well-being of marginalized individuals; we explicitly do not support evaluating or mitigating degree bias to ethics-wash inherently harmful applications of graph learning. Furthermore, any algorithm proposed to alleviate degree bias will not be a 'silver bullet' solution; graph learning practitioners must adopt a sociotechnical approach: (1) critically examining the societal factors that contribute to their networks have degree disparities to begin with, and (2) monitoring their GNNs in deployment and continually adapting their degree bias evaluations and algorithms. In addition, alleviating degree bias does not necessarily address other forms of unfairness in graph learning, e.g., equal opportunity with respect to protected social groups [2], dyadic fairness [25], preferential attachment bias [44]; fairness algorithms are contextual and not one-size-fits-all.

Responsible Research Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See §L.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the theorems, formulas, and proofs in the paper are numbered and crossreferenced. The assumptions are clearly stated in the main body of the paper and referenced in the proofs of theorems. Formal proofs are provided in the appendix; while proof sketches are not included in the main body of the paper, we provide extensive interpretation of theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details to reproduce experiments are provided in the main body of the paper, §D, and §C. The code is released via Github.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is released via Github. The data are available through PyTorch Geometric [15], and data processing details are provided in §C.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See §D and §C.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are reported over 10 random seeds in all figures except for the PCA plots. The factors of variability include model parameter initialization and training dynamics. All error bars are 1-sigma and represent the standard deviation (not standard error) of the mean. We implicitly assume that errors are normally-distributed. Error bars are computed using PyTorch's std function [40]. We provide this information at the beginning of §3.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See §D.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See §M.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See §C. The original paper that produced each dataset is cited. Links to libraries and datasets are included; the versions of software dependencies can be found in the requirements.txt file via Github. We also provide the name and a link to the license (which includes terms of use and copyright information, where applicable) of each library and dataset.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release our code via Github under an MIT license. We provide a README that explains how to reproduce our experiments. Training, license compliance, and consent details are included in §D. Limitations are discussed in §L.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of degree bias for node classification using graph neural networks (GNNs). Much prior work has shown that GNNs tend to be much more accurate for nodes with higher degree than for lower degree. The authors survey 38 papers that pose hypotheses, sometimes contradictory, for why this degree bias exists. In this paper, the authors provide theoretical analyses on linearized versions of message-passing GNNs using random walk (RW) and symmetric (SYM) graph filters. Their bounds yield some insights on the origins of degree bias at both test time and training time and provide evidence to support some hypotheses from prior work but not others. Finally, they run experiments using the RW, SYM, and graph attention (ATT) filters on a variety of data sets, confirming their theoretical results. They conclude by providing some criteria that mitigation strategies for degree bias should target.\n\nSTRENGTHS:\n- Provides a very general theoretical analysis on the misclassification probabilities of nodes using message-passing GNNs with RW and SYM graph filters.\n- Empirical results support predictions from theoretical results.\n- Theoretical and empirical results yield insights as to what types of mitigation strategies for degree bias may be most effective.\n- Thoroughly explores hypotheses for degree bias in the existing literature and connects the theoretical results in this paper to provide evidence for some hypotheses and against others.\n- Excellent use of hyperlinks to guide reader back and forth between portions of the supplement, figures, and body text. I really enjoyed reading this paper and would likely have found it much more frustrating if I had to navigate back and forth manually.\n- Addresses a topic of great interest to the NeurIPS community, as illustrated by the large body of recent work on the topic.\n\nWEAKNESSES:\n- Theorem 2 bounds the squared inverse coefficient of variation $R_{i,c^′}$ based on the inverse collision probability $1/\\sum_{l=0} \\alpha_i^{(l)}$. There is no relationship or bound on the inverse collision probability as a function of node degree beyond a single layer, with the authors providing only empirical results. Thus, the theoretical results are not \"end to end\", with this one link between misclassification probability and node degree missing.\n- Analysis is for linearized versions of GNNs and not the GNNs themselves.\n\nMinor presentation issues:\n- Error bars in Figure 1 and similar figures in Appendix E are not explained in the figure caption. I see that they are explained later in the beginning of Section 3. I suggest adding (or moving) the description to the figure caption.\n- Figure 2: Caption lists FAIR ATT, but those plots are not shown.\n\nQUESTIONS:\n1. Of the four target criteria you present in Section 6, is there any ordering or predicted importance that you would recommend future work to target?\n2. I noted the lack of a relationship between inverse collision probability and node degree as a weakness. Do you have any potential leads on whether it would be possible to bound this based on other quantities in the graph? This would allow your theoretical results to potentially be \"end to end\".\n\nLIMITATIONS:\nLimitations are discussed in the supplementary material. The paper would be strengthened if a short summary of the main limitations (maybe 2 or 3 lines) could be added to the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
The paper provides a comprehensive analysis of the origins of degree bias in message-passing GNNs, proving that high-degree test nodes have a lower probability of misclassification, regardless of how GNNs are trained.  They surveyed 38 papers on degree bias and found that existing hypotheses are often not rigorously validated and can be contradictory.  They also show that some GNNs may adjust their loss on low-degree nodes more slowly during training, but with sufficient training epochs, these models can achieve their maximum possible training accuracy. The theoretical findings are validated on eight real-world networks, and a roadmap to alleviate degree bias is proposed.

STRENGTHS:
+This paper explores the important issue of the origins of degree bias in Graph Neural Networks (GNNs). 

+The authors conduct a comprehensive survey on degree bias, carefully identifying and validating existing hypotheses while pointing out those that are contradictory or lack validation. 

+The study examines the effects of different graph filters and analyzes degree bias separately during both test time and training time. 

+The paper provides a thorough theoretical analysis and supports its claims with empirical studies on eight datasets.

WEAKNESSES:
-It appears that the findings in this paper are limited to homophilous graphs, as the results do not extend to heterophilic graphs, according to the paper's analysis. 

-The current analysis is based on graphs with a single edge type. Can these findings be expanded to heterogeneous graphs, where the edges can have different types? It could be worth studying to explore this possibility.

QUESTIONS:
Please address the questions regarding weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the causes and characteristics of degree bias in graph neural networks (GNNs), particularly in node classification tasks. It contributes to the field by providing a rigorous theoretical analysis supported by empirical evidence across multiple datasets, revealing that degree bias is influenced by factors such as homophily and diversity of neighbors. The study also proposes a structured approach to mitigate this bias, enhancing the fairness and effectiveness of GNN applications in social and recommendation systems.\n\nSTRENGTHS:\noriginality: medium\nquality: good\nclarity: good\nsignificance: medium\n\nWEAKNESSES:\nSome degree-related papers should be included and discussed in this paper, for example:\n\n[1] takes a unified view to explain the over-smoothing and heterophily problems simultaneously by profiling nodes with two metrics: the relative degree of a node (compared to its neighbors) and the node-level heterophily.\n\n[2] investigates how does node degree, homophily and class variance influence the node distinguishability.\n\n[3] finds that the effectiveness of graph convolution operations in enhancing separability is determined by the Euclidean distance of the neighborhood distributions and the square root of the average node degree. Furthermore, they find that topological noise negatively affects separability by effectively lowering the average node degree.\n\n[4] observes that the prediction accuracy of high-degree nodes is usually significantly lower under heterophily and the hypothesis that \"nodes with higher degrees are naturally favored by GNNs\" no longer holds. In fact, the degree-wise bias is more sensitive, but not necessarily beneficial, to high-degree nodes with regard to varying graph conditions.\n\n\n\n[1] Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. In2022 IEEE International Conference on Data Mining (ICDM) 2022 Nov 28 (pp. 1287-1292). IEEE.\n\n[2] When Do Graph Neural Networks Help with Node Classification? Investigating the Homophily Principle on Node Distinguishability. Advances in Neural Information Processing Systems. 2024 Feb 13;36.\n\n[3] Understanding Heterophily for Graph Neural Networks. In Forty-first International Conference on Machine Learning.\n\n[4] Liao N, Liu H, Zhu Z, Luo S, Lakshmanan LV. Benchmarking Spectral Graph Neural Networks: A Comprehensive Study on Effectiveness and Efficiency. arXiv preprint arXiv:2406.09675. 2024 Jun 14.\n\nQUESTIONS:\n1. How does homophily relate to degree bias? I expect some conclusions in this paper but cannot find them.\n\n2. The \"expected similarity of the neighborhoods\" in section 5 is very similar to the similarity matrix defined in [5]\n\n3. The criteria proposed in section 6 should be verified in real-world datasets.\n\n\n\n\n[5] Revisiting heterophily for graph neural networks. Advances in neural information processing systems. 2022 Dec 6;35:1362-75.\n\nLIMITATIONS:\nthe authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGiven the wide-adoption of GNN-based node classification and the potential risk of degree-related bias, this paper systematically reviews the degree-bias paper in previous literature and proposes a theoretical framework to analyze the degree-related bias. Several insightful observations have been drawn with empirical verification.\n\nSTRENGTHS:\n(1) A systematic investigation of a long-standing yet not addressed research problem, degree bias. Comprehensive summary of previous works on this has been presented\n\n(2) The degree-related bias is urgently related to social benefit.\n\nWEAKNESSES:\n(1) Some experimental results are not supportive of some claims.\n\n(2) Some theoretical analyses lack intuitive justification from a graph topology perspective, e.g., the connection with network homophily/heterophily.\n\nQUESTIONS:\n(1) Since the degree bias focused on here is node classification, would it be better to note this special case somewhere in the paper, or at least for the title, would it be better to reword and limit the task scope to node classification? Is there any insight or potential thought on degree-related bias in link prediction and graph classification (here the degree would correspond to graph density)?\n\n(2) In Figure 5, when looking at the Squirrel and Chameleon datasets, there is no significant difference in high-degree and low-degree test loss, is there a specific reason for their difference from the other datasets?\n\n(3) Maybe it is a typo, but in the Equation at the bottom of page 4, if we unify different layers' weight transformation matrices as of dimension $\\mathbb{R}^{d^0\\times C}$, would it cause some dimension mismatch during matrix multiplication? Why does every layer of convolution consider the project from space $\\mathbb{R}^{d^0}$ to $\\mathbb{R}^{C}$? Moreover, it might be better to add back the Equation number.\n\n(4) It is not so clear about \"the diversity of neighborhoods\" until the formal definition in Line 168. It might be better to provide some concise illustrations to help readers understand the diversity of neighbors when it was mentioned for the very first time.\n\n(5) Based on Theorem 2, in addition to decreasing the negative $\\sum_{l=0}^{L}\\beta_{i, c'}^{(l)}$ so that $R_{i, c'}$ could increase, could we also increase the positive $\\sum_{l=0}^{L}\\beta_{i, c'}^{(l)}$ so that $R_{i, c'}$ could increase? Are these two scenarios corresponding to some graph local structure? Based on my understanding, as $\\beta_{i, c'}^{(l)}$ measures of the local subgraph difference between class $c'$ and class $c$, it really boils down to the local homophily and local heterophily, if the value is pretty small, then it might correspond to a very small $\\beta$, which might correspond to a equal contribution between class c and c' and hence the subgraph is really like a mixture between nodes from these two classes and hence cannot work very well. To sum up this point, my thinking is that it would be better to provide some intuition for some theoretical findings from a topology perspective.\n\n(6) It might be interesting to add some analysis when a distribution shift happens.\n\n(7) For the training loss visualization in Figure 2, it is really hard to see the difference of low/high-degree nodes for RW and ATT.\n\nLIMITATIONS:\nThe paper addressed all limitations with the following exceptions:\n\nThe paper mainly studies degree-related bias for node classification. There are some other tasks such as link prediction and graph classification worth similar analysis as well.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-24"}