Subgroup Generalization and Fairness of Graph Neural Networks

1 Introduction . Graph Neural Networks ( GNNs ) [ 13 , 35 , 20 ] are a family of machine learning models that can be used to model non-Euclidean data as well as inter-related samples in a flexible way . In recent years , there have been enormous successful applications of GNNs in various areas , such as drug discovery [ 18 ] , computer vision [ 29 ] , transportation forecasting [ 49 ] , recommender systems [ 48 ] , etc . Depending on the type of prediction target , the application tasks can be roughly categorized into node-level , edge-level , subgraph-level , and graph-level tasks [ 46 ] . In contrast to the marked empirical success , theoretical understanding of the generalization ability of GNNs has been rather limited . Among the existing literature , some studies [ 9 , 11 , 25 ] focus on the analysis of graph-level tasks where each sample is an entire graph and the samples of graphs are IID . A very limited number of studies [ 36 , 42 ] explore GNN generalization for node-level tasks but they assume the nodes ( and their associated neighborhoods ) are IID samples , which does not align with the commonly seen graph-based semi-supervised learning setups . Baranwal et al . [ 3 ] investigate GNN generalization without IID assumptions but under a specific data generating mechanism . In this work , our first contribution is to provide a novel PAC-Bayesian analysis for the generalization ability of GNNs on node-level tasks with non-IID assumptions . In particular , we assume the node features are fixed and the node labels are independently sampled from distributions conditioned on the node features . We also assume the training set and the test set can be chosen as arbitrary subsets of nodes on the graph . We first prove two general PAC-Bayesian generalization bounds ( Theorem 1 ∗School of Information , University of Michigan , Ann Arbor , Michigan , USA †Equal contribution . ‡Department of EECS , University of Michigan , Ann Arbor , Michigan , USA 4Code available at https : //github.com/TheaperDeng/GNN-Generalization-Fairness . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and Theorem 2 ) under this non-IID setup . Subsequently , we derive a generalization bound for GNN ( Theorem 3 ) in terms of characteristics of the GNN models and the node features . Notably , the generalization bound for GNN is influenced by the distance between the test nodes and the training nodes in terms of their aggregated node features . This suggests that , given a fixed training set , test nodes that are “ far away ” from all the training nodes may suffer from larger generalization errors . Based on this analysis , our second contribution is the discovering of a type of unfairness that arises from theoretically predictable accuracy disparity across some subgroups of test nodes . We further conduct a empirical study that investigates the prediction accuracy of four popular GNN models on different subgroups of test nodes . The results on multiple benchmark datasets indicate that there is indeed a significant disparity in test accuracy among these subgroups . We summarize the contributions of this work as follows : ( 1 ) We establish a novel PAC-Bayesian analysis for graph-based semi-supervised learning with non-IID assumptions . ( 2 ) Under this setup , we derive a generalization bound for GNNs that can be applied to an arbitrary subgroup of test nodes . ( 3 ) As an implication of the generalization bound , we predict that there would be an unfairness of GNN predictions that arises from accuracy disparity across subgroups of test nodes . ( 4 ) We empirically verify the existence of accuracy disparity of popular GNN models on multiple benchmark datasets , as predicted by our theoretical analysis . 2 Related Work . 2.1 Generalization of Graph Neural Networks . The majority of existing literature that aims to develop theoretical understandings of GNNs have focused on the expressive power of GNNs ( see Sato [ 34 ] for a survey along this line ) , while the number of studies trying to understand the generalizability of GNNs is rather limited . Among them , some [ 9 , 11 , 25 ] focus on graph-level tasks , the analyses of which can not be easily applied to node-level tasks . As far as we know , Scarselli et al . [ 36 ] , Verma and Zhang [ 42 ] , and Baranwal et al . [ 3 ] are the only existing studies investigating the generalization of GNNs on node-level tasks , even though node-level tasks are more common in reality . Scarselli et al . [ 36 ] present an upper bound of the VC-dimension of GNNs ; Verma and Zhang [ 42 ] derive a stability-based generalization bound for a single-layer GCN [ 20 ] model . Yet , both Scarselli et al . [ 36 ] and Verma and Zhang [ 42 ] ( implicitly ) assume that the training nodes are IID samples from a certain distribution , which does not align with the common practice of node-level semi-supervised learning . Baranwal et al . [ 3 ] investigate the generalization of graph convolution under a specific data generating mechanism ( i.e. , the contextual stochastic block model [ 8 ] ) . Our work presents the first generalization analysis of GNNs for non-IID node-level tasks without strong assumptions on the data generating mechanism . 2.2 Fairness of Machine Learning on Graphs . The fairness issues of machine learning on graphs start to receive research attention recently . Following conventional machine learning fairness literature , the majority of previous work along this line [ 1 , 5–7 , 22 , 32 , 39 , 50 ] concerns about fairness with respect to a given sensitive attribute , such as gender or race , which defines protected groups . In practice , the fairness issues of learning on graphs are much more complicated due to the asymmetric nature of the graph-structured data . However , only a few studies [ 19 ] investigate the unfairness caused by the graph structure without knowing a sensitive feature . Moreover , in a node-level semi-supervised learning task , the non-IID sampling of training nodes brings additional uncertainty to the fairness of the learned models . This work is the first to present a learning theoretic analysis under this setup , which in turn suggests how the graph structure and the selection of training nodes may influence the fairness of machine learning on graphs . 2.3 PAC-Bayesian Analysis . PAC-Bayesian analysis [ 27 ] has become one of the most powerful theoretical framework to analyze the generalization ability of machine learning models . We will briefly introduce the background in Section 3.2 , and refer the readers to a recent tutorial [ 14 ] for a systematic overview of PAC-Bayesian analysis . We note that Liao et al . [ 25 ] recently present a PAC-Bayesian generalization bound for GNNs on IID graph-level tasks . Both Liao et al . [ 25 ] and this work utilize results from Neyshabur et al . [ 30 ] , a PAC-Bayesian analysis for ReLU-activated neural networks , in part of our proofs . Compared to Neyshabur et al . [ 30 ] , the key contribution of Liao et al . [ 25 ] is the derivation of perturbation bounds of two types of GNN architectures ; while the key contribution of this work is the novel analysis under the setup of non-IID node-level tasks . There is also an existing work of PAC-Bayesian analysis for transductive semi-supervised learning [ 4 ] . But it is different from our problem setup and , in particular , it can not be used to analyze the generalization on subgroups . 3 Preliminaries . In this section , we first formulate the problem of node-level semi-supervised learning . We also provide a brief introduction of the PAC-Bayesian framework . 3.1 The Problem Formulation and Notations . Semi-supervised node classification . Let G = ( V , E ) ∈ GN be an undirected graph , with V = { 1 , 2 , . . . , N } being the set of N nodes and E ⊆ V × V being the set of edges . And GN is the space of all undirected graphs with N nodes . The nodes are associated with node features X ∈ RN×D and node labels y ∈ { 1 , 2 , . . . , K } N . In this work , we focus on the transductive node classification setting [ 47 ] , where the node features X and the graph G are observed prior to learning , and every quantity of interest in the analysis will be conditioned on X and G. Without loss of generality , we treat X and G as fixed throughout our analysis , and the randomness comes from the labels y . In particular , we assume that for each node i ∈ V , its label yi is generated from an unknown conditional distribution Pr ( yi | Zi ) , where Z = g ( X , G ) and g : RN×D × GN → RN×D ′ is an aggregation function that aggregates the features over ( multi-hop ) local neighborhoods5 . We also assume that the node labels are generated independently conditional on their respective aggregated features Zi ’ s . Given a small set of the labeled nodes , V0 ⊆ V , the task of node-level semi-supervised learning is to learn a classifier h : RN×D × GN → RN×K from a function family H and perform it on the remaining unlabeled nodes . Given a classifier h , the classification for a node i is obtained by ŷi = argmax k∈ { 1 , ... , K } hi ( X , G ) [ k ] , where hi ( X , G ) is the i-th row of h ( X , G ) and hi ( X , G ) [ k ] refers to the k-th element of hi ( X , G ) . Subgroups . In Section 4 , we will present an analysis of the GNN generalization performance on any subgroup of the set of unlabeled nodes , V \ V0 . Note that the analysis on any subgroup is a stronger result than that on the entire unlabeled set , as any set is a subset of itself . Later we will show that the analysis on subgroups ( rather than on the entire set ) further allows us to investigate the accuracy disparity across subgroups . We denote a collection of subgroups of interest as V1 , V2 , . . . , VM ⊆ V \ V0 . In practice , a subgroup can be defined based on an attribute of the nodes ( e.g. , a gender group ) , certain graph-based properties , or an arbitrary partition of the nodes . We also define the size of each subgroup as Nm : = |Vm| , m = 0 , . . . , M . Margin loss on each subgroup . Now we can define the empirical and expected margin loss of any classifier h ∈ H on each subgroup Vm , m = 0 , 1 , . . . , M . Given a sample of observed node labels yi ’ s , the empirical margin loss of h on Vm for a margin γ ≥ 0 is defined as L̂γm ( h ) : = 1 Nm ∑ i∈Vm 1 [ hi ( X , G ) [ yi ] ≤ γ + max k 6=yi hi ( X , G ) [ k ] ] , ( 1 ) where 1 [ · ] is the indicator function . The expected margin loss is the expectation of Eq . ( 1 ) , i.e. , Lγm ( h ) : = Eyi∼Pr ( y|Zi ) , i∈VmL̂ γ m ( h ) . ( 2 ) 5A simple example is gi ( X , G ) = 1|N ( i ) |+1 ( Xi + ∑ j∈N ( i ) Xj ) , where gi ( X , G ) is the i-th row of g ( X , G ) and N ( i ) : = { j | ( i , j ) ∈ E } is the set of 1-hop neighbors of node i . To simplify the notation , we define ym : = { yi } i∈Vm , ∀m = 0 , . . . , M , so that Eq . ( 2 ) can be written as Lγm ( h ) = EymL̂γm ( h ) . We note that the classification risk and empirical risk of h on Vm are respectively equal to L0m ( h ) and L̂0m ( h ) .

Summary: This work presents PAC-Bayesian analysis for GNNs under a non-IID semi-supervised learning setup. It first analyzes the generalization performance for general settings, and then applies the results to GNNs. The theoretical results have been backed up with empirical evaluations on several graph benchmarks.