Subgroup Generalization and Fairness of Graph Neural Networks

1 Introduction . Graph Neural Networks ( GNNs ) [ 13 , 35 , 20 ] are a family of machine learning models that can be used to model non-Euclidean data as well as inter-related samples in a flexible way . In recent years , there have been enormous successful applications of GNNs in various areas , such as drug discovery [ 18 ] , computer vision [ 29 ] , transportation forecasting [ 49 ] , recommender systems [ 48 ] , etc . Depending on the type of prediction target , the application tasks can be roughly categorized into node-level , edge-level , subgraph-level , and graph-level tasks [ 46 ] . In contrast to the marked empirical success , theoretical understanding of the generalization ability of GNNs has been rather limited . Among the existing literature , some studies [ 9 , 11 , 25 ] focus on the analysis of graph-level tasks where each sample is an entire graph and the samples of graphs are IID . A very limited number of studies [ 36 , 42 ] explore GNN generalization for node-level tasks but they assume the nodes ( and their associated neighborhoods ) are IID samples , which does not align with the commonly seen graph-based semi-supervised learning setups . Baranwal et al . [ 3 ] investigate GNN generalization without IID assumptions but under a specific data generating mechanism . In this work , our first contribution is to provide a novel PAC-Bayesian analysis for the generalization ability of GNNs on node-level tasks with non-IID assumptions . In particular , we assume the node features are fixed and the node labels are independently sampled from distributions conditioned on the node features . We also assume the training set and the test set can be chosen as arbitrary subsets of nodes on the graph . We first prove two general PAC-Bayesian generalization bounds ( Theorem 1 ∗School of Information , University of Michigan , Ann Arbor , Michigan , USA †Equal contribution . ‡Department of EECS , University of Michigan , Ann Arbor , Michigan , USA 4Code available at https : //github.com/TheaperDeng/GNN-Generalization-Fairness . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and Theorem 2 ) under this non-IID setup . Subsequently , we derive a generalization bound for GNN ( Theorem 3 ) in terms of characteristics of the GNN models and the node features . Notably , the generalization bound for GNN is influenced by the distance between the test nodes and the training nodes in terms of their aggregated node features . This suggests that , given a fixed training set , test nodes that are “ far away ” from all the training nodes may suffer from larger generalization errors . Based on this analysis , our second contribution is the discovering of a type of unfairness that arises from theoretically predictable accuracy disparity across some subgroups of test nodes . We further conduct a empirical study that investigates the prediction accuracy of four popular GNN models on different subgroups of test nodes . The results on multiple benchmark datasets indicate that there is indeed a significant disparity in test accuracy among these subgroups . We summarize the contributions of this work as follows : ( 1 ) We establish a novel PAC-Bayesian analysis for graph-based semi-supervised learning with non-IID assumptions . ( 2 ) Under this setup , we derive a generalization bound for GNNs that can be applied to an arbitrary subgroup of test nodes . ( 3 ) As an implication of the generalization bound , we predict that there would be an unfairness of GNN predictions that arises from accuracy disparity across subgroups of test nodes . ( 4 ) We empirically verify the existence of accuracy disparity of popular GNN models on multiple benchmark datasets , as predicted by our theoretical analysis . 2 Related Work . 2.1 Generalization of Graph Neural Networks . The majority of existing literature that aims to develop theoretical understandings of GNNs have focused on the expressive power of GNNs ( see Sato [ 34 ] for a survey along this line ) , while the number of studies trying to understand the generalizability of GNNs is rather limited . Among them , some [ 9 , 11 , 25 ] focus on graph-level tasks , the analyses of which can not be easily applied to node-level tasks . As far as we know , Scarselli et al . [ 36 ] , Verma and Zhang [ 42 ] , and Baranwal et al . [ 3 ] are the only existing studies investigating the generalization of GNNs on node-level tasks , even though node-level tasks are more common in reality . Scarselli et al . [ 36 ] present an upper bound of the VC-dimension of GNNs ; Verma and Zhang [ 42 ] derive a stability-based generalization bound for a single-layer GCN [ 20 ] model . Yet , both Scarselli et al . [ 36 ] and Verma and Zhang [ 42 ] ( implicitly ) assume that the training nodes are IID samples from a certain distribution , which does not align with the common practice of node-level semi-supervised learning . Baranwal et al . [ 3 ] investigate the generalization of graph convolution under a specific data generating mechanism ( i.e. , the contextual stochastic block model [ 8 ] ) . Our work presents the first generalization analysis of GNNs for non-IID node-level tasks without strong assumptions on the data generating mechanism . 2.2 Fairness of Machine Learning on Graphs . The fairness issues of machine learning on graphs start to receive research attention recently . Following conventional machine learning fairness literature , the majority of previous work along this line [ 1 , 5–7 , 22 , 32 , 39 , 50 ] concerns about fairness with respect to a given sensitive attribute , such as gender or race , which defines protected groups . In practice , the fairness issues of learning on graphs are much more complicated due to the asymmetric nature of the graph-structured data . However , only a few studies [ 19 ] investigate the unfairness caused by the graph structure without knowing a sensitive feature . Moreover , in a node-level semi-supervised learning task , the non-IID sampling of training nodes brings additional uncertainty to the fairness of the learned models . This work is the first to present a learning theoretic analysis under this setup , which in turn suggests how the graph structure and the selection of training nodes may influence the fairness of machine learning on graphs . 2.3 PAC-Bayesian Analysis . PAC-Bayesian analysis [ 27 ] has become one of the most powerful theoretical framework to analyze the generalization ability of machine learning models . We will briefly introduce the background in Section 3.2 , and refer the readers to a recent tutorial [ 14 ] for a systematic overview of PAC-Bayesian analysis . We note that Liao et al . [ 25 ] recently present a PAC-Bayesian generalization bound for GNNs on IID graph-level tasks . Both Liao et al . [ 25 ] and this work utilize results from Neyshabur et al . [ 30 ] , a PAC-Bayesian analysis for ReLU-activated neural networks , in part of our proofs . Compared to Neyshabur et al . [ 30 ] , the key contribution of Liao et al . [ 25 ] is the derivation of perturbation bounds of two types of GNN architectures ; while the key contribution of this work is the novel analysis under the setup of non-IID node-level tasks . There is also an existing work of PAC-Bayesian analysis for transductive semi-supervised learning [ 4 ] . But it is different from our problem setup and , in particular , it can not be used to analyze the generalization on subgroups . 3 Preliminaries . In this section , we first formulate the problem of node-level semi-supervised learning . We also provide a brief introduction of the PAC-Bayesian framework . 3.1 The Problem Formulation and Notations . Semi-supervised node classification . Let G = ( V , E ) ∈ GN be an undirected graph , with V = { 1 , 2 , . . . , N } being the set of N nodes and E ⊆ V × V being the set of edges . And GN is the space of all undirected graphs with N nodes . The nodes are associated with node features X ∈ RN×D and node labels y ∈ { 1 , 2 , . . . , K } N . In this work , we focus on the transductive node classification setting [ 47 ] , where the node features X and the graph G are observed prior to learning , and every quantity of interest in the analysis will be conditioned on X and G. Without loss of generality , we treat X and G as fixed throughout our analysis , and the randomness comes from the labels y . In particular , we assume that for each node i ∈ V , its label yi is generated from an unknown conditional distribution Pr ( yi | Zi ) , where Z = g ( X , G ) and g : RN×D × GN → RN×D ′ is an aggregation function that aggregates the features over ( multi-hop ) local neighborhoods5 . We also assume that the node labels are generated independently conditional on their respective aggregated features Zi ’ s . Given a small set of the labeled nodes , V0 ⊆ V , the task of node-level semi-supervised learning is to learn a classifier h : RN×D × GN → RN×K from a function family H and perform it on the remaining unlabeled nodes . Given a classifier h , the classification for a node i is obtained by ŷi = argmax k∈ { 1 , ... , K } hi ( X , G ) [ k ] , where hi ( X , G ) is the i-th row of h ( X , G ) and hi ( X , G ) [ k ] refers to the k-th element of hi ( X , G ) . Subgroups . In Section 4 , we will present an analysis of the GNN generalization performance on any subgroup of the set of unlabeled nodes , V \ V0 . Note that the analysis on any subgroup is a stronger result than that on the entire unlabeled set , as any set is a subset of itself . Later we will show that the analysis on subgroups ( rather than on the entire set ) further allows us to investigate the accuracy disparity across subgroups . We denote a collection of subgroups of interest as V1 , V2 , . . . , VM ⊆ V \ V0 . In practice , a subgroup can be defined based on an attribute of the nodes ( e.g. , a gender group ) , certain graph-based properties , or an arbitrary partition of the nodes . We also define the size of each subgroup as Nm : = |Vm| , m = 0 , . . . , M . Margin loss on each subgroup . Now we can define the empirical and expected margin loss of any classifier h ∈ H on each subgroup Vm , m = 0 , 1 , . . . , M . Given a sample of observed node labels yi ’ s , the empirical margin loss of h on Vm for a margin γ ≥ 0 is defined as L̂γm ( h ) : = 1 Nm ∑ i∈Vm 1 [ hi ( X , G ) [ yi ] ≤ γ + max k 6=yi hi ( X , G ) [ k ] ] , ( 1 ) where 1 [ · ] is the indicator function . The expected margin loss is the expectation of Eq . ( 1 ) , i.e. , Lγm ( h ) : = Eyi∼Pr ( y|Zi ) , i∈VmL̂ γ m ( h ) . ( 2 ) 5A simple example is gi ( X , G ) = 1|N ( i ) |+1 ( Xi + ∑ j∈N ( i ) Xj ) , where gi ( X , G ) is the i-th row of g ( X , G ) and N ( i ) : = { j | ( i , j ) ∈ E } is the set of 1-hop neighbors of node i . To simplify the notation , we define ym : = { yi } i∈Vm , ∀m = 0 , . . . , M , so that Eq . ( 2 ) can be written as Lγm ( h ) = EymL̂γm ( h ) . We note that the classification risk and empirical risk of h on Vm are respectively equal to L0m ( h ) and L̂0m ( h ) .

Summary: This paper studies the non-IID sample generalization in Graph Neural Networks in a subgroup PAC-Bayesian analysis framework. Non-IID training samples are widely existing in real-world applications. It first derives several interesting theories based on several reasonable assumption on smoothness and node features. Several key observations are found between different subgroups in aggregated-feature and graph distance space support the theoretical discoveries.