Debiased Graph Neural Networks with Agnostic Label Selection Bias

1 INTRODUCTION . Graph Neural Networks ( GNNs ) are powerful deep learning algorithms on graphs with various applications ( Scarselli et al. , 2008 ; Kipf & Welling , 2016 ; Veličković et al. , 2017 ; Hamilton et al. , 2017 ) . Existing GNNs mainly learn a node embedding through aggregating the features from its neighbors , and such message-passing framework is supervised by node label in an end-to-end manner . During this training procedure , GNNs will effectively learn the correlation between the structure pattern and node feature with node label , so that GNNs are capable of learning the embeddings of new nodes and inferring their labels . One basic requirement of GNNs making precise prediction on unseen test nodes is that the distribution of labeled training and test nodes is same , i.e. , the structure and feature of labeled training and test nodes follow the similar pattern , so that the learned correlation between the current graph and label can be well generalized to the new nodes . However , in reality , there are two inevitable issues . ( 1 ) Because it is difficult to control the graph collection in an unbiased environment , the relationship between the collected real-world graph and the labeled nodes is inevitably biased . Training on such graph will cause biased correlation with node label . Taking a scientist collaboration network as an example , if most scientists with “ machine learning ” ( ML ) label collaborate with those with “ computer vision ” ( CV ) label , existing GNNs may learn spurious correlation , i.e. , scientists who cooperate with CV scientist are ML scientists . If a new ML scientist only connects with ML scientists or the scientists in other areas , it will be probably misclassified . ( 2 ) The test node in the real scenario is usually not available , implying that the distribution of new nodes is agnostic . Once the distribution is inconsistent with that in the training nodes , the performance of all the current GNNs will be hindered . Even transfer learning is able to solve the distribution shift problem , however , it still needs the prior of test distribution , which actually can not be obtained beforehand . Therefore , the agnostic label selection bias greatly affects the generalization ability of GNNs on unknown test data . In order to observe selection bias in real graph data , we conduct an experimental investigation to validate the effect of selection bias on GNNs ( details can be seen in Section 2.1 ) . We select training nodes with different biased degrees for each dataset , making the distribution of training nodes and test nodes inconsistent . The results clearly show that selection bias drastically hinders the performance of GNNs on unseen test nodes . Moreover , with heavier bias , the performance drops more . Further , we theoretically analyze how the data selection bias results in the estimation bias in GNN parameters ( details can be seen in Section 2.2 ) . Based on the stable learning technique ( Kuang et al. , 2020 ) , we can assume that the learned embeddings consist of two parts : stable variables and unstable variables . The data selection bias will cause the spurious correlation between these two kinds of variables . Thereby we prove that with the inevitable model misspecification , the spurious correlation will further cause the parameter estimation bias . Once the weakness of the current GNNs with selection bias is identified , one natural question is “ how to remove the estimation bias in GNNs ? ” In this paper , we propose a novel Debiased Graph Neural Network ( DGNN ) framework for stable graph learning by jointly optimizing a differentiated decorrelation regularizer and a weighted GNN model . Specifically , the differentiated decorrelation regularizer is able to learn a set of sample weights under differentiated variable weights , so that the spurious correlation between stable and unstable variables would be greatly eliminated . Based on the causal view analysis of decorrelation regularizer , we theoretically prove that the weights of variables can be differentiated by the regression weights . Moreover , to better combine the decorrelation regularizer with GNNs , we prove that adding the regularizer to the embedding learned by the second to last layer could be both theoretically sound and flexible . Then the sample weights learned by decorrelation regularizer are used to reweight GNN loss so that the parameter estimation could be unbiased . In summary , the contributions of this paper are three-fold : i ) We investigate a new problem of learning GNNs with agnostic label selection bias . The problem setting is general and practical for real applications . ii ) We bring the idea of variable decorrelation into GNNs to relieve bias influence on model learning and propose a general framework DGNN which could be adopted to various GNNs . iii ) We conduct the experiments on real-world graph benchmarks with two kinds of agnostic label selection bias , and the experimental results demonstrate the effectiveness and flexibility of our model . 2 EFFECT OF LABEL SELECTION BIAS ON GNNS . In this section , we first formulate our target problem as follows : Problem 1 ( Semi-supervised Learning on Graph with Agnostic Label Selection Bias ) . Given a training graph Gtrain = { Atrain , Xtrain , Ytrain } , where Atrain ∈ RN×N ( N nodes ) represents the adjacency matrix , Xtrain ∈ R N×D ( D features ) refers to the node features and Ytrain ∈ R n×C ( n labeled nodes , C classes ) refers to the available labels for training ( n ≪ N ) , the task is to learn a GNN gθ ( ⋅ ) with parameter θ to precisely predict the label of nodes on test graph Gtest = { Atest , Xtest , Ytest } , where distribution Ψ ( Gtrain ) ≠ Ψ ( Gtest ) . 2.1 EXPERIMENTAL INVESTIGATION . We conduct an experimental investigation to examine whether the state-of-the-art GNNs are sensitive to the selection bias . The main idea is that we will perform two representative GNNs : GCN ( Kipf & Welling , 2016 ) and GAT ( Veličković et al. , 2017 ) on three widely used graph datasets : Cora , Citeseer , Pubmed ( Sen et al. , 2008 ) with different degrees of bias . If the performance drops sharply in comparison with the scenarios without selection bias , this will demonstrate that GNNs can not generalize well in selection bias setting . To simulate the agnostic selection bias scenario , we first follow the inductive setting in Wu et al . ( 2019 ) that masks the validation and test nodes as the training graph Gtrain in the training phase , and then infer the labels of validation and test nodes with whole graph Gtest . In this way , the distribution of test node can be considered agnostic . Following Zadrozny ( 2004 ) , we design a biased label selection method on training graph Gtrain . The selection variable e is introduced to control whether the node will be selected as labeled nodes , where e = 1 means selected and 0 otherwise . For node i , we compute its neighbor distribution ratio : ri = ∣ { j∣j ∈ Ni , yj ≠ yi } ∣/∣Ni∣ , where Ni is neighborhood of node i in Gtrain and yj ≠ yi means the label of central node i is not the label of its neighborhood node j . And ri measures the difference between the label of central node i with the labels of its neighborhood . Then we average all the nodes ’ r to get a threshold t. For each node , the probability to be selected is : P ( ei = 1∣ri ) = { ri ≥ t 1 − ri < t , where ∈ ( 0.5 , 1 ) is used to control the degree of selection bias and the larger means heavier bias . We set as { 0.7 , 0.8 , 0.9 } to get three bias degrees for each dataset , termed as Light , Medium , Heavy , respectively . We select 20 nodes for each class for training and the validation and test nodes are same as Yang et al . ( 2016 ) . Furthermore , we take the unbiased datasets as baselines , where the labeled nodes are selected randomly . Figure 1 is the results of GCN and GAT on biased datasets . The dashed lines mean the performances of GCN/GAT on unbiased datasets and the solid lines refer to the results on biased datasets . We can find that : i ) The dashed lines are all above the corresponding coloured solid lines , indicating that selection bias greatly affects the GNNs ’ performance . ii ) All solid lines decrease monotonically with the increase of bias degree , demonstrating that heavier bias will cause larger performance decrease . 2.2 THEORETICAL ANALYSIS . The above experiment empirically verifies the effect of selection bias on GNNs . Here we theoretically analyze the effect of selection bias on estimating the parameters in GNNs . First , because biased labeled nodes have biased neighborhood structure , GNNs will encode this biased information into the node embeddings . Based on stable learning technique ( Kuang et al. , 2020 ) , we make following assumption : Assumption 1 . All the variables of embeddings learned by GNNs for each node can be decomposed as H = { S , V } , where S represents the stable variables and V represents the unstable variables . Specifically , for both training and test environment , E ( Y∣S = s , V = v ) = E ( Y∣S = s ) . Under Assumption 1 , the distribution shift between training set and test set is mainly induced by the variation in the joint distribution over ( S , V ) , i.e. , P ( Strain , Vtrain ) ≠ P ( Stest , Vtest ) . However , there is an invariant relationship between stable variable S and outcome Y in both training and test environments , which can be expressed as P ( Ytrain∣Strain ) = P ( Ytest∣Stest ) . Assumption 1 can be guaranteed by Y⊥V∣S . Thus , one can solve the stable prediction problem by developing a function f ( ⋅ ) based on S. However , one can hardly identify such variables in GNNs . Without loss of generality , we take Y as continuous variable for analysis and have the following assumption : Assumption 2 . The true generation process of target variable Y contains not only the linear combination of stable variables S , but also the nonlinear transformation of stable variables . Based on the above assumptions , we formalize the label generation process as follows : Y = f ( X , A ) + ε = G ( X , A ; θg ) SβS + G ( X , A ; θg ) V βV + g ( G ( X , A ; θg ) S ) + ε , ( 1 ) where G ( X , A ; θg ) ∈ RN×p denotes an unknown function of X and A that learns node embedding and it can be learned by a GNN , such as GCN and GAT , the output variables of G ( X , A ; θg ) can be decomposed as stable variables G ( X , A ; θg ) S ∈ RN×m and unstable variables G ( X , A ; θg ) V ∈ R N×q ( m + q = p ) , βS ∈ R m×1 and βV ∈ R q×1 are the linear coefficients can be learned by the last layer of GNNs , ε is the independent random noise , and g ( ⋅ ) is the nonlinear transformation function of stable variables . According to Assumption 1 , we know that coefficients of unstable variables G ( X , A ; θg ) V are actually 0 ( i.e. , βV =0 ) . For a classical GNN model with linear regressor , its prediction function can be formulated as : Ŷ = Ĝ ( X , A ; θg ) S β̂S + Ĝ ( X , A ; θg ) V β̂V + ε . ( 2 ) Compared with Eq . ( 1 ) , we can find that the parameters of GNN could be unbiasedly estimated if the nonlinear term g ( G ( X , A ; θg ) S ) = 0 , because the GNN model will have the same label generation mechanism as Eq . ( 1 ) . However , limited by the nonlinear power of GNNs ( Xu et al. , 2019 ) , it is reasonable to assume that there is a nonlinear term g ( G ( X , A ; θg ) S ) ≠ 0 that can not be fitted by the GNNs . Under this assumption , next , we taking a vanilla GCN ( Kipf & Welling , 2016 ) as an example to illustrate how the distribution shift will induce parameter estimation bias . A two-layer GCN can be formulated as Âσ ( ÂXW ( 0 ) ) W ( 1 ) , where Â is the normalized adjacency matrix , W is the transformation matrix at each layer and σ ( ⋅ ) is the Relu activation function . We decompose GCN as two parts : one is embedding learning part Âσ ( ÂXW ( 0 ) ) , which can be decomposed as [ ST , VT ] , corresponding to Ĝ ( X , A ; θg ) S and Ĝ ( X , A ; θg ) V in Eq . ( 2 ) , and the other part is W ( 1 ) , where the learned parameters can be decomposed as [ β̃S , β̃V ] , corresponding to [ β̂S , β̂V ] in Eq . ( 2 ) . We aim at minimizing the square loss : LGCN = ∑ni=1 ( STi β̃S +VTi β̃V −Yi ) 2 . According to the derivation rule of partitioned regression model , we have : β̃V − βV = ( 1 n n ∑ i=1 V T i Vi ) −1 ( 1 n n ∑ i=1 V T i g ( Si ) ) + ( 1 n n ∑ i=1 V T i Vi ) −1 ( 1 n n ∑ i=1 V T i Si ) ( βS − β̃S ) , ( 3 ) β̃S − βS = ( 1 n n ∑ i=1 S T i Si ) −1 ( 1 n n ∑ i=1 S T i g ( Si ) ) + ( 1 n n ∑ i=1 S T i Si ) −1 ( 1 n n ∑ i=1 S T i Vi ) ( βV − β̃V ) , ( 4 ) where n is labeled node size , Si is i-th sample of S , 1 n ∑ni=1 VTi g ( Si ) = E ( VTg ( S ) ) + op ( 1 ) , 1 n ∑ni=1 VTi Si = E ( VTS ) + op ( 1 ) and op ( 1 ) is the error which is negligible . Ideally , β̃V − βV = 0 indicates that there is no bias between the estimated and the real parameter . However , if E ( VTS ) ≠ 0 or E ( VTg ( S ) ) ≠ 0 in Eq . ( 3 ) , β̃V will be biased , leading to the biased estimation on β̃S in Eq . ( 4 ) as well . Since the correlation between V and S ( or g ( S ) ) might shift in test phase , the biased parameters learned in training set is not the optimal parameters for predicting testing nodes . Therefore , to increase the stability of prediction , we need to unbiasedly estimate the parameters of β̃V by removing the correlation between V and S ( or g ( S ) ) on training graph , making E ( VTS ) = 0 or E ( VTg ( S ) ) = 0 . Note that 1 n ∑ni=1 STi g ( Si ) in Eq . ( 4 ) can also cause estimation bias , but the relation between S and g ( S ) is stable across environments , which do not influence the stability to some extent .

Summary: This paper presents a novel method to remove the selection bias of graph data, which is neglected by previous methods. Specifically, the authors suspect that all variables observed by GNNs can be decomposed into two parts, stable variables and unstable variables.  Then, DGNN, a differentiable decorrelation regularization is proposed to reweight each variable pair to eliminate estimation bias. Experiments on three datasets confirm its effectiveness.