Uncertainty-Aware Prediction for Graph Neural Networks

1 INTRODUCTION . Inherent uncertainties introduced by different root causes have emerged as serious hurdles to find effective solutions for real world problems . Critical safety concerns have been brought due to lack of considering diverse causes of uncertainties , resulting in high risk due to misinterpretation of uncertainties ( e.g. , misdetection or misclassification of an object by an autonomous vehicle ) . Graph neural networks ( GNNs ) ( Kipf & Welling , 2016 ; Veličković et al. , 2018 ) have gained tremendous attention in the data science community . Despite their superior performance in semi-supervised node classification and/or regression , they didn ’ t allow to deal with various types of uncertainties . Predictive uncertainty estimation ( Malinin & Gales , 2018 ) using Bayesian NNs ( BNNs ) has been explored for classification prediction or regression in the computer vision applications , with wellknown uncertainties , aleatoric and epistemic uncertainties . Aleatoric uncertainty only considers data uncertainty derived from statistical randomness ( e.g. , inherent noises in observations ) while epistemic uncertainty indicates model uncertainty due to limited knowledge or ignorance in collected data . On the other hand , in the belief or evidence theory , Subjective Logic ( SL ) ( Josang et al. , 2018 ) considered vacuity ( or lack of evidence ) as uncertainty in an subjective opinion . Recently other uncertainties such as dissonance , consonance , vagueness , and monosonance ( Josang et al. , 2018 ) are also introduced . This work is the first that considers multidimensional uncertainty types in both DL and belief theory domains to predict node classification and out-of-distribution ( OOD ) detection . To this end , we incorporate the multidimensional uncertainty , including vacuity , dissonance , aleatoric uncertainty , and epistemic uncertainty in selecting test nodes for Bayesian DL in GNNs . We perform semi-supervised node classification and OOD detection based on GNNs . By leveraging the modeling and learning capability of GNNs and considering multidimensional uncertainties in SL , we propose a Bayesian DL framework that allows simultaneous estimation of different uncertainty types associated with the predicted class probabilities of the test nodes generated by GNNs . We treat the predictions of a Subjective Bayesian GNN ( S-BGNN ) as nodes ’ subjective opinions in a graph modeled as Dirichlet distributions on the class probabilities , and learn the S-BGNN model by collecting the evidence from the given labels of the training nodes ( see Figure 1 ) . This work has the following key contributions : • A Subjective Bayesian framework to predictive uncertainty estimation for GNNs . Our pro- posed framework directly predicts subjective multinomial opinions of the test nodes in a graph , with the opinions following Dirichlet distributions with each belief probability as a class probability . Our proposed framework is a generative model , so it cal be highly applicable across all GNNs and allows simultaneously estimating different types of associated uncertainties with the class probabilities . • Efficient approximate inference algorithms : We propose a Graph-based Kernel Dirichlet distribution Estimation ( GKDE ) method to reduce error in predicting Dirichlet distribution . We designed an iterative knowledge distillation algorithm that treats a deterministic GNN as a teacher network while considering our proposed Subjective Bayesian GNN model ( a realization of our proposed framework for a specific GNN ) as a distilled network . This allows the expected class probabilities based on the predicted Dirichlet distributions ( i.e. , outputs of our trained Bayesian model ) to match the predicted class probabilities of the deterministic GNN model , along with uncertainty estimated in the predictions . • Comprehensive experiments for the validation of the performance of our proposed framework . Based on six real graph datasets , we compared the performance of our propose framework with that of other competitive DL algorithms . For a fair comparison , we tweaked the DL algorithms to consider various uncertainty types in predicted decisions . 2 RELATED WORK . Epistemic Uncertainty in Bayesian Deep Learning ( BDL ) : Machine/deep learning ( M/DL ) research mainly considered aleatoric uncertainty ( AU ) and epistemic uncertainty ( EU ) using BNNs for computer vision applications . AU consists of homoscedastic uncertainty ( i.e. , constant errors for different inputs ) and heteroscedastic uncertainty ( i.e. , different errors for different inputs ) ( Gal , 2016 ) . A BDL framework was presented to estimate both AU and DU simultaneously in regression settings ( e.g. , depth regression ) and classification settings ( e.g. , semantic segmentation ) ( Kendall & Gal , 2017 ) . Later , a new type of uncertainty , called distributional uncertainty ( DU ) , is defined based on distributional mismatch between the test and training data distributions ( Malinin & Gales , 2018 ) . Dropout variational inference ( Gal & Ghahramani , 2016 ) is used as one of key approximate inference techniques in BNNs . Other methods ( Eswaran et al. , 2017 ; Zhang et al. , 2018 ) measure overall uncertainty in node classification but didn ’ t consider uncertainty decomposition and GNNs . Uncertainty Quantification in Belief/Evidence Theory : In the belief/evidence theory domain , uncertainty reasoning has been substantially explored , such as Fuzzy Logic ( De Silva , 2018 ) , DempsterShafer Theory ( DST ) ( Sentz et al. , 2002 ) , or Subjective Logic ( SL ) ( Jøsang , 2016 ) . Belief theory focuses on reasoning of inherent uncertainty in information resulting from unreliable , incomplete , deceptive , and/or conflicting evidence . SL considered uncertainty in subjective opinions in terms of vacuity ( i.e. , lack of evidence ) and vagueness ( i.e. , failing in discriminating a belief state ) ( Jøsang , 2016 ) . Recently , other uncertainty types have been studied , such as dissonance ( due to conflicting evidence ) and consonance ( due to evidence supporting composite states ) ( Josang et al. , 2018 ) . In deep NNs , SL is considered to train a deterministic NN for supervised classification in computer vision applications ( Sensoy et al. , 2018 ) . However , they didn ’ t consider a generic way of estimating multidimensional uncertainty using Bayesian DL for GNNs used for the applications in graph data . 3 PROPOSED APPROACH . Now we define the problem of uncertainty-aware semi-supervised node classification and then present a Bayesian GNN framework to address the problem . 3.1 PROBLEM DEFINITION . Given an input graph G = ( V , E , r , yL ) , where V = { 1 , · · · , N } is a ground set of nodes , E ⊆ V×V is a ground set of edges , r = [ r1 , · · · , rN ] T ∈ RN×d is a node-level feature matrix , ri ∈ Rd is the feature vector of node i , yL = { yi | i ∈ L } are the labels of the training nodes L ⊂ V , and yi ∈ { 1 , . . . , K } is the class label of node i . We aim to predict : ( 1 ) the class probabilities of the testing nodes : pV\L = { pi ∈ [ 0 , 1 ] K | i ∈ V \ L } ; and ( 2 ) the associated multidimensional uncertainty estimates introduced by different root causes : uV\L = { ui ∈ [ 0 , 1 ] m | i ∈ V \ L } , where pi , k is the probability that the class label yi = k and m is the total number of uncertainty types . 3.2 MULTIDIMENSIONAL UNCERTAINTY QUANTIFICATION . Multiple uncertainty types may be estimated , such as aleatoric uncertainty , epistemic uncertainty , vacuity , dissonance , among others . The estimation of the first two types of uncertainty relies on the design of an appropriate Bayesian DL model with parameters , θ . Following ( Gal , 2016 ) , node i ’ s aleatoric uncertainty is : Aleatoric [ pi ] = EProb ( θ|G ) [ H ( yi|r ; θ ) ] , where H ( · ) is Shannon ’ s entropy of Prob ( pi|r ; θ ) . The epistemic uncertainty of node i is estimated by : Epistemic [ pi ] = H [ EProb ( θ|G ) [ ( yi|r ; θ ) ] ] − EProb ( θ|G ) [ H ( yi|r ; θ ) ] ( 1 ) where the first term indicates entropy ( or total uncertainty ) . Vacuity and dissonance can be estimated based on the subjective opinion for each testing node i ( Josang et al. , 2018 ) . Denote i ’ s subjective opinion as [ bi1 , · · · , biK , vi ] , where bik ( ≥ 0 ) is the belief mass of the k-th category , vi ( ≥ 0 ) is the uncertainty mass ( i.e. , vacuity ) , and K is the total number of categories , where ∑K k=1 bik + vi = 1. i ’ s dissonance is obtained by : ω ( bi ) = K∑ k=1 ( bik∑Kj=1 , j 6=k bijBal ( bij , bik ) ∑K j=1 , j 6=k bij ) , ( 2 ) where the relative mass balance between a pair of belief masses bij and bik is expressed by Bal ( bij , bik ) = 1 − |bij − bik|/ ( bij + bik ) . To develop a Bayesian GNNs framework that predicts multiple types of uncertainty , we estimate vacuity and dissonance using a Bayesian model . In SL , a multinomial opinion follows a Dirichlet distribution , Dir ( pi|αi ) , where αi ∈ [ 1 , ∞ ] K represents the distribution parameters . Given Si = ∑K k=1 αik , belief mass bi and uncertainty mass vi can be obtained by bik = ( αik − 1 ) /Si and vi = K/Si . 3.3 PROPOSED BAYESIAN DEEP LEARNING FRAMEWORK . Let p = [ p1 , . . . , pN ] > ∈ RN×K denote the class probabilities of the node in V , where pi = [ pi1 , . . . , piK ] > refers to the class probabilities of a specific node i . As shown in Figure 1 , our proposed Bayesian GNN framework can be described by the generative process : • Sample θ from a predefined prior distribution , i.e. , N ( 0 , I ) . • For each node i ∈ V : ( 1 ) Sample the class probabilities pi from a Dirichlet distribution : Dir ( pi|αi ) , where αi = fi ( r ; θ ) is parameterized by a GNN network α = f ( r ; θ ) : RN×d → [ 1 , ∞ ] N×K that takes the attribute matrix r as input and directly outputs all the node-level Dirichlet parameters α = [ α1 , · · · , αN ] , and θ refer to the hyper-parameters of the GNN network ; and ( 2 ) Sample yi ∼ Cat ( yi|pi ) , a categorical distribution on pi . In this design , the graph dependencies among the class labels in yL and yV\L are modeled via the GNN network f ( r ; θ ) . Our proposed framework is different from the traditional Bayesian GNN network ( Zhang et al. , 2018 ) in that the output of the former are the parameters of node-level Dirichlet distributions ( α ) , but the output of the latter are directly node-level class probabilities ( p ) . The conditional probability of p , Prob ( p|r ; θ ) , can be obtained by : Prob ( p|r ; θ ) = ∏N i=1 Dir ( pi|αi ) , αi = fi ( r ; θ ) ( 3 ) where the Dirichlet probability function Dir ( pi|αi ) is defined by : Dir ( pi|αi ) = Γ ( Si ) ∏K k=1 Γ ( αik ) ∏K k=1 pαik−1ik , Si = ∑K k=1 αik ( 4 ) Based on the proposed Bayesian GNN framework , the joint probability of y conditioned on the input graph G and the node-level feature matrix r can be estimated by : Prob ( y|r ; G ) = ∫ ∫ Prob ( y|p ) Prob ( p|r ; θ ) Prob ( θ|G ) dpdθ , ( 5 ) where Prob ( θ|G ) is the posterior probability of the parameters θ conditioned on the input graph G , which are estimated in Sections 3.4 and 3.6 . The aleatoric uncertainty and the epistemic uncertainty can be estimated using the equations described in Section 3.2 . The vacuity associated with the class probabilities ( pi ) of node i can be estimated by : Vacuity ( pi ) = EProb ( θ|G ) [ vi ] = EProb ( θ|G ) [ K/ ∑K k=1 αik ] . The dissonance of node i is estimated as : Disso . [ pi ] = EProb ( θ|G ) [ ω ( bi ) ] , where ω ( bi ) is defined in Eq . ( 2 ) .

Summary: The authors proposed a Bayesian graph neural network framework for node classification. The proposed models outperformed the baselines in six node classification tasks. The main contribution is to evaluate various uncertainty measures for the uncertainty analysis of Bayesian graph neural networks. The authors show that vacuity and aleatoric measure are important to detect out-of-distribution and the dissonance uncertainty plays a key role for improving performance.