Uncertainty-Aware Prediction for Graph Neural Networks

1 INTRODUCTION . Inherent uncertainties introduced by different root causes have emerged as serious hurdles to find effective solutions for real world problems . Critical safety concerns have been brought due to lack of considering diverse causes of uncertainties , resulting in high risk due to misinterpretation of uncertainties ( e.g. , misdetection or misclassification of an object by an autonomous vehicle ) . Graph neural networks ( GNNs ) ( Kipf & Welling , 2016 ; Veličković et al. , 2018 ) have gained tremendous attention in the data science community . Despite their superior performance in semi-supervised node classification and/or regression , they didn ’ t allow to deal with various types of uncertainties . Predictive uncertainty estimation ( Malinin & Gales , 2018 ) using Bayesian NNs ( BNNs ) has been explored for classification prediction or regression in the computer vision applications , with wellknown uncertainties , aleatoric and epistemic uncertainties . Aleatoric uncertainty only considers data uncertainty derived from statistical randomness ( e.g. , inherent noises in observations ) while epistemic uncertainty indicates model uncertainty due to limited knowledge or ignorance in collected data . On the other hand , in the belief or evidence theory , Subjective Logic ( SL ) ( Josang et al. , 2018 ) considered vacuity ( or lack of evidence ) as uncertainty in an subjective opinion . Recently other uncertainties such as dissonance , consonance , vagueness , and monosonance ( Josang et al. , 2018 ) are also introduced . This work is the first that considers multidimensional uncertainty types in both DL and belief theory domains to predict node classification and out-of-distribution ( OOD ) detection . To this end , we incorporate the multidimensional uncertainty , including vacuity , dissonance , aleatoric uncertainty , and epistemic uncertainty in selecting test nodes for Bayesian DL in GNNs . We perform semi-supervised node classification and OOD detection based on GNNs . By leveraging the modeling and learning capability of GNNs and considering multidimensional uncertainties in SL , we propose a Bayesian DL framework that allows simultaneous estimation of different uncertainty types associated with the predicted class probabilities of the test nodes generated by GNNs . We treat the predictions of a Subjective Bayesian GNN ( S-BGNN ) as nodes ’ subjective opinions in a graph modeled as Dirichlet distributions on the class probabilities , and learn the S-BGNN model by collecting the evidence from the given labels of the training nodes ( see Figure 1 ) . This work has the following key contributions : • A Subjective Bayesian framework to predictive uncertainty estimation for GNNs . Our pro- posed framework directly predicts subjective multinomial opinions of the test nodes in a graph , with the opinions following Dirichlet distributions with each belief probability as a class probability . Our proposed framework is a generative model , so it cal be highly applicable across all GNNs and allows simultaneously estimating different types of associated uncertainties with the class probabilities . • Efficient approximate inference algorithms : We propose a Graph-based Kernel Dirichlet distribution Estimation ( GKDE ) method to reduce error in predicting Dirichlet distribution . We designed an iterative knowledge distillation algorithm that treats a deterministic GNN as a teacher network while considering our proposed Subjective Bayesian GNN model ( a realization of our proposed framework for a specific GNN ) as a distilled network . This allows the expected class probabilities based on the predicted Dirichlet distributions ( i.e. , outputs of our trained Bayesian model ) to match the predicted class probabilities of the deterministic GNN model , along with uncertainty estimated in the predictions . • Comprehensive experiments for the validation of the performance of our proposed framework . Based on six real graph datasets , we compared the performance of our propose framework with that of other competitive DL algorithms . For a fair comparison , we tweaked the DL algorithms to consider various uncertainty types in predicted decisions . 2 RELATED WORK . Epistemic Uncertainty in Bayesian Deep Learning ( BDL ) : Machine/deep learning ( M/DL ) research mainly considered aleatoric uncertainty ( AU ) and epistemic uncertainty ( EU ) using BNNs for computer vision applications . AU consists of homoscedastic uncertainty ( i.e. , constant errors for different inputs ) and heteroscedastic uncertainty ( i.e. , different errors for different inputs ) ( Gal , 2016 ) . A BDL framework was presented to estimate both AU and DU simultaneously in regression settings ( e.g. , depth regression ) and classification settings ( e.g. , semantic segmentation ) ( Kendall & Gal , 2017 ) . Later , a new type of uncertainty , called distributional uncertainty ( DU ) , is defined based on distributional mismatch between the test and training data distributions ( Malinin & Gales , 2018 ) . Dropout variational inference ( Gal & Ghahramani , 2016 ) is used as one of key approximate inference techniques in BNNs . Other methods ( Eswaran et al. , 2017 ; Zhang et al. , 2018 ) measure overall uncertainty in node classification but didn ’ t consider uncertainty decomposition and GNNs . Uncertainty Quantification in Belief/Evidence Theory : In the belief/evidence theory domain , uncertainty reasoning has been substantially explored , such as Fuzzy Logic ( De Silva , 2018 ) , DempsterShafer Theory ( DST ) ( Sentz et al. , 2002 ) , or Subjective Logic ( SL ) ( Jøsang , 2016 ) . Belief theory focuses on reasoning of inherent uncertainty in information resulting from unreliable , incomplete , deceptive , and/or conflicting evidence . SL considered uncertainty in subjective opinions in terms of vacuity ( i.e. , lack of evidence ) and vagueness ( i.e. , failing in discriminating a belief state ) ( Jøsang , 2016 ) . Recently , other uncertainty types have been studied , such as dissonance ( due to conflicting evidence ) and consonance ( due to evidence supporting composite states ) ( Josang et al. , 2018 ) . In deep NNs , SL is considered to train a deterministic NN for supervised classification in computer vision applications ( Sensoy et al. , 2018 ) . However , they didn ’ t consider a generic way of estimating multidimensional uncertainty using Bayesian DL for GNNs used for the applications in graph data . 3 PROPOSED APPROACH . Now we define the problem of uncertainty-aware semi-supervised node classification and then present a Bayesian GNN framework to address the problem . 3.1 PROBLEM DEFINITION . Given an input graph G = ( V , E , r , yL ) , where V = { 1 , · · · , N } is a ground set of nodes , E ⊆ V×V is a ground set of edges , r = [ r1 , · · · , rN ] T ∈ RN×d is a node-level feature matrix , ri ∈ Rd is the feature vector of node i , yL = { yi | i ∈ L } are the labels of the training nodes L ⊂ V , and yi ∈ { 1 , . . . , K } is the class label of node i . We aim to predict : ( 1 ) the class probabilities of the testing nodes : pV\L = { pi ∈ [ 0 , 1 ] K | i ∈ V \ L } ; and ( 2 ) the associated multidimensional uncertainty estimates introduced by different root causes : uV\L = { ui ∈ [ 0 , 1 ] m | i ∈ V \ L } , where pi , k is the probability that the class label yi = k and m is the total number of uncertainty types . 3.2 MULTIDIMENSIONAL UNCERTAINTY QUANTIFICATION . Multiple uncertainty types may be estimated , such as aleatoric uncertainty , epistemic uncertainty , vacuity , dissonance , among others . The estimation of the first two types of uncertainty relies on the design of an appropriate Bayesian DL model with parameters , θ . Following ( Gal , 2016 ) , node i ’ s aleatoric uncertainty is : Aleatoric [ pi ] = EProb ( θ|G ) [ H ( yi|r ; θ ) ] , where H ( · ) is Shannon ’ s entropy of Prob ( pi|r ; θ ) . The epistemic uncertainty of node i is estimated by : Epistemic [ pi ] = H [ EProb ( θ|G ) [ ( yi|r ; θ ) ] ] − EProb ( θ|G ) [ H ( yi|r ; θ ) ] ( 1 ) where the first term indicates entropy ( or total uncertainty ) . Vacuity and dissonance can be estimated based on the subjective opinion for each testing node i ( Josang et al. , 2018 ) . Denote i ’ s subjective opinion as [ bi1 , · · · , biK , vi ] , where bik ( ≥ 0 ) is the belief mass of the k-th category , vi ( ≥ 0 ) is the uncertainty mass ( i.e. , vacuity ) , and K is the total number of categories , where ∑K k=1 bik + vi = 1. i ’ s dissonance is obtained by : ω ( bi ) = K∑ k=1 ( bik∑Kj=1 , j 6=k bijBal ( bij , bik ) ∑K j=1 , j 6=k bij ) , ( 2 ) where the relative mass balance between a pair of belief masses bij and bik is expressed by Bal ( bij , bik ) = 1 − |bij − bik|/ ( bij + bik ) . To develop a Bayesian GNNs framework that predicts multiple types of uncertainty , we estimate vacuity and dissonance using a Bayesian model . In SL , a multinomial opinion follows a Dirichlet distribution , Dir ( pi|αi ) , where αi ∈ [ 1 , ∞ ] K represents the distribution parameters . Given Si = ∑K k=1 αik , belief mass bi and uncertainty mass vi can be obtained by bik = ( αik − 1 ) /Si and vi = K/Si . 3.3 PROPOSED BAYESIAN DEEP LEARNING FRAMEWORK . Let p = [ p1 , . . . , pN ] > ∈ RN×K denote the class probabilities of the node in V , where pi = [ pi1 , . . . , piK ] > refers to the class probabilities of a specific node i . As shown in Figure 1 , our proposed Bayesian GNN framework can be described by the generative process : • Sample θ from a predefined prior distribution , i.e. , N ( 0 , I ) . • For each node i ∈ V : ( 1 ) Sample the class probabilities pi from a Dirichlet distribution : Dir ( pi|αi ) , where αi = fi ( r ; θ ) is parameterized by a GNN network α = f ( r ; θ ) : RN×d → [ 1 , ∞ ] N×K that takes the attribute matrix r as input and directly outputs all the node-level Dirichlet parameters α = [ α1 , · · · , αN ] , and θ refer to the hyper-parameters of the GNN network ; and ( 2 ) Sample yi ∼ Cat ( yi|pi ) , a categorical distribution on pi . In this design , the graph dependencies among the class labels in yL and yV\L are modeled via the GNN network f ( r ; θ ) . Our proposed framework is different from the traditional Bayesian GNN network ( Zhang et al. , 2018 ) in that the output of the former are the parameters of node-level Dirichlet distributions ( α ) , but the output of the latter are directly node-level class probabilities ( p ) . The conditional probability of p , Prob ( p|r ; θ ) , can be obtained by : Prob ( p|r ; θ ) = ∏N i=1 Dir ( pi|αi ) , αi = fi ( r ; θ ) ( 3 ) where the Dirichlet probability function Dir ( pi|αi ) is defined by : Dir ( pi|αi ) = Γ ( Si ) ∏K k=1 Γ ( αik ) ∏K k=1 pαik−1ik , Si = ∑K k=1 αik ( 4 ) Based on the proposed Bayesian GNN framework , the joint probability of y conditioned on the input graph G and the node-level feature matrix r can be estimated by : Prob ( y|r ; G ) = ∫ ∫ Prob ( y|p ) Prob ( p|r ; θ ) Prob ( θ|G ) dpdθ , ( 5 ) where Prob ( θ|G ) is the posterior probability of the parameters θ conditioned on the input graph G , which are estimated in Sections 3.4 and 3.6 . The aleatoric uncertainty and the epistemic uncertainty can be estimated using the equations described in Section 3.2 . The vacuity associated with the class probabilities ( pi ) of node i can be estimated by : Vacuity ( pi ) = EProb ( θ|G ) [ vi ] = EProb ( θ|G ) [ K/ ∑K k=1 αik ] . The dissonance of node i is estimated as : Disso . [ pi ] = EProb ( θ|G ) [ ω ( bi ) ] , where ω ( bi ) is defined in Eq . ( 2 ) .

Summary: This paper proposes to model various uncertainty measures in Graph Convolutional Networks (GCN) by Bayesian MC Dropout. Compared to existing Bayesian GCN methods, this work stands out in two aspects: 1) in terms of prediction, it considers multiple uncertainty measures including aleatoric, epistemic, vacuity and dissonance (see paper for definitions); 2) in terms of generative modeling, the GCN first predicts the parameters of a Dirichlet distribution, and then the class probabilities are sampled from the Dirichlet. Training/inference roughly follows MC Dropout, with two additional priors/teachers: 1) the prediction task is guided by a deterministic teacher network (via KL(model || teacher)), and 2) the Dirichlet parameters are guided by a kernel-based prior (via KL(model || prior)). Experiments on six datasets showed superior performance in terms of the end prediction task, as well as better uncertainty modeling in terms of out-of-distribution detection.