Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification

1 Introduction . Accurate and rigorous uncertainty estimation is key for reliable machine learning models in safetycritical domains [ 67 ] . It quantifies the confidence of machine learning models , thus allowing them to validate knowledgeable predictions or flag predictions on unknown input domains . Uncertainty is commonly divided in aleatoric and epistemic uncertainty [ 28 ] . The aleatoric uncertainty accounts for irreducible uncertainty ( e.g. , due to inherent sensor noise ) . The epistemic uncertainty accounts for a lack of information for accurate prediction ( e.g. , test data significantly different from training data ) . Traditionally , machine learning models assume i.i.d . inputs , thus performing predictions based on input features only . For uncertainty estimation on i.i.d . inputs , a large class of definitions , models and evaluation methods have been introduced [ 28 , 62 , 3 , 78 , 50 ] . Further , uncertainty estimation has been successfully applied to different tasks e.g . out-of-distribution ( OOD ) or shift detection [ 78 ] , active learning [ 75 , 55 ] , continual learning [ 4 ] or reinforcement learning [ 18 ] . In contrast , uncertainty estimation on interdependent nodes is more complex than on i.i.d . inputs and under-explored [ 3 ] . A node in an attributed graph is characterized by two types of information : its features and its neighborhood . While the feature information indicates the node position in the feature space – similarly to i.i.d . inputs – , the neighborhood information indicates the additional node position in the network space . To leverage the neighborhood information , recent graph neural networks ( GNNs ) successfully proposed to enrich and correct the possibly noisy information of the features of a single node by aggregating them with the features of its neighborhood [ 46 , 92 , 48 ] . It naturally leads to the distinction between predictions without network effects based exclusively on their own node feature representation , and predictions with network effects based on neighborhood ∗equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . aggregation . The aggregation step commonly assumes network homophily which states that nodes with similar properties tend to connect to each other more densely , thus violating the i.i.d . assumption between node features given their neighborhood . The core motivation of our work is to transfer some of the existing uncertainty estimation definitions , models and evaluations from i.i.d . inputs to interdependent node inputs by leveraging both the feature and the neighborhood information . In particular , we aim at an accurate quantification of the aleatoric and epistemic uncertainty without and with network effect under network homophily ( see Fig . 1 ) . Our contribution . In this work , we consider uncertainty estimation on semi-supervised node classification . First , we derive three axioms which materialize reasonable uncertainty for non-independent inputs . These axioms cover the traditional notions of aleatoric and epistemic uncertainty and distinguish between the uncertainty with and without network effects . Second , we propose Graph Posterior Network ( GPN ) 2 for uncertainty estimation for node classification and prove formally that it follows the axiom requirements contrary to popular GNNs . Third , we build an extensive evaluation setup for uncertainty estimation which relies on the assessment of uncertainty estimation quality of OOD detection and robustness against shifts of the attributed graph properties . Both OOD data and attributed graph shifts distinguish between attribute and structure anomalies . The theoretical properties of GPN manifest in these experiments where it outperforms all other baselines on uncertainty evaluation . 2 Related Work . In this section , we cover the related work for predictive uncertainty estimation for i.i.d . inputs and for graphs . To this end , we review the commonly accepted axioms defining the desired uncertainty estimation under different circumstances , the methods capable of consistent uncertainty quantification and the evaluation validating the quality of the uncertainty estimates in practice . Uncertainty for i.i.d . inputs – The related work for uncertainty quantification on i.i.d . inputs is rich as for example shown in a recent survey [ 3 ] . Axioms : Far from ID data , the predicted uncertainty is expected to be high [ 66 , 15 , 51 , 30 ] . Close to ID data , the desired uncertainty is more complicated . Indeed , while some works expected models to be robust to small dataset shifts [ 78 , 89 ] , other works expected to detect near OOD classes based on uncertainty [ 98 , 50 , 13 ] . Methods : Many methods already exist for uncertainty quantification for i.i.d . inputs like images or tabular data . A first family of models quantifies uncertainty by aggregating statistics ( e.g . mean , variance or entropy ) from sub-networks with different weights . Important examples are ensemble [ 52 , 96 , 97 , 38 ] , dropout [ 88 ] or Bayesian Neural Networks ( BNN ) [ 9 , 20 , 59 , 24 , 21 ] . Most of these approaches require multiple forward-passes for uncertainty quantification . Further , dropout and BNN may have other pitfalls regarding their limited applicability to more complex tasks [ 77 , 41 , 34 , 27 ] . A second family quantifies uncertainty by using the logit information . Important examples are temperature scaling which rescale the logits after training [ 35 , 56 ] and energy-based models which interpret the logits as energy scores [ 57 , 33 ] . A third family of model quantifies uncertainty based on deep Gaussian Processes ( GP ) . Important examples use GP at activation-level [ 68 ] or at ( last ) layer-level [ 53 , 51 , 91 , 8 ] . Finally , a last 2Project page including code at https : //www.daml.in.tum.de/graph-postnet family of models quantifies uncertainty by directly parameterizing a conjugate prior distribution over the target variable . Important examples explicitly parameterize prior distributions [ 86 , 63 , 60 , 61 , 6 ] or posterior distributions [ 14 , 15 ] . Methods based on GP and conjugate prior usually have the advantage of deterministic and fast inference . Evaluation : Previous works have already proposed empirical evaluation of uncertainty estimation by looking at accuracy , calibration or OOD detection metrics under dataset shifts or adversarial perturbations for i.i.d . inputs [ 78 , 50 ] . In contrast with all these approaches , this work studies uncertainty quantification for classification of interdependent nodes . Uncertainty for graphs – Notably , the recent survey [ 3 ] points out that there is only a limited number of studies on uncertainty quantification on GNN and semi-supervised learning . Moreover , they recommend proposing new methods . Axioms : To the best of our knowledge , only [ 23 ] proposed explicit axioms for node classification for non-attributed graphs . They expect disconnected nodes to recover prior predictions and nodes with higher beliefs to be more convincing . In this work , we clarify the desired uncertainty estimation for node classification on attributed graphs based on motivated and explicit axioms . Methods : The largest family of models for uncertainty for graphs are dropoutor Bayesian-based methods . Important examples propose to drop or assign probabilities to edges [ 83 , 16 , 37 , 19 , 42 ] . Further works proposed to combine the uncertainty on the graph structure with uncertainty on the transformation weights similarly to BNN [ 22 , 101 , 79 , 80 ] . Importantly , these models do not directly quantify uncertainty on the prediction . Similarly to the i.i.d . case , a second family of models focuses on deterministic uncertainty quantification . Important examples mostly use Graph Gaussian Processes , which do not easily scale to large graphs [ 74 , 103 , 58 , 12 ] . Only [ 102 ] explicitly parameterized a Dirichlet conjugate prior . They combined it with multiple components ( Graph-Based Kernel , dropout , Teacher Network , loss regularizations ) which can not easily distinguish between uncertainty without and with network effects . In contrast , GPN is a simple approach based on conjugate prior parametrization and disentangles uncertainty with and without network effects . Evaluation : The evaluation of most of those methods was not focused on the quality of the uncertainty estimates but on the target task metrics ( e.g . accuracy for classification , distance to ground truth for regression ) . Some methods focus on robustness of the target task metrics against adversarial perturbations [ 36 , 107 , 106 ] . Other methods only relied on uncertainty quantification to build more robust models [ 104 , 25 ] . For node classification , only few works evaluated uncertainty by using Left-Out classes or detection of missclassified samples [ 102 ] , active learning [ 74 ] or visualization [ 12 ] . Note that proposed uncertainty evaluations on molecules at graph level [ 100 , 84 , 5 , 40 , 90 ] is an orthogonal problem . In this work , we propose a sound and extensive evaluation for uncertainty in node classification . It distinguishes between OOD nodes w.r.t . features and structure , and graph dataset shifts w.r.t . the percentage of perturbed node features and the percentage of perturbed edges . 3 Uncertainty Quantification for Node Classification . We consider the task of ( semi-supervised ) node classification on an attributed graph G = ( A , X ) with adjacency matrixA ∈ { 0 , 1 } N×N and node attribute matrixX ∈ RN×D . We aim at inferring the labels y ( v ) ∈ { 1 , ... , C } plus the the aleatoric uncertainty u ( v ) alea and the epistemic uncertainty u ( v ) epist of unlabeled nodes v ∈ T given a set of labelled nodes u ∈ U in the graph where V = T ∪ U denotes the set of vertices . 3.1 Axioms . Uncertainty estimation in the setting of interdependent inputs is not well-studied . It often leaves the expected behavior and interpretations for uncertainty estimation unclear . Thus , we need wellgrounded axioms to derive meaningful models . In this section , we aim at specifying the desired uncertainty predictions under various circumstances in homophilic attributed graphs . To this end , we propose three axioms which are based on the two following distinctions . The first distinction differentiates between aleatoric and epistemic uncertainty which are commonly used concepts under the i.i.d . assumptions [ 28 , 62 ] . The second distinction differentiates between uncertainty without and with network effects which are motivated by the concepts of attribute and structure anomalies used in the attributed graph setting [ 11 ] . These new axioms cover all possible combinations encountered by these distinctions and extend the axioms proposed by [ 23 ] for non-attributed graphs . We designed the axioms to be informal and generic so that they are application independent , model-agnostic and do not require complex mathematical notations similarly to [ 23 , 76 ] . In practice , formal definitions need to instantiate general concepts like aleatoric/epistemic uncertainty and with/without network effects noting that some definitions might be more convenient depending on the task . The first axiom deals with ( epistemic and aleatoric ) uncertainty estimation without network effects ( see Fig . 1a , 1c ) . : Axiom 3.1 . A node ’ s prediction in the absence of network effects should only depend on its own features . A node with features more different from training features should be assigned higher uncertainty . Axiom 3.1 states that if a node v has no neighbors , then the final prediction p ( v ) should only depend on its own node features x ( v ) . Further , for anomalous features the model should fall back to safe prior predictions , indicating high aleatoric and epistemic uncertainty . This aligns with [ 23 ] which expects to recover prior predictions for non-attributed nodes without network effect , and [ 66 , 15 ] which expect to recover prior predictions far from training data for i.i.d . inputs . The second axiom deals with epistemic uncertainty estimation with network effects ( see Fig . 1c , 1d ) : Axiom 3.2 . All else being equal , if a node ’ s prediction in the absence of network effects is more epistemically certain , then its neighbors ’ predictions in the presence of network effects should become more epistemically certain . Axiom 3.2 states that a node v with confident feature predictions x ( v ) is expected to be convincing and make its neighbors u ∈ N ( v ) more confident . Conversely , a node with anomalous features is expected to make its neighborhood less confident . This axiom materializes the network homophily assumption at the epistemic level i.e . connected nodes have similar epistemic uncertainty estimates . For non-attributed graphs , [ 23 ] similarly expects a more confident node to have more influence on a direct neighbor . The third axiom deals with aleatoric uncertainty estimation with network effects ( see Fig . 1a , 1b ) : Axiom 3.3 . All else being equal , a node ’ s prediction in the presence of network effects should have higher aleatoric uncertainty if its neighbors ’ predictions in the absence of network effects have high aleatoric uncertainty . Further , a node prediction in the presence network effects should have higher aleatoric uncertainty if its neighbors ’ predictions in the absence network effects are more conflicting . Axiom 3.3 states that no clear classification decision should be made for a node v if no clear classification decisions can be made for its neighbors . Further , the classification decision becomes less certain if a neighbor has a conflicting classification decision . Note that this axiom is more subtle than the direct application of network homophily at the aleatoric level . Indeed a node can have a high aleatoric uncertainty contrary to its neighbors which predict different classes with low aleatoric uncertainty . This aligns with the intuition that conflicting information from the neighborhood provides an irreducible uncertainty to the considered node .

Summary: This paper introduced a well-grounded framework for uncertainty estimation on interdependent nodes. This paper first proposed explicit and motivated axioms describing desired properties for aleatoric and epistemic uncertainty in the absence or in the presence of network effects. Then it proposed GPN, a GNN for uncertainty estimation which provably follows our axioms. Furthermore, extensive experiments was conducted to evaluate the uncertainty performances of a broad range of baselines for OOD detection and robustness against node features or edge shifts. GPN outperforms all baselines in these experiments.