Title: Improvements on Uncertainty Quantification for Node Classification via Distance Based Regularization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Uncertainty Quantification

Graph Posterior Network

Bayesian
Abstract: Deep neural networks have achieved significant success in the last decades, but they are not well-calibrated and often produce unreliable predictions. A large number of literature relies on uncertainty quantification to evaluate the reliability of a learning model, which is particularly important for applications of out-of-distribution (OOD) detection and misclassification detection. We are interested in uncertainty quantification for interdependent node-level classification. We start our analysis based on graph posterior networks (GPNs) that optimize the uncertainty cross-entropy (UCE)-based loss function. We describe the theoretical limitations of the widely-used UCE loss. To alleviate the identified drawbacks, we propose a distance-based regularization that encourages clustered OOD nodes to remain clustered in the latent space. We conduct extensive comparison experiments on eight standard datasets and demonstrate that the proposed regularization outperforms the state-of-the-art in both OOD detection and misclassification detection.

Improvements on Uncertainty Quantification for Node Classification via Distance-Based Regularization

Abstract

Deep neural networks have achieved significant success in the last decades, but they are not well-calibrated and often produce unreliable predictions. A large number of literature relies on uncertainty quantification to evaluate the reliability of a learning model, which is particularly important for applications of out-ofdistribution (OOD) detection and misclassification detection. We are interested in uncertainty quantification for interdependent node-level classification. We start our analysis based on graph posterior networks (GPNs) that optimize the uncertainty cross-entropy (UCE)-based loss function. We describe the theoretical limitations of the widely-used UCE loss. To alleviate the identified drawbacks, we propose a distance-based regularization that encourages clustered OOD nodes to remain clustered in the latent space. We conduct extensive comparison experiments on eight standard datasets and demonstrate that the proposed regularization outperforms the state-of-the-art in both OOD detection and misclassification detection.

1 Introduction

In recent years, deep neural networks (DNNs) have been widely used in various fields [10, 28]. However, some neural networks provide under-confident [36] or over-confident [14] predictions, limiting their practical applications in risk-constrained and safety-critical fields, such as drug discovery [38], autonomous driving [30], and medical diagnosis [2]. Take autonomous drug design for an example. Uncertainty estimation on the reliability of model predictions helps to support molecular reasoning and experimental design by saving considerable time and resources [23]. It is important to estimate the predictive uncertainty of a DNN, i.e., indicating when its predictions are likely incorrect. There are two main types of uncertainty: epistemic uncertainty (knowledge uncertainty) and aleatoric uncertainty (data uncertainty) [9]. Epistemic uncertainty is due to the lack of knowledge about unseen data. Aleatoric uncertainty is caused by the inherent complexity of the data, which cannot be reduced by increasing the training data, including sources of noise such as homoscedastic or heteroscedastic noise [17]. These two uncertainty types are typically used for out-of-distribution (OOD) and misclassification detection, respectively.

Most models have been introduced for uncertainty estimation on i.i.d. inputs, such as image and tabular data. However, the uncertainty estimation for classifying interdependent nodes in attributed graph data, such as social networks and citation networks, is under-explored. This work focuses on the node classification tasks with great potential to generalize to others with interdependent inputs.

Linlin Yu

∗

Among various graph neural networks (GNNs) for processing graph data structures [18, 35, 13, 8], graph posterior network (GPN) has been developed for semi-supervised node classification tasks [32] that achieves state-of-the-art results in uncertainty estimation.

The major contributions of this paper are three-fold: (1) We theoretically analyze the limitations of GPN at OOD detection when minimizing uncertainty cross-entropy (UCE), a widely used loss function for uncertainty estimation. (2) Motivated by the aforementioned limitations, we propose a distance-based regularization that considers the prior knowledge that OOD-specific features are useful for learning representational space mappings. (3) We conduct extensive experiments comparing our proposed model with five state-of-the-art baselines on eight graph datasets for two uncertainty quantification tasks: OOD detection and misclassification detection tasks. The results demonstrate that our proposed regularization can improve the quality of uncertainty quantification.

2 Related Work

This section reviews existing uncertainty estimation methods for i.i.d data and graph data.

Uncertainty Quantification for i.i.d inputs -There is plentiful research on uncertainty quantification on i.i.d. inputs as discussed in a recent survey [1]. The first family quantifies the predictive uncertainty of a DNN via multiple forward passes, such as deep ensembles and dropout-based Bayesian neural networks (BNNs). Deep ensembles [19] intuitively sample multiple predictions by training an ensemble of deep neural networks and aggregate the results. Dropout-based methods [11] utilize multiple stochastic forward passes implemented with different dropout initializations to approximate the posterior distribution of network weights. However, the substantial memory and computational demands required for training and testing make it impractical for real-time applications. The second family quantifies uncertainty using deterministic single forward-pass neural networks, including density-based methods and distribution-based methods. The density-based approaches typically fit a distribution (e.g., class-wise Gaussian distribution [3, 20, 34]) in the representation space of a pre-trained or fine-tuned DNN, followed by the associated PDF function to quantify different uncertainty types. The distribution-based methods train a deterministic neural network that directly predicts the conjugate prior distribution of the class probabilities of the input feature vector, called Dirichlet distribution, for uncertainty quantification. The predicted Dirichlet distribution can be interpreted as an approximation of the posterior distribution of class probabilities conditioned on the input feature vector. Popular distribution-based models are prior networks [22], evidential networks [29], and posterior networks (PN) [7], all using UCE as the loss function with different regularizations to improve the quality of uncertainty quantification.

Uncertainty Quantification for Graphs -As pointed out in the survey [1], uncertainty quantification on GNNs and semi-supervised learning is under-explored. Most existing models for uncertainty quantification on graphs are either dropout-based or BNN-based methods that typically drop or assign probabilities to edges. There are two approaches using deterministic single-pass GNNs to quantify uncertainty. One is called graph-based kernel Dirichlet distribution estimation (GKDE) [39], which consists of evidential GCN, graph-based kernel, teacher network, dropout, and loss regularization. Another method is the GPN model that combines PN and personalized page rank (PPR) message passing to disentangle uncertainty with and without network effects. In addition, a recent method [37] used standard classification loss for OOD detection on graphs together with an energy function that is directly extracted from GNN, however, it is limited to OOD detection, not generally on the topic of uncertainty quantification.

3 Preliminary

We discuss the problem setting of uncertainty quantification on the task of semi-supervised node classification in Section 3.1. In particular, we use a deep neural network to predict the multinomial uncertainty for each node and evaluate the aleatoric uncertainty and epistemic uncertainty by the prediction result. In Section 3.2, we give a brief review of the GPN model [32], which serves as a fundamental framework for our analysis and motivation for the proposed approach.

3.1 Problem Setting

We define a graph with attributed node-level features G = ( V , E , X, Y L ) , where V is a set of nodes on the graph with cardinality N and E ⊂ V × V denotes a set of graph edges that can be represented by an adjacency matrix W . A feature matrix is denoted by X = [ x 1 , . . . , x N ] T ∈ R N × d , in which each row x i ∈ R d is a feature vector of node i with dimension d . Under the semi-supervised learning setting, a set of labels is available, denoted by Y L = { y i | i ∈ L } , where L ⊂ V and y i ∈ { 1 , . . . , K } for K classes.

Our goal is to design and learn a deterministic GNN based on G that takes the feature matrix X and the adjacency matrix W as INPUT and predicts the parameters of a Dirichlet distribution for each node i ∈ V as OUTPUT, denoted as α i , which is often referred to as the concentration parameters. Therefore, the network function F Θ can be expressed as: A = F Θ ( X,W ) , where A := [ α i ] i ∈V is a matrix and Θ refers to network parameters. The statistical relations between the class label y i , the vector of class probabilities p i , and the Dirichlet parameters α i can be represented as:

Based on the predictions of A , the expected vector of class probabilities ¯ p i := E [ p i | α i ] = [ α i 1 /α i 0 , · · · , α iK /α i 0 ] T , where α i 0 = ∑ K k =1 α ik is called the Dirichlet strength. The aleatoric and epistemic uncertainties about the classification of each node i can be calculated as:

respectively. The aleatoric uncertainty is measured by the negative of the largest class probability in ¯ p i . This uncertainty is higher when the largest class probability in ¯ p i is lower, which implies that the model is less confident and the probabilities are more evenly spread across classes. On the other hand, the epistemic uncertainty is measured by the negative of the Dirichlet strength α i 0 , whose value is higher when the Dirichlet strength is lower, meaning that the model is unfamiliar with the feature vector of node i and the predicted Dirichlet distribution is less concentrated around a specific point or set of points on the probability simplex [22]. We note that a high aleatoric uncertainty may not indicate a high epistemic uncertainty and vice versa. For example, two evidence parameters α i = [1 , · · · , 1] and α j = [1000 , · · · , 1000] have the same aleatoric uncertainty: -1 /K , since they have the same projected class probabilities; but their epistemic uncertainties differ drastically: K versus 1000 K . Please refer to [32, 33] for rationales of aleatoric and epistemic uncertainties in (2).

3.2 Graph Posterior Network

Our framework is based on graph posterior network (GPN) [32], which extends posterior network (PN) [7] to semi-supervised node classification. GPN consists of three main steps. First, a feature encoder maps the original features onto a low-dimensional latent space with a simple two-layer multi-layer perception (MLP) encoder. Second, a radial normalizing flow [27] estimates the density of the latent space per class. Lastly, a personalized page rank message passing scheme [13] diffuses the pseudo counts (density multiplied by the number of training nodes) by taking the graph structure into account. We summarize the three steps with notations as follows,

Multi-layer perceptron for representation learning: z i = f ( x i ; θ ) or f θ in short.

Normalizing flow for density estimation: g ϕ for short, and more specifically

where N k is the number of training nodes belonging to the class k , z i is the embedding vector of node i obtained via the first step, P ( z i | k ; ϕ ) is the conditional density per class k estimated by a normalizing flow module, and ϕ denotes the parameters of this module. GPN also includes the evidence computed prior to the graph aggregation, defined by

Personalized page rank (PPR) for evidence diffusion: β aggr i,k = ∑ j ∈V Π ppr i,j g ϕ ( z j ) k , where Π ppr i,j refer to the dense PPR scores implicitly reflecting the importance of node j from the perspective of node i . Then we can get the predicted concentrate parameters α with a uniform prior 1 for a non-degenerated Dirichlet distribution, i.e.,

As opposed to GPN, PN is designed for uncertainty estimation for i.i.d. inputs, which only considers the first two steps to predict the Dirichlet distribution Dir ( α feat i ) .

Given the labels of training nodes: Y L = { y i | i ∈ L } , GPN is trained by minimizing the following Bayesian loss:

The first term in (6), called uncertainty cross entropy (UCE) [5], is defined by

where Y = [ y i ] i ∈ [ N ] , y i ∈ { 0 , 1 } K is the one-hot encoded ground-truth class of the node i , and Ψ is the digamma function, in terms of the Gamma function by: Ψ( x ) = Γ ′ ( x ) Γ( x ) . Minimizing UCE is known to increase confidence in classifying observed data (training nodes in this context). The second term in (6) is based on the entropy of each node-level Dirichlet distribution Dir ( α i ) that favors smooth distributions. For more details on GPN, please refer to Appendix C.

4 Our Contributions

This section provides a series of theoretical analyses relating to the UCE loss term and the GPN model for detecting the OOD nodes, followed by a partial remedy to derived issues via two distance-based regularizations. Specifically, we prove in Theorem 1 that under certain conditions, UCE can be made arbitrarily small with the limiting case of UCE equal to zero in Corollary 3. Theorem 4 gives a construction to make the UCE to be zero. As UCE does not involve the OOD nodes, Theorem 6 and Corollary 7 elucidate scenarios for possibly detecting the OOD nodes. Lastly, Theorem 8 presents a special situation where GPN fails to detect the OOD nodes.

4.1 Theoretical analysis

The loss function plays a pivotal role in learning effective representation functions and density estimations. In this context, we establish several theorems (Theorems 1 and 4) to describe some demanding assumptions on f θ and g ϕ that achieve the minimum UCE loss. We then describe a limitation of UCE in separating ID and OOD nodes in Theorem 6 and Corollary 7 for PN, which means that we only consider the first two steps in GPN. The main conclusion of our analysis is that the UCE loss function alone is insufficient to learn a representation space that separates OOD from ID nodes. We take graph connectivity into account in Theorem 8 to study some scenarios where GPN is ineffective for OOD detection. Although our theorems do not completely characterize graph learning, they provide some insights into the behavior of the network parameters in PN/GPN when minimizing the UCE loss.

Theorem 1. If the underlying distribution of feature vectors belonging to class k, denoted by X k , is disjoint to each other and both the MLP module ( f θ ) and the normalizing flow module ( h ϕ ) can be arbitrarily complex, then ∀ ϵ > 0 there exists a configuration of f θ and g ϕ such that UCE ( A , Y ) < ϵ .

For the proofs, please refer to Appendix B. Here, we elaborate on the ideal configuration that satisfies the conclusion of Theorem 1. We assume the MLP function f θ is arbitrarily complex such that it maps x i ∈ X k into a bounded ball in the representational space, i.e.,

where each ball B ( z k , r k ) is centered at a point z k and bounded in size with r k < r for a positive value r . Furthermore, we choose the normalizing flow g ϕ to be

where Vol ( · ) refers to the volume of the ball. The conclusion in Theorem 1 states that for every ϵ, there exists a suitable upper bound r of all the balls such that UCE ( A , Y ) < ϵ .

An implication of Theorem 1 is that UCE is not sufficient to separate OOD from ID nodes. Example 2 illustrates a scenario that OOD nodes can be close to ID nodes in the learned representation space even though they can be separated in the feature space based on OOD-specific features, which are unfortunately discarded. In other words, the learned representation space by GPN based on the UCE loss is not guaranteed to preserve the distance between OOD and ID nodes in its representation learning step.

Example 2 (Lost Features) . Suppose two ID classes in a citation network contain bags of words for SVM and neural networks papers respectively. Additionally, the OOD nodes contain bags of words from reinforcement learning papers. Note that frequencies of keywords are used to discriminate different classes. Then the keywords, 'actor critic' and 'policy network' are able to separate OOD nodes from ID nodes, but are irrelevant features for discriminating between the two ID classes. UCE, as a discriminatory loss, is only applied on ID nodes, and hence it is almost impossible to learn representations that respect the OOD-specific features such as 'actor-critic' and 'policy networks'.

Limitations and discussions -The first assumption in Theorem 1 regarding the distinct class-specific distributions of feature vectors might not be realistic in practice since certain ID classes may not be clearly distinguishable due to noise in features or class labels. Nevertheless, our intuition suggests that if the UCE loss is inadequate for separating OOD from ID nodes in situations where they are separable, it is even more likely to falter in the more complex, non-separable cases. As for the second assumption in Theorem 1, it is true that arbitrary complexity of f θ and g ϕ does not fully respect the inductive bias [24] of the network design, such as the MLP layers with ReLU for the feature encoder f ( x ; θ ) , our analysis remains insightful and informative about the structures these networks are likely to exhibit. For example, we may expect from Theorem 1 that the representation of each class may favor an embedded space that compresses OOD-specific features, while density estimation g ϕ tends to have higher peaks over smaller volumes as the model consolidates the representation space.

We note that in [7, Theorem 1] and [32, Theorem], the authors demonstrated that PN/GPN is able to achieve reasonable uncertainty estimation when the feature encoder is a ReLU network and PPR diffusion is removed to disregard network effects. Unfortunately, the analysis is based on extreme node features, specifically as δ →∞ ,

for any node i with a high probability. Moreover, Theorem 1 in the GPN paper [32] holds even when the supports of disjoint ID and OOD classes in the latent space overlap. In other words, this result does not prevent distant nodes from having similar representations. In summary, the analysis based on extreme node features may provide limited insights about the issues of GPN studied in this section. See Appendix D for detailed discussions.

By taking ϵ → 0 , Theorem 1 reduces to Corollary 3 where UCE is equal to 0.

Corollary 3. In the ideal case, where the representation function f θ maps the support of each class, X k , to a countable set (with measure zero), Z k and there exists a normalizing flow that has infinite density on the point set for every class, one achieves UCE ( A , Y ) = 0 .

Next, we aim for the construction of a specific case to make UCE equal to zero. As it is challenging to analyze the joint minimization on θ and ϕ , we assume that the normalizing flow can be chosen optimally. For this purpose, we consider a simplified problem where the true density function is assumed to be known for a given θ and hence it can be used to replace the learning of the normalizing flow. As a result, the problem reduces to the learning of the representation network θ .

Theorem 4. Let X k be the true distributions for classk in the original feature space. Suppose the normalizing flow module g ϕ obtains the true analytic solution. If the true distributions {X k } are disjoint, then the f ˆ θ that minimizes the UCE loss projects the support of each class in the original space to a disjoint point set Z k , where Z k is defined by the projection of X k to the representation space, i.e., Z k = f θ ( X k ) .

Notice that the true analytical solution ˆ ϕ in Theorem 4 is a function of θ , i.e., ˆ ϕ = ϕ ( θ ) . It is possible to know an analytical form of ϕ ( θ ) . For example, if the data points belonging to each class are sampled from a known Gaussian distribution in the original feature space and the representation network is a linear projection function, then the true density of the projected data points belonging to each class can be derived based on any configuration of the known Gaussian distribution in the original feature space.

δ

δ

f

θ

(

X

k

)

X

k

f

-

1

θ

B

(

z

k

, r

k

)

far-OOD

Near-OOD

δ

Feature

Space

Latent

Space

Before discussing OOD detection, we start with the definition of OOD nodes.

Definition 5. We define out-of-distribution (OOD) nodes to be the nodes that do not belong to any of the K in-distribution (ID) classes. We denote all the ID distribution supports by X [ K ] = ⋃ K k =1 X k .

In order to detect the OOD nodes, we need a good representation, e.g., f θ ( X [ K ] ) ⋂ f θ ( X-X [ K ] ) = ∅ . However, it is possible that ∪ K k =1 f -1 θ ( Z k ) = X , which implies that any OOD node in the feature space is mapped to the same distribution of an ID class in the representation space. To this end, we require the preimage, f -1 θ , to be well-behaved in the sense that f -1 θ ( Z k ) must be contained within a bounded region near the support of the true distribution X k . Explicitly, we require there exists a constant δ > 0 such that d ( x , ˆ x ) < δ, ∀ ˆ x ∈ f -1 θ ( Z k ) and ∀ x ∈ X k for each k . The choice of δ depends on additional information about the dataset. An excessively large δ causes overlap between classes, thus increasing the likelihood of improperly identifying the OOD nodes as ID. On the other hand, a relatively small δ forces the model to overfit, which labels the non-training nodes as OOD.

We characterize in Theorem 6 that under certain conditions, OOD can be detected correctly if they are far away from the ID distribution. One such condition is that the function f θ is well-fit, meaning that it maps the support of X k inside B ( z k , r k ) for every k in [ K ] . We denote a set of all well-fit functions by Γ . Please refer to Figure 1 for a geometric illustration of these relevant quantities.

Theorem 6 (Far OOD) . Under the same assumptions in Theorem 4 and two additional assumptions: (i) X k is bounded and (ii) for any ˆ x ∈ f -1 θ ( z ) with z ∈ Z k , there exists x k ∈ X k such that d ( x k , ˆ x ) < δ , if the true distribution of OOD nodes does not overlap with the region ∪ θ ∈ Γ ∪ k ∪ z k ∈Z k { f -1 θ ( z k ) } , then minimizing UCE can learn a representation projection of f θ that detect all the OOD points farther than δ from X k .

Corollary 7 (Near OOD) . Following Theorem 6, if ˆ x ∈ ∪ θ ∈ Γ ∪ k ∪ z k ∈Z k { f -1 θ ( z k ) } follows the distribution of OOD nodes, then the classification of ˆ x depends on the choice of θ ∈ Γ .

The main purpose of Theorem 6 is to show a desired behavior for OOD detection is induced by point-wise effects. In practice, we suggest the careful construction of f θ 's topology coupled with proper regularization terms to achieve the point-wise effect.

Lastly, we consider the setting of GPN to use a graph layer after the feature-wise evidence predictions.

Theorem 8. Under the following conditions: (1) The features of some class 0 and OOD nodes belong to S ; (2) The OOD nodes are only connected to class 0 and themselves; (3) Other nodes belong to other regions i.e. x ∈ X -S if x / ∈ X 0 ⋃ X OOD; (4) Other nodes' features are non-degenerate in their associated region; and (5) the endowed graph neural network layer must produce evidence for each node between the highest and lowest feature evidence found among its neighbors. A graph with arbitrary homophily can achieve a global minimum on UCE with the associated OOD nodes achieving arbitrarily large evidence, while simultaneously having perfect accuracy.

Limitations and Discussions. In Theorem 8, we provide a special situation where the GPN architecture may fail to detect OOD nodes by predicting large evidence values for belonging to class 0. In addition, we show in Appendix B that if GPN has bad initial feature predictions, even ideal graph construction coupled with an ideal graph neural network (with a homophily degree 1.0) fails to

prevent the OOD nodes from being misclassified as ID nodes. For homophily graphs with degrees less than 1.0, the majority of nodes for each class have similar evidence values after graph diffusion layers, except that the nodes between some pairs of classes may have different evidence values.

4.2 Distance-Based Regularization

As discussed in Section 4.1, the UCE loss function alone is insufficient to learn a representation space that separates OOD from ID nodes using the GPN model. We propose a heuristic remedy that enforces distance minimization on the graph. Ideally, we should design a distance formula that can preserve the distance relationship among all the feature vectors. However, distance preservation likely increases variation in the latent space as we cannot compress the classes' support in the representation space to be arbitrarily small, while simultaneously preserving distances.

Instead, we consider distance minimization, as it helps prevent the model from discarding relevant features while decreasing variation between nodes in the representation space. Here we give two formulations directly. The simpler, theorem-motivated term, is the distance regularization on the latent space,

The regularization encourages nearby points in the graph representation to remain nearby in the latent space. In other words, this regularization (9) discourages the overlap f θ ( X [ K ] ) ⋂ f θ ( X - X [ K ] ) .

We also minimize the "distance" on the produced evidence through a divergence-based regularization,

where divkl refers to the Kullback-Leibler divergence and two symmetric terms are considered. This divergence-based formulation likely decreases the variation in evidence between neighboring nodes, because high variation in the latent space between neighboring nodes need not be mapped to similar evidence.

In summary, we augment the GPN mode with either one of the proposed regularization terms in (9) and (10), thus leading to the objective function as follows,

with two positive parameters λ 1 , λ 2 . The first term is the standard UCE loss function. The second term is regarded by GPN as an entropy regularizer. The last term, R, is chosen to be either R D or R α , a decision implemented through hyperparameter tuning. We have included a theoretical result in Appendix B that provides a rationale for the proposed distance regularizations.

5 Experiments

In this section, we conduct extensive experiments on two tasks of OOD detection and misclassification detection. We compare the proposed framework (11) for uncertainty estimation of semi-supervised node classification using 8 datasets with a comparison to 5 baseline methods. The code is available at https://github.com/neoques/Graph-Posterior-Network.

5.1 Experiment Setup

Datasets We use three citation networks (i.e. CoraML, CiteSeer, Pubmed) [4], two co-purchase datasets [31] (i.e. AmazonComputers, AmazonPhotos), two coauthor datasets [31] (i.e. CoauthorCS and CoauthorPhysics) and a large dataset OGBN Arxiv [16]. A detailed description of these datasets is in Appendix E. We show the result of three citation datasets in the main paper and the remaining results in Appendix F.

Baselines We present the results for uncertainty estimation using five baseline methods. Among these, two evidence collection models, namely graph kernel density estimation (GKDE) [39] and label propagation (LP) [32], assuming that OOD nodes are located far away from the training nodes, while easily misclassified nodes reside near the boundaries between classes. We compare to a modified GCN model, referred to as VGCN-Energy [21], a Bayesian-based model, called GKDE-GCN [39], and GPN [32] as baselines in our evaluation. We also introduce a Graph Neural Network called APPNP [12] as one baseline for the misclassification detection task and report the ROC score. Details of these baselines can be found in Appendix D.

Metrics To assess the classification performance of ID nodes, we rely on the metric ID-ACC , which calculates the fraction of correct predictions among all predictions. As for evaluating uncertainty estimation, we employ the metrics AUC-ROC and AUC-PR as evaluation measures. The rankings are based on the scores of epistemic or aleatoric uncertainty. OOD detection is treated as a binary classification task, where the positive class corresponds to OOD nodes and the negative class pertains to ID nodes. Please refer to (2) for the calculation of aleatoric uncertainty and epistemic uncertainty. For the Dirichlet-based models, the epistemic uncertainty has a similar practical interpretation to vacuity in the belief theory, assessed using AUC scores. On the other hand, in VGCN-Energy, the calculation of epistemic uncertainty is based on the energy value and is represented as u epis i = energy. The misclassification detection task is also a binary classification problem, where the positive cases correspond to wrongly classified nodes and the negative cases represent correctly classified nodes. The calculation of uncertainty for misclassification detection is performed in the same manner as OOD detection except that u epis i = -max k α k i . Prior studies [32, 39] has indicated that aleatoric uncertainty is generally more effective for identifying misclassifications, whereas epistemic uncertainty is more appropriate for detecting out-of-distribution instances.

Model Setup For all the baseline methods, we maintain consistency by employing the same set of model hyperparameters as provided by GPN. Specifically for some model hyperparameters such as latent dimension and weight decay, we adopt the same settings as GPN. Inspired by [26], we explore multiple choices of activation functions in the representation networks. In addition to the default ReLU used in GPN, we experiment with Sigmoid and GELU activation functions. Through empirical evaluation, we discover that the choice of activation function significantly impacts the performance of certain datasets, which is demonstrated in Appendix E.4. Besides, hyperparameters that we tune include entropy regularization weight, distance-based regularization format (whether R D or R α ), and weighting parameters ( λ 1 , λ 2 ), which are optimized based on the validation cross-entropy for each specific dataset. For a comprehensive overview of the hyperparameter configuration and ablation study, please refer to Appendix D.

5.2 Results

OOD Detection OOD detection aims to detect whether an input example is OOD given the predicted uncertainty estimation. For the semi-supervised node classification, we adopt the Left-OutClasses setting where we assume several categories as OOD (details can be found in Appendix D), as considered in [7, 39]. Different from the independent input setting, we retain the OOD nodes in the graph but exclude their labels from the training and validation sets. This implies that the loss function does not involve OOD labels, but the model has encountered the OOD node features during the training phase. Similarly to [32], we also remove the last graph propagation layer for comparison as 'w/o network' where the final result only depends on the node features and no graph structure involved. This configuration, referred to as the 'w/o network' setting, results in a final output that solely relies on the node features, with no involvement of the graph structure.

The results on CoraML, Citeseer, and PubMed are presented in Table 1 and the results on the other 5 datasets are shown in Table 6 in Appendix E. Our model achieves the best ID accuracy for four datasets and demonstrates comparable performance to GPN for the remaining four datasets Furthermore, we observe an improvement in the ROC (Receiver Operating Characteristic) rankings based on predicted epistemic uncertainty with propagation, ranging from +1% to +8% compared to GPN. Consistent with previous studies [32], our results demonstrate that prediction models incorporating evidence propagation consistently outperform those without propagation across all datasets. This observation highlights the significant impact of graph structure on uncertainty estimation. Moreover, when comparing aleatoric uncertainty and epistemic uncertainty as ranking scores, we find that epistemic

uncertainty outperforms aleatoric uncertainty in the OOD detection task. This finding aligns with literature [39, 32] and emphasizes the superiority of epistemic uncertainty for OOD detection.

Alea: Aleatoric, Epi.: Epistemic, w/: with propagation, w/o: without propagation

AUROC and AUPR for the OOD Detection: ID-ACC means the accuracy on ID nodes. AUROC and AUPR scores are given for OOD detection based on different uncertainty metrics, where [Alea w/] is the aleatoric score with propagation layer, [Epi w/] is the epistemic score with propagation layer and [Epi w/o] is the epistemic score without propagation. n.a. means either model or metric not applicable.

Misclassification Detection In addition to OOD detection, we conduct misclassification detection on the clean graph for evaluating the predictive uncertainty estimation. Table 2 presents the results for three datasets, while Table 7 in Appendix D is for the other 5 datasets. We observe a significant improvement ranging from +12% to +50% in our model's AUC-PR scores. While it is true that our method performs worse than the best of the baselines in terms of AUROC, the differences are within approximately 3% for six of the eight datasets.: Amazon Computers, Amazon Photos, Coauthor CS, Coauthor Physics, and ODBG Arxiv, and PubMed. Despite having a lower AUROC compared to the best of the baselines, our method exhibits a higher AUPR. This suggests that our method may excel at identifying true positives among the top-ranked nodes when compared to GPN, while the best baselines may be more effective at distinguishing between true positives and negatives among the lower-ranked nodes.

Alea: Aleatoric, Epi.: Epistemic, w/: with propagation, w/o: without propagation

AUROC and AUPR for misclassification detection: AUROC and AUPR scores are given for misclassification detection based on different uncertainty metrics, where [Alea w/] is the aleatoric score with propagation layer, [Epi w/] is the epistemic score with propagation layer and [Epi w/o] is the epistemic score without propagation. n.a. means either model or metric not applicable

5.3 Ablation Study

Our proposed model differs from the GPN model in three main aspects. First, we use validation cross entropy (CE) instead of hold-out datasets to select hyperparameters. Second, we consider the

activation function for the MLP layers as one of the hyperparameters for selection. We expect feature value rescaling through non-linear activation of feature values to affect the density predictions. Third, we incorporate one of the proposed distance-based regularization terms to the loss function used in GPN.We conduct an ablation study to demonstrate the contribution of these three components. The results for CiteSeer and PubMed are shown in Table 3 and the remaining datasets are included in Appendix E. Hyperparameter tuning using validation cross-entropy improves GPN's performance, especially in cases where the choice of activation function has a significant impact on specific datasets. Additionally, we consistently observe performance enhancements from the distance-based regularization in both datasets, demonstrating the effectiveness of the proposed distance awareness regularization term.

Alea: Aleatoric, Epi.: Epistemic, w/: with propagation

GPN refers to the original GPN paper with its default hyperparameters and ReLU as the middle activation function, GPN-CE is the original GPN model with re-tuned Dirichlet entropy regularization weight based on validation cross-entropy; GPN-CE-ACT is the original GPN model with re-tuned entropy regularization weight and activation function based on cross-entropy; GPN-CE-GD/(Ours) adds the distance-based regularization term while tuning the two weights and activation function.

6 Limitations

Our theoretical analyses mainly study the limitations of the UCE loss function when separating OOD from ID nodes in the learned representation space. If we include the entropy-based regularization term in Equation (6) with a sufficiently large weight λ , some of our theoretical findings may not remain applicable. However, the entropy-based regularization term is designed to favor smooth Dirichlet distributions but not to preserve the distance between OOD and ID nodes. We conjecture that the resulting loss function is still insufficient to learn a representation space that separates OOD from ID nodes, even though they are separable in the original feature space. In addition, our proposed regularization terms in Section 4.2 are more effective for homophily graphs than heterophily graphs, as neighboring nodes are less likely to belong to the same class in a heterophily graph than those in a homophily graph.

7 Conclusion

This paper contributed to a better understanding of uncertainty quantification for node classification. We investigated the limitations of the widely used UCE loss function. Motivated by the theoretical analysis, we proposed a distance-based regularization that helps learn a representation network that is more effective for the uncertainty quantification task. Experimentally, we demonstrated our approach outperforms the state-of-the-art in two specific applications of uncertainty quantification for node classification: OOD detection and misclassification detection.

8 Acknowledgments

This work is supported by the National Science Foundation (NSF) under Grant No #2220574, #2107449, #1846690, and #1750911. The work of Feng Chen is also supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (DOI/IBC) contract number 140D0423C0026. The US Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA DOI/IBC or the U.S. Government.

References

M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion , 76:243-297, 2021.

E. Begoli, T. Bhattacharya, and D. Kusnezov. The need for uncertainty quantification in machine-assisted medical decision making. Nature Machine Intelligence , 1(1):20-23, 2019.

M. Biloš, B. Charpentier, and S. Günnemann. Uncertainty on asynchronous time event prediction. Advances in Neural Information Processing Systems , 32, 2019.

A. Bojchevski and S. Günnemann. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. arXiv preprint arXiv:1707.03815 , 2017.

B. Charpentier, M. Bilos, and S. Günnemann. Uncertainty on asynchronous time event prediction. Advances in neural information processing systems , 32:12831-12840, 2019.

B. Charpentier, O. Borchert, D. Zügner, S. Geisler, and S. Günnemann. Natural posterior network: Deep bayesian predictive uncertainty for exponential family distributions. In International Conference on Learning Representations , 2022.

B. Charpentier, D. Zügner, and S. Günnemann. Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts. Advances in Neural Information Processing Systems , 33:1356-1367, 2020.

P. de Haan, T. S. Cohen, and M. Welling. Natural graph networks. Advances in neural information processing systems , 33:3636-3646, 2020.

A. Der Kiureghian and O. Ditlevsen. Aleatory or epistemic? does it matter? Structural safety , 31(2):105112, 2009.

A. Filos, P. Tigkas, R. McAllister, N. Rhinehart, S. Levine, and Y. Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In International Conference on Machine Learning , pages 3145-3153. PMLR, 2020.

Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

J. Gasteiger, A. Bojchevski, and S. Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997 , 2018.

J. Gasteiger, A. Bojchevski, and S. Günnemann. Combining neural networks with personalized pagerank for classification on graphs. In International Conference on Learning Representations , 2019.

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

M. Hein, M. Andriushchenko, and J. Bitterwolf. Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 41-50, 2019.

W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:2211822133, 2020.

A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems , 30, 2017.

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

J. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems , 33:7498-7512, 2020.

W. Liu, X. Wang, J. Owens, and Y. Li. Energy-based out-of-distribution detection. Advances in neural information processing systems , 33:21464-21475, 2020.

A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems , 31, 2018.

L. H. Mervin, S. Johansson, E. Semenova, K. A. Giblin, and O. Engkvist. Uncertainty quantification in drug design. Drug discovery today , 26(2):474-489, 2021.

T. M. Mitchell. The need for biases in learning generalizations. 1980.

E. Nalisnick, A. Matsukawa, Y. W. Teh, and B. Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using typicality. arXiv preprint arXiv:1906.02994 , 2019.

P. Ramachandran, B. Zoph, and Q. V. Le. Searching for activation functions. arXiv preprint arXiv:1710.05941 , 2017.

D. Rezende and S. Mohamed. Variational inference with normalizing flows. In International conference on machine learning , pages 1530-1538. PMLR, 2015.

A. G. Roy, J. Ren, S. Azizi, A. Loh, V. Natarajan, B. Mustafa, N. Pawlowski, J. Freyberg, Y. Liu, Z. Beaver, et al. Does your dermatology classifier know what it doesn't know? detecting the long-tail of unseen conditions. Medical Image Analysis , 75:102274, 2022.

M. Sensoy, L. Kaplan, and M. Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems , 31, 2018.

S. Shafaei, S. Kugele, M. H. Osman, and A. Knoll. Uncertainty in machine learning: A safety perspective on autonomous driving. In Computer Safety, Reliability, and Security: SAFECOMP 2018 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Västerås, Sweden, September 18, 2018, Proceedings 37 , pages 458-464. Springer, 2018.

O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 , 2018.

M. Stadler, B. Charpentier, S. Geisler, D. Zügner, and S. Günnemann. Graph posterior network: Bayesian predictive uncertainty for node classification. Advances in Neural Information Processing Systems , 34:18033-18048, 2021.

D. Ulmer, C. Hardmeier, and J. Frellsen. Prior and posterior networks: A survey on evidential deep learning methods for uncertainty estimation. Transactions on Machine Learning Research , 2023.

J. Van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal. Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning , pages 9690-9700. PMLR, 2020.

P. Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

X. Wang, H. Liu, C. Shi, and C. Yang. Be confident! towards trustworthy graph neural networks via confidence calibration. Advances in Neural Information Processing Systems , 34:23768-23779, 2021.

Q. Wu, Y. Chen, C. Yang, and J. Yan. Energy-based out-of-distribution detection for graph neural networks. arXiv preprint arXiv:2302.02914 , 2023.

J. Yu, D. Wang, and M. Zheng. Uncertainty quantification: Can we trust artificial intelligence in drug discovery? Iscience , page 104814, 2022.

X. Zhao, F. Chen, S. Hu, and J.-H. Cho. Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems , 33:12827-12836, 2020.

The supplementary materials are organized as follows. Appendix A provides the background knowledge on the Dirichlet distribution. In Appendix B we review the architecture of the graph posterior network (GPN) [32] together with our discussion on oversight of [32, Theorem 1]. Appendix C details the proofs of all the theorems and corollaries discussed in the main paper. We provide detailed descriptions of baseline models, datasets, and hyperparameter tuning for the experiments in Appendix D . Lastly, Appendix E includes more experimental results that we are unable to fit into the paper.

A Dirichlet Distribution

A non-degenerate Dirichlet distribution, denoted by Dir ( α ) , is parameterized by the concentration parameters α = [ α 1 , · · · , α K ] ⊺ with α k > 1 for k ∈ [ K ] . More specifically, the Dirichlet distribution with parameters α 1 , · · · , α K has a probability density function (pdf) given by

where { p k } K k =1 belongs to the standard probability simplex, thus ∑ K k =1 p k = 1 and p k ∈ [0 , 1] , ∀ k ∈ [ K ] , and the normalizing constant B ( α ) is expressed in terms of the Gamma function Γ( · ) , i.e.,

Under the semi-supervised learning setting, a set of labels is available, denoted by L ⊂ V . For i ∈ L , the class label y i ∈ { 1 , . . . , K } can be converted into a one-hot vector y i with y ik = 1 if the sample belongs to the k -th class and y ij = 0 for j = k. By arranging α i and y i into matrices A := [ α i ] i ∈ L and Y := [ y i ] i ∈ L , the UCE loss function is defined as:

where α i 0 = ∑ K k =1 α ik and Ψ is the digamma function given by

̸

B Detailed Framework of GPN

In this section, we review the architecture of GPN, followed by two examples that reveal a limitation of Theorem 1 in the GPN paper [32].

Multi-layer Perceptron (MLP). Instead of deep convolution layers used in many neural networks designed for image classification task [7], GPN utilizes two simple perceptron layers with ReLU activation function as the encoding network, which maps high dimensional data to a latent space with a much smaller dimension, avoiding the curse of dimensionality for the density estimation on a (mapped) latent representation [25]. As each node is independent of the others in this step, the encoding map only considers the node features without any graph structure involved. Mathematically, the mapping can be expressed by

where θ := { W 1 , W 2 , b 1 , b 2 } denotes a set of learning parameters. For simplicity we use the notation z i = f θ ( x i ) .

Normalizing Flow. Normalizing flow is used to estimate the density P ( z i | k ; ϕ ) for k ∈ [ K ] and learning parameters ϕ as an invertible transformation q ( · ; k ) of a base distribution, e.g. Normal distribution, which denotes the distribution of class k in the latent space. The default flow in GPN is the radial flow [27], given by

where z 0 is a reference point and p z ( · ) ∼ N (0 , 1) . After estimating the density of the node i belonging to a specific class k , the pseudo evidence counts are scaled to the probability, i.e., β k i ∝ P ( z i | k ; ϕ ) , GPN sets

where N k is the number of training nodes that belong to the class k .

Personalized Page Rank. GPN applies a personalized page rank (PPR) module to diffuse the evidence among neighboring nodes. It is motivated by the work of Approximate Personalized Propagation of Neural Predictions (APPNP) [12] that is designed to decouple the prediction (only based on node features) with any encoding network and propagate with a personalized page rank (PPR) module (only based on edge information). In particular, PPR provides a personalized influence score matrix for each node that considers L hop of neighbors without involving any new parameters to learn and L is a hyperparameter:

where γ is a hyper-parameter relating to the teleport probability, ˆ A denotes the symmetrically normalized graph adjacency matrix with added self-loops (i.e., ˆ A := D -1 / 2 AD -1 / 2 with the standard adjacency matrix A ), and l denotes the layer index with β (0) obtained after the normalizing flow. The output of PPR is a set of concentration parameters, denoted by α = h γ ( β (0) ) .

Collectively for MLP, normalizing flow, and PPR, the network in GPN can be expressed by

for each node i , where the addition of 1 guarantees that the concentration parameter is strictly positive. In addition, an entropy regularization was considered by GPN defined by,

where α i 0 = ∑ K k =1 α ik .

Next, we provide two examples to describe oversight of [6, Theorem 1] and [32, Theorem 1] in the sense that both theorems assume an impossibility. Particularly the assumption is that a two-layer ReLU network can be represented by a set of affine mappings, each being full rank, from a finite set of regions to the latent space. However, we construct Example 9 and Example 10 to show this assumption is impossible.

Example 9. We start with a simple case where a two-layer ReLU network with input, hidden layer, and output of a scalar (1-dimensional) is considered for an easier interpretation of the results. One simple example of a two-layer ReLU network is expressed by

Following [15], we split the latent space into two affine regions, i.e.,

labeled by Q (0) = [0 , ∞ ) and Q (1) = ( -∞ , 0] . We see the associated V (0) = 1 and V (1) = 0 in the affine representation (17) that certainly do not have independent rows, as required by [6, Theorem 1].

Example 10 extends the 1D case in Example 9 into a higher d -dimension, showing that there is always at least one affine region that produces a single value, i.e. f θ ( Q ( l ) ∗ ) = { v } when mapped into a ReLU network f θ .

Example 10. We consider the ReLU network,

where B,C ∈ R d × d are matrices of full rank. Denote x j to be the solution to the equation,

where e j is the j th Euclidean standard basis. As B is assumed to be full rank, there is the unique solution of the corresponding x j .

Notice that the polytope,

has non-zero measure in R d . Note that the ReLU network is constant by construction, as σ ReLU ( -e j ) = 0 . In other words, we have for x ∈ S that

As in the previous example, the existence of S means that there exists some V ( · ) = 0 d,d which contradicts the assumption that all V s have independent rows. Under this setting, the density does not approach zero, which is the conclusion of Theorem 1 in [32].

C Proofs

In this section, we provide the proof of all the theorems and corollaries. Note that until Theorem 8, We ignore the graph component h γ and focus solely on the representational layer f θ and normalizing flow layer g ϕ .

Proof of Theorem 1 and Corollary 3. As f θ is arbitrary by assumption, we choose it in such a way that it maps a point in X k to a point inside the ball centered at z k with radius r k , denoted by B ( z k , r k ) ,

where z k ∈ Z with a minimal distance R between any two of them, i.e., d ( z k , z m ) > R, ∀ k, m ∈ [ K ] , and we define r > r k , ∀ k ∈ [ K ] . We then choose the normalizing flow to be,

We add the value of 1 in the normalizing flow to produce valid evidence measures. We also assume that N k = µ ( X k ) > 0 , where µ is the Lebesgue measure function.

The global minimum of UCE occurs when UCE is equal to 0 for every class. Recall that

Using (23), we consider an upper bound of the right-hand side by integrating over the larger region, that is,

According the recurrence relation of the digamma function: Ψ( x +1) = Ψ( x ) + 1 /x , we readily derive that,

Taking the limit of the right-hand side yields

It is straightforward for the following two limits to hold,

thus leading to

On the other hand, as Ψ( ∑ k ∈ [ K ] g k ( z )) -Ψ( g k ( z )) ≥ 0 , we have

which implies that UCE → 0 as r → 0 . For r = 0 , UCE is equal to zero, which leads to Corollary 3.

Proof of Theorem 4. Wedenote the parameters of g ϕ that represent the true analytic solution ˆ ϕ = ϕ ( θ ) . Note that this is a function with respect to the choice of θ , that is, the true distribution is dependent on the representational mapping. In this proof, we focus on finding a value of θ s.t.,

As the true distribution is dependent on the representational mapping, we should consider a joint minimization problem with respect to both the representation map and density distribution.

We will separate the proof into two cases. Specifically, we prove Case 1 by contradiction, showing that if the set Z k mapped to by f θ from X k has a non-zero measure, then the global minimizer UCE = 0 can not be achieved. We then prove Case 2, under the assumption that a true analytical solution may achieve density evidence at a point, by showing that we may achieve the global minimizer on a point set.

Case 1: Non-Zero Measure. Suppose the true distribution on this set is a non-degenerate distribution. As the natural definitions of a probability distribution 1 = ∫ Z k dµ, the UCE loss can be expressed by

In order for the measure of Z k to have a density of ϵ > 0 , there exists a subset of Z k with non-zero measure δ k > 0 , denoted Z ∗ k . Using similar techniques as the proof of Theorem 1 in reverse, we obtain,

then for some k ∈ [ K ] there exists some Z ∗ k ,

As Ψ is strictly increasing, then δ 2 = Ψ( K + N k ϵ ) -Ψ(1 + N k ϵ ) > 0 , which implies that

Therefore, we prove that if f θ maps to a measurable set, the UCE loss is necessarily non-zero.

Case 2: Zero Measure Sets. Corollary 3 shows that the zero UCE is achievable. If Case 1 fails, then we can conclude that only on a disjoint set Z k with measure 0 for each k is permissible to achieve the UCE to be 0. The exact choice of this set depends on the precise definitions of the probability distributions on a point set and their ability to achieve infinite densities. Here we constrain these possibilities by requiring the range of f θ to have non-zero measure or to be a point set if having zero measure 2 .

Proof of Theorem 6. Pick x ∈ X s.t. d ( x , x k ) > δ for x k ∈ X k see that for any f θ where θ ∈ Γ we have that x is necessarily not mapped to z k ∈ Z (if it were mapped in Z then the preimage would contain it and thus we would have d ( x , x k ) < δ , which is a contradiction with our selection of x ). Recall that the density for any point mapped to z k is infinite. That is the density of the associated points mapped to the point set is necessarily infinite and the density of points mapped elsewhere is necessarily smaller, namely 0, with the associated evidence 1 . Our selected x then has no evidence in favor of it belonging to a class k while any point in x k ∈ X k must have infinite evidence by our choice of a well-fit θ .

Proof of Corollary 7. Notice that we can choose two types of such θ ,

Case 1 Let θ for class k be chosen such that,

Case 2

If x ∈ X k is mapped to z k then it is endowed with infinite density, moreover, it is believed to be an ID node belonging to class k . Thus, the nearby OOD being detected for these UCE minimizers is determined by arbitrary choice.

Proof of Theorem 8. First note that the ID nodes are mapped to have infinite evidence achieved at the points in the latent space Z k . As the representations of the OOD nodes are in Z k they are also endowed with infinite evidence. That is graph layers can only help separate nodes by pulling them towards the center of their own classes w.r.t. to the representation space this is only helpful if their representations are separate to begin with.

Lastly, we give a toy example showing heuristically that the proposed regularization yields a better separation of the OOD nodes from IDs, compared to the original GPN model without the distance-based regularization.

Example 11. Consider two ID classes (Class 1 and Class 2) and one OOD class with the following construction:

All nodes belonging to Class 1 have feature values sampled from x (1) = [1 , 0 , 0] .

All nodes belonging to Class 2 have feature values sampled from x (2) = [ -1 , 0 , 0] .

All nodes belonging to the OOD class 2 have feature values sampled from x ( OOD ) = [0 , 1 , v ] .

We sample v from the uniform distribution U ( -1 , 1) independently for each sample in each class.

All nodes are connected to every node within their own class, leading to a graph of homophily 1.

Suppose the density function is true density distribution

Denote the PPR layer by ˆ h that uses the right normalized adjacency matrix AD -1 rather than symmetrically normalized D -1 / 2 AD -1 / 2 , used in APPNP.

Suppose f θ is a linear function (i.e. no activation function) explicitly, W =   W 11 W 12 W 21 W 22 W 31 W 32   .

Then GPN with our regularization can learn an embedding that makes it possible to separate classes 1, 2, and OOD nodes. Without regularization, OOD nodes lie between ID classes in the latent space.

Proof. A simple calculation for the project leads to

Clearly, the values of W 31 and W 32 would be smaller with the distance minimization term applied than without, as v is selected randomly. Moreover neither W 31 nor W 32 affects the model's ability to separate the two classes as desired. We explicitly calculate both UCE and the distance-based regularization in the objective function, while ignoring the Dirichlet regularization, thus leading to the following objective function,

First, we explicitly work out the distance-based regularization term

which is minimized when W 31 , W 32 go to zero.

Next, we consider the UCE loss portion. See that as we estimate the true density using g we will have no overlap between the two distributions Z 1 , Z 2 . We are left with W =   ,

If W is to remain full rank this will necessarily require either W 21 or W 22 to be non-zero. Thus OOD nodes will be mapped as we see in (41) to some distinct values - which can be separated after the application of APPNP as we expect APPNP to only average the values within each class.

D Additional Experimental Details

D.1 Descriptions of Baselines

Graph-based Kernel Dirichlet distribution Estimation (GKDE) [39]: Based on the high homophily property of most graphs (neighboring nodes tend to share the same class label), GKDE derives the evidence with the help of the node-level distances (shortest path in the graph) with training nodes belonging to the same class.

Label Propagation (LP) [32]: Following the idea of GKDE, LP collects the evidence by relying on the density of labeled nodes in neighborhoods rather than distance. An initial condition per class is defined and then a Personalized Page Rank is used as the diffusion.

VGCN-Energy [21]: It is a GCN-based model with energy score as the uncertainty estimation which maps each node to a single, non-probabilistic scalar called the energy. The energy score can be calculated as follows

where l is the predicted logits of a neural network and temperature parameter T = 1 .

GKDE-GCN [39]: GKDE-GCN utilizes a GCN network to estimate the multisource uncertainty by a Dirichlet distribution and then sample probability as well as the class prediction. The evidence derived from the aforementioned GKDE is as a teacher of concentration parameters of Dirichlet Distribution, and another deterministic GCN predicting the probability is used as a teacher for sampled probability. The overall loss is composed of the KL divergence between these two teachers with the corresponding distribution and Bayes risk with respect to the squared loss of sampled class prediction.

APPNP [12]: Given that message passing neural network suffers from the over-smoothing problem that limits the depth of the neural network, APPNP proposed to decouple the prediction and propagation where the prediction depends on the node features and propagation depends on interactions between nodes through edges. APPNP first uses any kind of neural network to embed the input space and diffuses information with a personalized page rank. For large graphs, they use power iteration to approximate a topic-sensitive page rank.

GPN [32]: GPN applies a normalizing flow to estimate the density of each class in the latent space embedded with an encoding network and then propagates the scaled density as the evidence.

D.2 Description of Datasets

We use three citation networks, labelled by CoraML, CiteSeer, Pubmed [4], two co-purchase Amazon datasets [31], labeled by Computers and Photos, two coauthor datasets [31], labeled by CoauthorCS and Physics, and a large dataset OGBN Arxiv [16]. We use the same train/val/test split of 5/15/80 as [32]. The details of the graphs and setups for the OOD detection are provided in Table 4.

D.3 Hyper-parameter tuning

We follow the same setting with [32]. In detail, we use the Adam optimizer with a learning rate of 0.01. For VGCN-Energy, we use a temperature of T = 1 . 0 . We carefully tune three hyperparameters: the distance-based regularization weight, Dirichlet entropy weight, and activation functions. We select the best parameters for each dataset separately that returns the highest validation cross-entropy. The detailed hyperparameters configuration is as Table 5.

We also consider the following activation functions in the encoding network with element-wise operations,

ReLU is the most popular activation function used in the hidden layer of neural networks, which brings efficient computation by only activating neurons with positive outputs. Sigmoid is popularly used for probability prediction because its output is always in the range (0,1) with a smooth gradient. GeLU has better nonlinearity and is widely used in Natural Language processing and computer vision. HardTanh is a more computation-efficient version of Tanh.

E Additional Experiments

E.1 Additional Experiments - OOD Detection

For Amazon Photos, Amazon Computers, Coauthor CS, Coauthor Physics, and OGBN Arxiv dataset, the OOD Detection results are shown in Table 6.

E.2 Additional Experiments - Misclassification Detection

For Amazon Photos, Amazon Computers, Coauthor CS, Coauthor Physics, and OGBN Arxiv dataset, the Misclassification Detection results are shown in Table 7.

E.3 Graph Distance Minimization

We plot the tSNE visualization of latent space with different distance-based regularization weights and symbol sizes denote the total evidence. We plot for coraML in Figure 2, CiteSeer in Figure 3, Coauthor CS in Figure 4, Coauthor Physics in Figure 5. With increasing weight, it tends to have a more separable latent representation for different categories while degenerate mappings occur when distance minimization is too large.

Alea: Aleatoric, Epi.: Epistemic, w/: with propagation, w/o: without propagation

Alea: Aleatoric, Epi.: Epistemic, w/: with propagation

CoraML

(a)

0

(b)

10

-

6

(c)

10

-

4

(d)

10

-

2

Class ID

Class ID

1

0

2

2

3

3

·OOD

OOD

Class ID

Class ID

12

0

0

12

3

3

·OOD

OOD

CiteSeer

(a)

0

(b)

10

-

6

(c)

10

-

4

(d)

10

-

2

Class ID

Class ID

1 2

0

3

OOD

OOD

Class ID

Class ID

0

0

12

2 0

3

3

OOD

OOD

CoauthorCS

(a)

0

(b)

10

-

6

(c)

10

-

4

(d)

10

-

2

Class ID

Class ID

0

1

UL

Class ID

Class ID

0

0

2

3

7

8

9

10

OOD

CoauthorPhysics

(a)

0

(b)

10

-

6

(c)

10

-

4

(d)

10

-

2

Class ID

Class ID

·1

0.

12

0

2

·OOD

OOD

Class ID

0

Class ID

012

1

2

·OOD

OOD

E.4 Graph Activation

In this subsection, we present the t-SNE visualizations of the learned representational space for various datasets in the following figures, without applying distance regularization. Instead, we introduce different activation functions. It is worth noting the notable distinction in quality when using the LogSigmoid activation function, which appears to be the smoothest among the activation functions employed on CiteSeer and Amazon Computers datasets. Once again, the size of each node corresponds to the square root of the learned evidence. Additionally, the color black indicates out-of-distribution (OOD) instances across all datasets, while distinct colors represent different classes.

CoraML

CiteSeer

PubMed

(a) RELU

(b) LogSigmoid

(c) HardTanh

Class ID

Class ID

Class ID

0

012

0

12

12

3

3

3

·OOD

·OOD

·OOD

Class ID

Class ID

Class ID

0

123

OOD

OOD

OD

K.!

Class ID

Class ID

Class ID

0

0

0+

1

1

OOD

OOD

OOD

8

3

6

AmazonPhotos

AmazonComputers

CoauthorCS

CoauthorPhysics

(a) RELU

(b) LogSigmoid

(c) HardTanh

Class ID

Class ID

Class ID

0

0

0

·1

1

1

2

2

3

4

OOD

OOD

OOD

Class ID

Class ID

Class ID

7.

0

0

1

0

1

2

2

3

4

OOD

OOD

OOD

Class ID

Class ID

Class ID

0

0

0

1

2

2

2

3

4

4

5

10

DOD

Class ID

Class ID

Class ID

0

0

0

·1

1

·1

2

2

2

OOD

OOD

OOD

E.5 Ablation Study

We show the full ablation study on three datasets: CoraML, CiteSeer and PubMed in Table 8. Table 8: Ablation Study with OOD Detection task (cont.)

* Alea: Aleatoric, Epi.: Epistemic, w/: with propagation

GPN is the original results from the GPN paper with default hyperparameters and ReLU as the middle activation function, GPN-CE is the original GPN model with re-tuned dirichlet entropy regularization weight; GPN-CE-ACT is the original GPN model with re-tuned entropy regularization weight and activation function; GPN-CE-ACT-GD/(Ours) add the distance-based regularization term and tuned the two weights and activation function.

Peer reviews:
comments: SUMMARY:
This paper focuses on uncertainty quantification in node classification. The authors examine the drawbacks of UCE. Inspired by our theoretical analysis, they introduced a distance-based regularization technique to obtain a better representation network for the uncertainty quantification. Experimental results show the superior performance of the proposed approach compared to the current state-of-the-art methods in out-of-distribution (OOD) detection and misclassification detection.

STRENGTHS:
1. This paper is well motivated, it discusses the limitations of GPN model on semi-supervised node classification tasks, and it provides detailed theoretical analysis to reveal the limitations of GPN for detecting OOD nodes

2. This paper proposes a distance regularizer to overcome the above limitation.

3. This paper provides extensive experimental results that demonstrate that the proposed method can achieve state-of-the-art performance on both out-of-distribution (OOD) detection and misclassification detection.

WEAKNESSES:
This paper is not in my area, I'm not familiar with existing work about both node classification and uncertainty qualification. So I cannot assess the contribution of this work reliably.

QUESTIONS:
The authors should better discuss the limitation of their work.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This manuscript studies the uncertainty quantification for the non-independent node-level prediction problem. The proposed framework is built upon the graph posterior network (GPN) with minimizing uncertainty cross-entropy as the main loss function. A distance-based regularization technique is further proposed to learn representational space mappings in order to handle OOD samples.

STRENGTHS:
The topic of quantifying the uncertainty of GNN predictions is timely and interesting. The overall presentation is easy to follow.

I checked the overall theoretical contributions and haven't found major flaws. 

The proposed two regularization terms are validated by extensive experiments.

WEAKNESSES:
I am not seeing major drawbacks, except there are some points related to the problem definition.

QUESTIONS:
1. How can Eq. (1) and (2) derive the aleatoric uncertainty and epistemic uncertainty? The authors do not mention the rationale behind this derivation, which makes it hard to understand the whole process.

LIMITATIONS:
I am not seeing major drawbacks.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The paper proposes a distance-based regularization to improve the OOD detection and misclassification detection tasks on the graph node classification. The regularization enforces the clustered OOD nodes to be close in the latent space.

STRENGTHS:
- The paper includes comprehensive experiments on multiple commonly used benchmark dataset and detailed ablation study. 

- The paper provides theoretic analysis on the limitation of the GPN.

WEAKNESSES:
- Equation 2 defines aleatoric and epistemic uncertainty. Why are they defined in this way? Is there any reference or reason?

- The writing is not clear. In section 4.1, there are a lot of theorems, but a sentence is missing to summarize what is the limitation of the UCE. Although at the beginning of section 4, there is a summary of those theorems, but it fails to present the overall idea of what the limitation is. Also, it is hard to connect Section 4.1 to 4.2. Why would the limitation of UCE loss mentioned in section 4.1 motivate the distance minimization on the graph?

- The proposed model improves AUPRC but has worse AUROC. It is not easy to tell whether the model is better. How does the model manage to improve AUPRC by 50% while remaining AUROC unchanged in Table 2? How could a model with 95% AUPRC has only 80% AUROC? Could the authors plot out the ROC and PR curves?

QUESTIONS:
The authors should improve writing in the revision by providing more contexts on the claims and definitions. It is also questionable on the results in the Table 2. I hope the author can clarify that.

LIMITATIONS:
Yes, the authors addresses limitations in the theoretic analysis.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n## Post Rebuttal Update \n\nI have engaged with the authors for the rebuttal, and found their responses informative, prompting me to increase my score from a 7 to an 8.\n\n## Original Review \nThe paper proposes theoretical results for Graph Posterior Networks (GPNs) that use the uncertainty cross-entropy loss (UCE), and shows that it is possible to trivially set the uCE loss to zero under certain conditions. The paper then proposes a way to regularise the UCE loss using distance, ensuring that OOD nodes that are clustered together are clustered together in latent space.\n\nSTRENGTHS:\nI like the theoretical analysis section of this paper. It constructs a way of obtaining zero UCE for a pathological example, showing that there are improvements required to be made to the GPN+UCE framework. Furthermore, it incorporates the fact that OOD points are not explicitly modeled in the UCE framework, and suggests ways to incorporate this. I also find the baselines considered pretty comprehensive, and the authors have done a good job covering a range of techniques. I also like the further ablations the paper has, such as selecting activation functions as well, using a validation CE loss, and I like that this has been properly ablated against. In general, I found the paper well motivated and written. I went through the proofs of Theorem 1-6 and found them accurate to my admittedly limited knowledge.\n\nWEAKNESSES:\nI have a few criticisms of the paper that I would like to get addressed - \n\n1. I would like a thorough description of all the assumptions made for Theorem 1, and where these assumptions would be violated or not, maybe with a toy example. For example, I'm not sure how strict the assumption \"If the underlying distribution of feature vectors belonging to class k, denoted by $X_k$ disjoint to each other,\" is, and would like a discussion of the drawbacks/limitations of the assumptions made in Theorem 1.\n2. Is the ball $B\\left(\\mathbf{z}_k, r_k\\right)$ guaranteed to have finite volume necessarily? There are pathological behaviours in neural networks that use ReLU activations, where it is possible to obtain high softmax scores (i.e. class probabilities) arbitrarily far from the training manifold [1], and this would make the volume of the ball very large potentially in some cases. \n\n\n[1] https://arxiv.org/abs/1812.05720\n\nQUESTIONS:\nIn Table 2, it seems like the regularisation consistently gives lower AUROC but higher AUPR compared to the GPN baseline. Can you explain why this might be the case?\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}