SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to detect the OOD nodes, we need a good representation, e.g.,  $f_{\theta}(\mathcal{X}_{[K]}) \cap f_{\theta}(\mathcal{X} - \mathcal{X}_{[K]}) = \emptyset$ . However, it is possible that  $\cup_{k=1}^{K} f_{\theta}^{-1}(\mathcal{Z}_k) = \mathcal{X}$ , which implies that any OOD node in the feature space is mapped to the same distribution of an ID class in the representation space. To this end, we require the preimage,  $f_{\theta}^{-1}$ , to be well-behaved in the sense that  $f_{\theta}^{-1}(\mathcal{Z}_k)$  must be contained within a bounded region near the support of the true distribution  $\mathcal{X}_k$ . Explicitly, we require there exists a constant  $\delta > 0$  such that  $d(\mathbf{x}, \hat{\mathbf{x}}) < \delta$ ,  $\forall \hat{\mathbf{x}} \in f_{\theta}^{-1}(\mathcal{Z}_k)$  and  $\forall \mathbf{x} \in \mathcal{X}_k$  for each  $k$ . The choice of  $\delta$  depends on additional information about the dataset. An excessively large  $\delta$  causes overlap between classes, thus increasing the likelihood of improperly identifying the OOD nodes as ID. On the other hand, a relatively small  $\delta$  forces the model to overfit, which labels the non-training nodes as OOD.
We characterize in Theorem 6 that under certain conditions, OOD can be detected correctly if they are far away from the ID distribution. One such condition is that the function  $f_{\theta}$  is well-fit, meaning that it maps the support of  $\mathcal{X}_k$  inside  $B(\mathbf{z}_k, r_k)$  for every  $k$  in  $[K]$ . We denote a set of all well-fit functions by  $\Gamma$ . Please refer to Figure 1 for a geometric illustration of these relevant quantities.
Theorem 6 (Far OOD). Under the same assumptions in Theorem 4 and two additional assumptions: (i)  $\mathcal{X}_k$  is bounded and (ii) for any  $\hat{\mathbf{x}}\in f_{\theta}^{-1}(\mathbf{z})$  with  $\mathbf{z}\in \mathcal{Z}_k$ , there exists  $\mathbf{x}_k\in \mathcal{X}_k$  such that  $d(\mathbf{x}_k,\hat{\mathbf{x}}) < \delta$ , if the true distribution of OOD nodes does not overlap with the region  $\cup_{\theta \in \Gamma}\cup_k\cup_{z_k\in \mathcal{Z}_k}\{f_\theta^{-1}(z_k)\}$ , then minimizing UCE can learn a representation projection of  $f_{\theta}$  that detect all the OOD points farther than  $\delta$  from  $\mathcal{X}_k$ .
Corollary 7 (Near OOD). Following Theorem 6, if  $\hat{\mathbf{x}}\in \cup_{\theta \in \Gamma}\cup_k\cup_{z_k\in \mathcal{Z}_k}\{f_\theta^{-1}(z_k)\}$  follows the distribution of OOD nodes, then the classification of  $\hat{\mathbf{x}}$  depends on the choice of  $\theta \in \Gamma$ .
The main purpose of Theorem 6 is to show a desired behavior for OOD detection is induced by point-wise effects. In practice, we suggest the careful construction of  $f_{\theta}$ 's topology coupled with proper regularization terms to achieve the point-wise effect.
Lastly, we consider the setting of GPN to use a graph layer after the feature-wise evidence predictions.
Theorem 8. Under the following conditions: (1) The features of some class 0 and OOD nodes belong to  $S$ ; (2) The OOD nodes are only connected to class 0 and themselves; (3) Other nodes belong to other regions i.e.  $\mathbf{x} \in X - S$  if  $\mathbf{x} \notin \mathcal{X}_0 \bigcup \mathcal{X}_{\mathrm{OOD}}$ ; (4) Other nodes' features are non-degenerate in their associated region; and (5) the endowed graph neural network layer must produce evidence for each node between the highest and lowest feature evidence found among its neighbors. A graph with arbitrary homophily can achieve a global minimum on UCE with the associated OOD nodes achieving arbitrarily large evidence, while simultaneously having perfect accuracy.
Limitations and Discussions. In Theorem 8, we provide a special situation where the GPN architecture may fail to detect OOD nodes by predicting large evidence values for belonging to class 0. In addition, we show in Appendix B that if GPN has bad initial feature predictions, even ideal graph construction coupled with an ideal graph neural network (with a homophily degree 1.0) fails to
prevent the OOD nodes from being misclassified as ID nodes. For homophily graphs with degrees less than 1.0, the majority of nodes for each class have similar evidence values after graph diffusion layers, except that the nodes between some pairs of classes may have different evidence values.
4.2 Distance-Based Regularization
As discussed in Section 4.1, the UCE loss function alone is insufficient to learn a representation space that separates OOD from ID nodes using the GPN model. We propose a heuristic remedy that enforces distance minimization on the graph. Ideally, we should design a distance formula that can preserve the distance relationship among all the feature vectors. However, distance preservation likely increases variation in the latent space as we cannot compress the classes' support in the representation space to be arbitrarily small, while simultaneously preserving distances.
Instead, we consider distance minimization, as it helps prevent the model from discarding relevant features while decreasing variation between nodes in the representation space. Here we give two formulations directly. The simpler, theorem-motivated term, is the distance regularization on the latent space,
$$
\mathrm {R} _ {D} \left(f _ {\theta} (X); \mathcal {G}\right) = \sum_ {i, j \in \mathcal {E}} \| f _ {\theta} (\mathbf {x} _ {i}) - f _ {\theta} (\mathbf {x} _ {j}) \| ^ {2}. \tag {9}
$$
The regularization encourages nearby points in the graph representation to remain nearby in the latent space. In other words, this regularization (9) discourages the overlap  $f_{\theta}(\mathcal{X}_{[K]}) \cap f_{\theta}(\mathcal{X} - \mathcal{X}_{[K]})$ .
We also minimize the "distance" on the produced evidence through a divergence-based regularization,
$$
\mathrm {R} _ {\alpha} \left(\boldsymbol {\alpha} ^ {\text {f e a t}}; \mathcal {G}\right) = \sum_ {(i, j) \in \mathcal {E}} \operatorname {d i v} _ {\mathrm {k l}} \left(\boldsymbol {\alpha} _ {i} ^ {\text {f e a t}}, \boldsymbol {\alpha} _ {j} ^ {\text {f e a t}}\right) + \operatorname {d i v} _ {\mathrm {k l}} \left(\boldsymbol {\alpha} _ {j} ^ {\text {f e a t}}, \boldsymbol {\alpha} _ {i} ^ {\text {f e a t}}\right), \tag {10}
$$
where  $\mathrm{div}_{\mathrm{kl}}$  refers to the Kullback-Leibler divergence and two symmetric terms are considered. This divergence-based formulation likely decreases the variation in evidence between neighboring nodes, because high variation in the latent space between neighboring nodes need not be mapped to similar evidence.
In summary, we augment the GPN mode with either one of the proposed regularization terms in (9) and (10), thus leading to the objective function as follows,
$$
\mathcal {L} = \operatorname {U C E} (\mathcal {A}, Y) - \lambda_ {1} \underbrace {\sum_ {i \in \mathbb {L}} \mathbb {H} \left(\operatorname {D i r} \left(\boldsymbol {\alpha} _ {i}\right)\right)} _ {\text {e n t r o p y r e g u l a r i z e r}} + \lambda_ {2} \underbrace {\operatorname {R} \left(f _ {\theta} (X) ; \mathcal {G}\right)} _ {\text {p r o p o s e d r e g u l a r i z e r}}, \tag {11}
$$
with two positive parameters  $\lambda_1, \lambda_2$ . The first term is the standard UCE loss function. The second term is regarded by GPN as an entropy regularizer. The last term, R, is chosen to be either  $\mathbf{R}_D$  or  $\mathbf{R}_{\alpha}$ , a decision implemented through hyperparameter tuning. We have included a theoretical result in Appendix B that provides a rationale for the proposed distance regularizations.
In this section, we conduct extensive experiments on two tasks of OOD detection and misclassification detection. We compare the proposed framework (11) for uncertainty estimation of semi-supervised node classification using 8 datasets with a comparison to 5 baseline methods. The code is available at https://github.com/neoques/Graph-Posterior-Network.
Datasets We use three citation networks (i.e. CoraML, CiteSeer, Pubmed) [4], two co-purchase datasets [31] (i.e. AmazonComputers, AmazonPhotos), two coauthor datasets [31] (i.e. CoauthorCS and CoauthorPhysics) and a large dataset OGBN Arxiv [16]. A detailed description of these datasets is in Appendix E. We show the result of three citation datasets in the main paper and the remaining results in Appendix F.
Baselines We present the results for uncertainty estimation using five baseline methods. Among these, two evidence collection models, namely graph kernel density estimation (GKDE) [39] and label propagation (LP) [32], assuming that OOD nodes are located far away from the training nodes, while easily misclassified nodes reside near the boundaries between classes. We compare to a modified GCN model, referred to as VGCN-Energy [21], a Bayesian-based model, called GKDE-GCN [39], and GPN [32] as baselines in our evaluation. We also introduce a Graph Neural Network called APPNP [12] as one baseline for the misclassification detection task and report the ROC score. Details of these baselines can be found in Appendix D.
Metrics To assess the classification performance of ID nodes, we rely on the metric ID-ACC, which calculates the fraction of correct predictions among all predictions. As for evaluating uncertainty estimation, we employ the metrics AUC-ROC and AUC-PR as evaluation measures. The rankings are based on the scores of epistemic or aleatoric uncertainty. OOD detection is treated as a binary classification task, where the positive class corresponds to OOD nodes and the negative class pertains to ID nodes. Please refer to (2) for the calculation of aleatoric uncertainty and epistemic uncertainty. For the Dirichlet-based models, the epistemic uncertainty has a similar practical interpretation to vacuity in the belief theory, assessed using AUC scores. On the other hand, in VGCN-Energy, the calculation of epistemic uncertainty is based on the energy value and is represented as  $u_{i}^{\mathrm{eps}} =$  energy. The misclassification detection task is also a binary classification problem, where the positive cases correspond to wrongly classified nodes and the negative cases represent correctly classified nodes. The calculation of uncertainty for misclassification detection is performed in the same manner as OOD detection except that  $u_{i}^{\mathrm{epis}} = -\max_{k}\alpha_{i}^{k}$ . Prior studies [32, 39] has indicated that aleatoric uncertainty is generally more effective for identifying misclassifications, whereas epistemic uncertainty is more appropriate for detecting out-of-distribution instances.
Model Setup For all the baseline methods, we maintain consistency by employing the same set of model hyperparameters as provided by GPN. Specifically for some model hyperparameters such as latent dimension and weight decay, we adopt the same settings as GPN. Inspired by [26], we explore multiple choices of activation functions in the representation networks. In addition to the default ReLU used in GPN, we experiment with Sigmoid and GELU activation functions. Through empirical evaluation, we discover that the choice of activation function significantly impacts the performance of certain datasets, which is demonstrated in Appendix E.4. Besides, hyperparameters that we tune include entropy regularization weight, distance-based regularization format (whether  $\mathbf{R}_D$  or  $\mathbf{R}_{\alpha}$ ), and weighting parameters  $(\lambda_1,\lambda_2)$ , which are optimized based on the validation cross-entropy for each specific dataset. For a comprehensive overview of the hyperparameter configuration and ablation study, please refer to Appendix D.
OOD Detection OOD detection aims to detect whether an input example is OOD given the predicted uncertainty estimation. For the semi-supervised node classification, we adopt the Left-Out-Classes setting where we assume several categories as OOD (details can be found in Appendix D), as considered in [7, 39]. Different from the independent input setting, we retain the OOD nodes in the graph but exclude their labels from the training and validation sets. This implies that the loss function does not involve OOD labels, but the model has encountered the OOD node features during the training phase. Similarly to [32], we also remove the last graph propagation layer for comparison as "w/o network" where the final result only depends on the node features and no graph structure involved. This configuration, referred to as the "w/o network" setting, results in a final output that solely relies on the node features, with no involvement of the graph structure.
The results on CoraML, CiteSeer, and PubMed are presented in Table 1 and the results on the other 5 datasets are shown in Table 6 in Appendix E. Our model achieves the best ID accuracy for four datasets and demonstrates comparable performance to GPN for the remaining four datasets. Furthermore, we observe an improvement in the ROC (Receiver Operating Characteristic) rankings based on predicted epistemic uncertainty with propagation, ranging from  $+1\%$  to  $+8\%$  compared to GPN. Consistent with previous studies [32], our results demonstrate that prediction models incorporating evidence propagation consistently outperform those without propagation across all datasets. This observation highlights the significant impact of graph structure on uncertainty estimation. Moreover, when comparing aleatoric uncertainty and epistemic uncertainty as ranking scores, we find that epistemic
uncertainty outperforms aleatoric uncertainty in the OOD detection task. This finding aligns with literature [39, 32] and emphasizes the superiority of epistemic uncertainty for OOD detection.
Table 1: AUROC and AUPR for the OOD Detection
Data	Model	ID-ACC	AUROC	AUPR
Alea w/	Epi w/	Epi w/o	Alea w/	Epi w/	Epi w/o
CoraML	LP	86.40	83.78	80.86	n.a.	74.80	71.15	n.a.
GKDE	83.02	74.46	71.86	n.a.	66.19	64.05	n.a.
VGCN-Energy	89.66	81.70	83.15	n.a.	75.67	78.44	n.a.
GKDE-GCN	89.33	82.23	82.09	n.a.	75.88	77.03	n.a.
GPN	88.51	83.25	86.28	80.95	75.79	79.97	72.81
Ours	90.06	83.94	87.20	76.12	76.26	80.36	63.32
Citeseer	LP	57.34	65.99	67.54	n.a.	48.12	48.59	n.a.
GKDE	49.62	63.75	63.91	n.a.	56.74	56.79	n.a.
VGCN-Energy	70.79	72.16	76.08	n.a.	53.71	58.35	n.a.
GKDE-GCN	70.76	73.34	76.19	n.a.	54.25	59.07	n.a.
GPN	69.79	72.46	70.74	66.65	55.14	50.52	44.93
Ours	72.51	75.22	78.98	73.21	62.30	58.63	52.73
PubMed	LP	89.18	80.32	79.64	n.a.	71.01	72.98	n.a.
GKDE	88.16	69.66	68.47	n.a.	55.81	54.33	n.a.
VGCN-Energy	94.77	72.58	72.63	n.a.	60.54	60.63	n.a.
GKDE-GCN	94.66	73.53	74.47	n.a.	61.36	61.96	n.a.
GPN	94.08	71.84	73.91	71.2	57.92	67.19	59.72
Ours	93.84	75.23	81.76	77.79	60.75	78.16	69.19
Alea: Aleatoric, Epi.: Epistemic, w/: with propagation, w/o: without propagation
AUROC and AUPR for the OOD Detection: ID-ACC means the accuracy on ID nodes. AUROC and AUPR scores are given for OOD detection based on different uncertainty metrics, where [Alea w/] is the aleatoric score with propagation layer, [Epi w/] is the epistemic score with propagation layer and [Epi w/o] is the epistemic score without propagation. n.a. means either model or metric not applicable.
Misclassification Detection In addition to OOD detection, we conduct misclassification detection on the clean graph for evaluating the predictive uncertainty estimation. Table 2 presents the results for three datasets, while Table 7 in Appendix D is for the other 5 datasets. We observe a significant improvement ranging from  $+12\%$  to  $+50\%$  in our model's AUC-PR scores. While it is true that our method performs worse than the best of the baselines in terms of AUROC, the differences are within approximately  $3\%$  for six of the eight datasets.: Amazon Computers, Amazon Photos, Coauthor CS, Coauthor Physics, and ODBG Arxiv, and PubMed. Despite having a lower AUROC compared to the best of the baselines, our method exhibits a higher AUPR. This suggests that our method may excel at identifying true positives among the top-ranked nodes when compared to GPN, while the best baselines may be more effective at distinguishing between true positives and negatives among the lower-ranked nodes.
Table 2: AUROC and AUPR for the Misclassification Detection
Data	Model	AUROC	AUPR
Alea w/	Epi w/	Alea w/	Epi w/
CoraML	APPNP	83.64	n.a	48.39	n.a
VGCN-Energy	81.02	n.a	48.30	n.a
GKDE-GCN	80.80	76.83	49.61	45.87
GPN	81.19	78.10	49.51	44.42
Ours	75.8	69.85	89.95	88.20
CiteSeer	APPNP	73.55	n.a.	51.70	n.a.
VGCN-Energy	74.64	n.a	48.30	n.a.
GKDE-GCN	75.45	73.83	54.78	53.57
GPN	75.89	74.16	60.78	59.32
Ours	69.15	68.62	72.67	72.36
PubMed	APPNP	80.98	n.a.	37.79	n.a.
VGCN-Energy	81.16	n.a	38.24	n.a
GKDE-GCN	80.95	73.99	39.64	33.19
GPN	80.46	75.38	40.74	35.11
Ours	80.13	72.87	95.41	92.79
Alea: Aleatoric, Epi.: Epistemic, w:/ with propagation, w/o: without propagation
AUROC and AUPR for misclassification detection: AUROC and AUPR scores are given for misclassification detection based on different uncertainty metrics, where [Alea w/] is the aleatoric score with propagation layer, [Epi w/] is the epistemic score with propagation layer and [Epi w/o] is the epistemic score without propagation. n.a. means either model or metric not applicable
Our proposed model differs from the GPN model in three main aspects. First, we use validation cross entropy (CE) instead of hold-out datasets to select hyperparameters. Second, we consider the
activation function for the MLP layers as one of the hyperparameters for selection. We expect feature value rescaling through non-linear activation of feature values to affect the density predictions. Third, we incorporate one of the proposed distance-based regularization terms to the loss function used in GPN. We conduct an ablation study to demonstrate the contribution of these three components. The results for CiteSeer and PubMed are shown in Table 3 and the remaining datasets are included in Appendix E. Hyperparameter tuning using validation cross-entropy improves GPN's performance, especially in cases where the choice of activation function has a significant impact on specific datasets. Additionally, we consistently observe performance enhancements from the distance-based regularization in both datasets, demonstrating the effectiveness of the proposed distance awareness regularization term.
Table 3: Ablation Study with OOD Detection task
Data	Model	ID-ACC	AUROC	AUPR
Alea w/	Epi w/	Epi w/o	Alea w/	Epi w/	Epi w/o
Citeseer	GPN	69.79	72.46	70.74	66.65	55.14	50.52	44.93
GPN-CE	70.98	74.20	73.75	68.41	58.12	53.55	46.60
GPN-CE-ACT	71.96	74.72	77.97	72.28	60.41	56.04	50.73
GPN-CE-GD	72.51	75.22	78.98	73.21	62.30	58.63	52.73
PubMed	GPN	94.08	71.84	73.91	71.2	57.92	67.19	59.72
GPN-CE	93.84	74.19	78.32	74.50	59.85	74.11	64.55
GPN-CE-ACT	93.84	74.19	78.32	74.50	59.85	74.11	64.55
GPN-CE-GD	93.84	75.23	81.76	77.79	60.75	78.16	69.19
Alea: Aleatoric, Epi.: Epistemic, w/: with propagation
GPN refers to the original GPN paper with its default hyperparameters and ReLU as the middle activation function, GPN-CE is the original GPN model with re-tuned Dirichlet entropy regularization weight based on validation cross-entropy; GPN-CE-ACT is the original GPN model with re-tuned entropy regularization weight and activation function based on cross-entropy; GPN-CE-GD/(Ours) adds the distance-based regularization term while tuning the two weights and activation function.
Our theoretical analyses mainly study the limitations of the UCE loss function when separating OOD from ID nodes in the learned representation space. If we include the entropy-based regularization term in Equation (6) with a sufficiently large weight  $\lambda$ , some of our theoretical findings may not remain applicable. However, the entropy-based regularization term is designed to favor smooth Dirichlet distributions but not to preserve the distance between OOD and ID nodes. We conjecture that the resulting loss function is still insufficient to learn a representation space that separates OOD from ID nodes, even though they are separable in the original feature space. In addition, our proposed regularization terms in Section 4.2 are more effective for homophily graphs than heterophily graphs, as neighboring nodes are less likely to belong to the same class in a heterophily graph than those in a homophily graph.
This paper contributed to a better understanding of uncertainty quantification for node classification. We investigated the limitations of the widely used UCE loss function. Motivated by the theoretical analysis, we proposed a distance-based regularization that helps learn a representation network that is more effective for the uncertainty quantification task. Experimentally, we demonstrated our approach outperforms the state-of-the-art in two specific applications of uncertainty quantification for node classification: OOD detection and misclassification detection.
This work is supported by the National Science Foundation (NSF) under Grant No #2220574, #2107449, #1846690, and #1750911. The work of Feng Chen is also supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (DOI/IBC) contract number 140D0423C0026. The US Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA DOI/IBC or the U.S. Government.
The supplementary materials are organized as follows. Appendix A provides the background knowledge on the Dirichlet distribution. In Appendix B we review the architecture of the graph posterior network (GPN) [32] together with our discussion on oversight of [32, Theorem 1]. Appendix C details the proofs of all the theorems and corollaries discussed in the main paper. We provide detailed descriptions of baseline models, datasets, and hyperparameter tuning for the experiments in Appendix D. Lastly, Appendix E includes more experimental results that we are unable to fit into the paper.
A Dirichlet Distribution
A non-degenerate Dirichlet distribution, denoted by  $\mathrm{Dir}(\alpha)$ , is parameterized by the concentration parameters  $\pmb{\alpha} = [\alpha_{1},\dots ,\alpha_{K}]^{\intercal}$  with  $\alpha_{k} > 1$  for  $k\in [K]$ . More specifically, the Dirichlet distribution with parameters  $\alpha_{1},\dots ,\alpha_{K}$  has a probability density function (pdf) given by
$$
\operatorname {p d f} (\mathbf {p}; \boldsymbol {\alpha}) = \frac {1}{B (\boldsymbol {\alpha})} \prod_ {k = 1} ^ {K} p _ {k} ^ {\alpha_ {k} - 1}, \tag {12}
$$
where  $\{p_k\}_{k = 1}^K$  belongs to the standard probability simplex, thus  $\sum_{k = 1}^{K}p_{k} = 1$  and  $p_k\in [0,1],\forall k\in [K]$ , and the normalizing constant  $B(\alpha)$  is expressed in terms of the Gamma function  $\Gamma (\cdot)$ , i.e.,
$$
B (\boldsymbol {\alpha}) = \frac {\prod_ {k} \Gamma \left(\alpha_ {k}\right)}{\Gamma \left(\Sigma_ {k} \alpha_ {k}\right)}. \tag {13}
$$
Under the semi-supervised learning setting, a set of labels is available, denoted by  $\mathbb{L} \subset \mathcal{V}$ . For  $i \in \mathbb{L}$ , the class label  $y_{i} \in \{1, \dots, K\}$  can be converted into a one-hot vector  $\mathbf{y}_{i}$  with  $y_{ik} = 1$  if the sample belongs to the  $k$ -th class and  $y_{ij} = 0$  for  $j \neq k$ . By arranging  $\alpha_{i}$  and  $\mathbf{y}_{i}$  into matrices  $\mathcal{A} := [\pmb{\alpha}_{i}]_{i \in \mathbb{L}}$  and  $\bar{Y} := [\mathbf{y}_{i}]_{i \in \mathbb{L}}$ , the UCE loss function is defined as:
$$
\operatorname {U C E} (\mathcal {A}, Y) = \sum_ {i \in \mathbb {L}} \sum_ {k \in [ K ]} y _ {i k} \left(\Psi \left(\alpha_ {i 0}\right) - \Psi \left(\alpha_ {i k}\right)\right), \tag {14}
$$
where  $\alpha_{i0} = \sum_{k=1}^{K} \alpha_{ik}$  and  $\Psi$  is the digamma function given by
$$
\Psi (x) = \frac {\Gamma^ {\prime} (x)}{\Gamma (x)}.
$$
B Detailed Framework of GPN
In this section, we review the architecture of GPN, followed by two examples that reveal a limitation of Theorem 1 in the GPN paper [32].