Revisiting Uncertainty Estimation for Node Classification: New Benchmark and Insights
Authors: Gleb Bazhenov
Denis Kuznedelev
Andrey Malinin
Artem Babenko
Liudmila Prokhorenkova
Venue: Submitted to ICLR 2023 (2023)
Keywords: uncertainty estimation
distribution shift
graph
node classification
benchmark

Abstract: Uncertainty estimation is an important task that can be essential for high-risk applications of machine learning. This problem is especially challenging for node-level prediction in graph-structured data, as the samples (nodes) are interdependent. Recently, several studies addressed node-level uncertainty estimation. However, there is no established benchmark for evaluating these methods in a unified setup covering diverse distributional shift. In this paper, we address this problem and propose such a benchmark together with a technique for the controllable generation of data splits with various types of distributional shift. Importantly, besides the standard feature-based distributional shift, we also consider shifts specifically designed for graph-structured data. In summary, our benchmark consists of several graph datasets equipped with various distributional shift on which we evaluate the robustness of models and uncertainty estimation performance. This allows us to compare existing solutions in a unified setup. Moreover, we decompose the current state-of-the-art Dirichlet-based framework and perform an ablation study on its components. In our experiments, we demonstrate that when faced with complex yet realistic distributional shift, most models fail to maintain high classification performance and consistency of uncertainty estimates with prediction errors. However, ensembling techniques help to partially overcome significant drops in performance and achieve better results than distinct models. Among single-pass models, Natural Posterior Network with GNN encoder achieves the best performance.

Under review as a conference paper at ICLR 2023
REVISITING UNCERTAINTY ESTIMATION FOR NODE CLASSIFICATION: NEW BENCHMARK AND INSIGHTS
Anonymous authors Paper under double-blind review
ABSTRACT
Uncertainty estimation is an important task that can be essential for high-risk ap- plications of machine learning. This problem is especially challenging for node- level prediction in graph-structured data, as the samples (nodes) are interdepen- dent. However, there is no established benchmark that allows for the evaluation of node-level uncertainty estimation methods in unified setup, covering diverse and meaningful distribution shifts. In this paper, we address this problem and pro- pose such benchmark, together with technique for the controllable generation data splits with various types distribution shifts. Importantly, we describe the shifts that are specific the graph- graph-structured data. Our benchmark consists of several graph datasets equipped with various distribution shifts on which we eval- uate the robustness of models and their uncertainty estimation performance. To il- lustrate the benchmark, we decompose the current state-of-the-art Dirichlet-based framework and perform an ablation study on its components. In our experiments on the proposed benchmark, we show that when faced with complex yet realistic distribution shifts, most models fail to maintain high classification performance and consistency of uncertainty estimates with prediction errors. However, ensem- bling techniques help to partially overcome significant drops in performance and achieve better results than distinct models.
INTRODUCTION
Uncertainty estimation is an important and challenging task with many applications in financial sys- tems, medical diagnostics, autonomous driving, etc. aims at quantifying the confidence of machine learning models and can be used to design more reliable decision-making systems. In particular, it enables one to solve such problems as misclassification detection, where the model has to assign higher uncertainty to the potential prediction errors, the model is required to yield higher uncertainty for the samples from an unknown distribution. Depending the source of uncertainty, it can be divided into data uncertainty. which describes the inherent noise in data due to the labeling mistakes or class overlap, and knowledge uncertainty, which accounts for insufficient amount of information for accurate predictions when the distribution of test data is different from the training one (Gal 2016 Malinin 2019).
-ittt-iiittiittionn (OOD) detection, when
The problem of uncertainty estimation for graph-structured data has recently started gain atten- tion. It is especially complex at the node level as one has to deal with interdependent samples that may come from different distributions, so their predictions can change significantly depending on the neighborhood. This problem has already been addressed in several studies, and the proposed meth- ods are commonly based on the Dirichlet distribution and introduce various extensions the Dirich- let framework (Sensoy et al.) 2018 Malinin & Gales] 2018 Malinin 2019 Charpentier et al.) 2020), such as graph-based kernel Dirichlet estimation (Zhao et al. 2020) or graph propagation of Dirichlet parameters (Stadler al. 2021).
However, the field of robustness and uncertainty estimation for node-level graph problems suffers from the absence of benchmarks with diverse and meaningful distribution shifts. Usually, the evalu- ation is limited to somewhat unrealistic distribution shifts, such as noisy node features (Stadler et al.) 2021) left-out classes (Zhao al. 2020 Stadler et al. 2021). Importantly, Gui et al. (2022) try to overcome this issue and systematically construct a graph OOD benchmark, in which they explic- itly make distinctions between covariate and concept shifts. However, the authors either consider

---

Under review as a conference paper at ICLR 2023
synthetic datasets ignore the graph structure when creating distribution shifts. The problem with the mentioned approaches is that, in real applications, distribution shifts can be much more complex and diverse, and may depend the global graph structure (for more detailed discussion, refer to Appendix C.. Thus, the existing benchmarks can be insufficient to reliably and comprehensively evaluate uncertainty estimation methods for graph-structured data. Therefore, the current status quo about the best uncertainty estimation methods for node classification remains unclear and requires further investigation.
In this work, we propose new benchmark for evaluating robustness and uncertainty estimation in general approach to
transductive node classification tasks. The main feature of our benchmark constructing the data splits with distribution shifts: it can be applied to any graph dataset, allows for generating shifts of different nature, and one can easily vary the sizes of splits. For demonstration purpose, we apply our method to 7 common node classification datasets and describe 3 particular strategies to induce distribution shifts. Using the proposed benchmark, we evaluate the robustness of various models and their ability to detect errors and OOD inputs. Thus, we show that the recently proposed Graph Posterior Network (Stadler et al. 202T] I consistently the best method for detecting the OOD inputs. However, the best results for the other tasks are achieved using Natural Posterior Networks (Charpentier et al., al,] [202]). We also confirm that ensembling often allows one to improve the model performance ensembles of GPNs achieve the best performance for OOD detection, while ensembles of NatPNs have the best predictive performance and error detection.
2 PROBLEM STATEMENT
We consider the problem of transductive node classification in attributed graph g - (A, Y) with an adjacency matrix A E {0, 10"x,, a node feature matrix X E Raxd and categorical targets vector Y E (1...., C}" We split the set of nodes V into several non-intersecting subsets depending on whether they are used for training, validation, or testing and if they belong to in-distribution (ID) or -uitrr--dittiiuution (OOD) subset. Let Y train denote the labels of train nodes Vusin- Given graph Guain = X, Y n, we aim at predicting the labels Y of test nodes Viest and estimating the uncertainty measure U, E IR associated with these predictions. The obtained uncertainty estimates are used to solve the misclassification detection and OOD detection problems.
PROPOSED BENCHMARK
This section describes our benchmark for evaluating uncertainty estimates and robustness distri- bution shifts for node-level graph problems. The most important ingredient of our benchmark is unified approach for the controllable generation of diverse distribution shifts that can be applied to any graph dataset. Our benchmark includes collection of common node classification datasets, several data split strategies, a set of problems for evaluating robustness and uncertainty estimation performance, and the associated metrics. We describe these components below.
3.1 GRAPH DATASETS
While approach can potentially be applied to any node classification node regression dataset, for our experiments, we pick the following 7 datasets commonly used in the literature: 3 citation networks, including CoraML, CiteSeer (McCallum et al.) 20000 Giles et al.] 1998 Getoor 2005
Sen et al., 2008) and PubMed (Namata et al., 2012)) 2 co-authorship graphs CoauthorPhysice and CoauthorCS (Shchur et al. (018). and co- co-purchase datasets AmazonPhoto and Amazon- Computers (McAuley et 2015:[Schurr et al. 2018).
3.2
DATA SPLITS
The most important ingredient of our benchmark is general generating data splits in a graph G to yield non ariviiiial yet reasonable distribution shifts. For this purpose, we make a distinction between the ID parts that are described by p( [X, A) and shifted (OOD) parts where the targets may come from a significantly different distribution p(Yau [X, A).
We define the following ID parts:
2

---

Under review as a conference paper at ICLR 2023
Train contains the nodes Vusia that are used for the regular training of models and repre- sent the only observations that take part in gradient computation;
Valid-IN In enables us to monitor the best model during the training stage by computing the validation loss for the nodes Vvalid-in and choose the best checkpoint;
Test-In -In is used for testing on the remaining in-distributtio nodes Viest-in and represents the most simple setup that requires the model to reproduce in-distribution dependencies. Both Valid-In In and Test- In parts are assumed to come from the exact same distribu- tion as Train.
At the same time, we introduce the following OOD parts:
Valid-Out contains the validation nodes Vealid-our that also can be used for monitoring but tends to be more difficult part of graph with potentially different dependencies;
Test-Out represents the most shifted part Vees and can be used for evaluating robust- ness of models to distribution shifts.
To construct a particular data split, we choose some characteristic o; and compute it for every node i V, as described in Section 3.3 This characteristic reflects some node property that may depend on features or graph structure. After that, we sort all nodes in ascending order of a; Some fraction of nodes with the smallest values of o; is considered be ID, while the remaining ones become and Test -Out based on their values of Ø; Importantly, this general split strategy is very flexible it allows one vary the size of the training part and to analyze the effect of this size on the robustness and the quality of uncertainty estimates. The type of distribution shift depends on the choice of a and can also be easily varied.
OOD and are split into Valid-an
In our experiments, we split the dataset in the following proportions. The half of the nodes with the smallest values of o; are assumed to be ID and are split into Train, Valid-In, and Test-In uniformly random in proportion 30%//0%//10% The second half contains the remaining OOD nodes split into Va Walid-Out and Test-On -Out in the ascending order a, proportion 10%/40%. As result, the Test-Out part has the most significant distribution shift.
3.3 DISTRIBUTION SHIFTS
To define our data splits, is necessary to choose some node characteristic or as a split factor. We aim to consider diverse characteristics which cover a variety of distribution shifts that may occur in practice. In a standard non-graph MI. setup, shifts typically happen only in feature space (or, more generally, the joint distribution of features and targets may become shifted). In graph learning tasks, there can be shifts specifically related to the graph structure: the training part can be biased towards more popular nodes or may consist of nodes from a particular region in the graph. Thus, we consider the following representative data split strategies.
Random This is a standard approach to constructing the data splits, where the nodes are selected uniformly random, i.e., is not realistic for practical applications but can be helpful for the analysis: it shows how well the model generalizes given that the distribution does not change. The random splitting strategy also allows for evaluating the robustness of models when the size of the training dataset varies.
take d, to be random position in a sorted list. This type of shift
Feature This approach represents a family of possible feature-based shifts that do not take into account the graph structure explicitly. There are multiple ways to construct such shifts, e.g., a split can be based on values of one particular feature (Gui et al. 2022). However, to follow our general setup described above, we base a split on a continuous characteristic that can be computed for any dataset. Namely, we project the original features
E IRd into IR22 via random linear transform W. where all entries Wiy are independent and come from (0, 1). After that, σ; is set to the distance between the node E V and the centroid of the projected data, so the most central nodes in terms of features are said to be ID, while OOD parts are close to periphery. This setup naturally corresponds to the situation when the training dataset consists of the most rypical elements, while some outliers may be encountered at the inference stage. Thus, this type of shift tests the robustness of models to ntn-standard feature combinations.
3

---

Under review as a conference paper at ICLR 2023
We visualize all the split strategies applied to the AmazonPhoto dataset in Figure 1 The figures for the remaining datasets can be found in Appendix A Here, one can see that the feature-base based split does not introduce a notable structural shift, i... the nodes are distributed across all regions of the graph. This fact is additionally confirmed by our analysis in Appendix B it is clear that there is no significant difference in the degree distribution and pairwise node distances between the ID and OOD parts. Our empirical observations confirm that the feature-based shifts are the easiest to handle by the considered methods.
(a) Feature
(b) PageRank
(c) Personalized PageRank
Figure 1: Visualization of data splits for AmazonPhoto dataset: ID is blue, OOD is red.
PageRank This strategy represents possible bias towards popularity. It is natural to expect the training set to consist of more popular items. For instance, in the web search, the importance of pages in the internet graph can be measured via PageRank (Page et al. 1999). For this application, the labeling of pages should start with important ones, since they are visited more often. Similar situations may happen for social networks, where it is natural to start labeling with the most influen- tial users, or citation networks, where the most cited pages should be labeled first. However, when applying the model, it essential make accurate predictions on less popular elements. Motivated by that, we introduce a PageRank-base for every node nodes with smaller values of PR (i.e., less important ones) come to the OOD subsets.
split. In particular, we compute the PageRank (PR) values V and define the measure 0, as the negative PR score, which means that the
As can be seen in Figure [] the PageRank- based split separates the most important nodes that belong to the cores of large clusters and the structural periphery, which consists of less important nodes in terms of PageRank. Our analysis in Appendix [B] confirms this observation: the degree distribution changes significantly across the ID and OOD subsets, tending to higher values for the ID nodes. The distance between such nodes also appears to be smaller on average. Our experiments prove that such a structural distribution shift creates a more severe challenge for the considered methods.
Personalized PageRank This strategy is focused on a potential bias towards locality, which may happen when labeling is performed by exploring the graph starting from some node. For instance, this may occur in a web search where a crawler has to explore the web graph following the links. Similarly, information about the users of a social network can usually be obtained via an API, and new users are discovered following the friends of known users. To model such a situation, we use the concept of Personalized PageRank (PPR) (Page et al. 1999). It represents the stationary distribution of a random walk that always restarts from some fixed node (see, e.g., (Klicpera et al. 2018) for more details). The associated distribution shift naturally combines popularity and locality: PPR is related to node importance since the stationary distribution concentrates more on higher-degree nodes. On the other hand, the locality also preserved since restarts always happen in a fixed node. For our splits, we select the node E V with the highest PR score restarting node and compute the PPR score for every node i E V. After that, we define the measure o, as negative PPR. The nodes with high PPR, which belong to the ID part, expected to be close to the restarting node, while far away nodes go to the OOD subset.
Figure I that locality is indeed preserved, the ID part consists of one compact region around the most important node chosen as the restarting one. Thus, the ID subset includes periphery nodes well as some nodes that were previously marked as the most important in the PR-based split but
4

---

Under review as a conference paper at ICLR 2023
now less important for the restarting node. The remaining nodes come to the OOD part. Our analysis in Appendix [ also provides strong evidence for the mentioned behavior: the PPR-based split strongly affects the distribution of pairwise distances within the ID/OOD parts the locality bias of the ID part makes the OOD nodes even more distant from each other. The shift of the degree distribution is also notable but not severe as for the PR-based split since here we consider only the popularity conditioned on some fixed node. Finally, our empirical results in Section confirm that the PPR-based split is the most challenging one for graph neural networks.
3.4 METRICS
To evaluate the classification performance, exploit standard Accuracy and compute these met- rics on Test-In and Test-Out parts. To analyze the aggregated performance, we also report Accuracy and AUROC the mixture of Test-In and Test-Out..
To evaluate the quality of uncertainty estimates, we consider two problems: error (misclassification) detection and OOD detection. To assess how well a model can detect misclassified samples,
the concept of Prediction Rejection Curve (PRC) (Malinin et al.] 2021: 2022). PRC traces the error rate replace model predictions with ground-truth labels in the order of decreasing uncertainty.
uncertainty is high for incorrectly classified samples, then the error rate is expected to drop quickly as we replace such predictions with ground true labels.
Thus, the Area Under Prediction Rejection Curve (AUPRC) evaluates the joint classification and un- certainty estimation performance, requiring the model not only to provide high prediction accuracy but also to signalize about possible errors through higher uncertainty scores. In our experiments, we compute AUPRCmadel on the merged test subset of nodes Vea using total uncertainty (TU), as the errors may occur due to the inherent noise in data or because of predicting OOD samples.
measure called Prediction Rejection Ratio (PRR) is also based on the Prediction Rejection Curve but evaluates only the ability of a model to detect misclassified samples. For this purpose, AUPRC normalized as follows. Let PRCmodel be the predicted uncertainty estimates, PRCrandom be random uncertainty estimates, and PRCoracle be estimates that perfectly sort samples according to the predic- tion errors (i.e., all misclassified samples have higher oracle uncertainty). Then, the PRR metric is defined as follows:
AUPRO C Gandom
-AUPRC modeel
PRR
AUPRC random - -AUPRC
The best value of this measure is 1 (for the perfect uncertainty estimates), while random uncertainty estimates give PRR O. Note that each model has its own oracle with the associated estimates that perfectly match its prediction errors. So the AUPRConele values of different models are independent and computed only based on the corresponding model predictions.
We also evaluate the ability of models to detect OOD samples. For this, we consider the mixture of Test In and -Out. A good model is expected to have higher knowledge uncertainty (KU) Gal 2016; Malinin 2019) values for the observations from Test-Out compared to Test-In Inn Here, we use the standard AUROC for the binary classification with positive events corresponding
to the observations coming from the OOD subset.
METHODS
We consider several methods for estimating uncertainty in graph-related problems. Specifically, we cover message-passingg neural networks, ensemble approaches Lakshminarayanan et al. (2017), and Dirichlet-basee methods that currently considered to be state-of-the-ae art for OOD detection that called GPN (Stadler et al., 2021). For Dirichlet-basee approaches, we conduct an ablation study to evaluate which design choices contribute most performance.
5

---

Under review as a conference paper at ICLR 2023
4.1 STANDARD METHODS
In this class of methods, the constructed model fe predicts the parameters μa, (x; I of the cat- egorical distribution Py(y (m.)) = P():();) the standard classification task, while the uncertainty estimates are obtained based on the entropy of this distribution.
simple baseline that serves us as lower bound for our further experiments with more advanced methods is MLP, which represents graph-agnostic MLP model and takes into account only the features of the current observation. Further, GNN is a simple GNN model based on a two-layer SAGE convolution (Hamilton et al.] (2017), which combines the information from both the central node and its neighborhood. As a training objective, these methods use the standard Cross-Entrop loss between the one- one-hot-enese hot-encoded target y and the predicted categorical vector μ4, For these meth- ods, we can only define uncertainty as the entropy u, H P(y: [4:] of the predictive categorical distribution.
4.2 DIRICHLET-BASED METHODS
The core idea behind the Dirichlet-based uncertainty estimation methods is to model the point- wise Dirichlet distribution pe(u; (z.) - | by predicting its parameters 3 = fo(x,) and updating the uniform prior distribution with parameters /3eior through their sum 3post 33eeat gh
Using this Dirichlet distribution, one can obtain the target categorical one as follows:
Po x.)) = E p(m: - ((()ll
implies that (M (e,)) has parameters BP /S,, where S; = Ex 8pe is called evidence or preci- sion. In other words, the parameters of the categorical distribution can be obtained from the Dirichlet ones by normalization. Importantly, the Dirichlet- methods allow us distinguish between to- tal and knowledge uncertainty as follows:
u total
H
[Po [.)]
H
[ ((P.111 B ( P(yilu)))
know u
S..
For this class of methods, the training objective is Expected Cross-Entropp with optional regular- isation term that is equal to the entropy of the predicted Dirichlet distribution p( (Sp
L. = E p(a. ) log P(3 [4:] - XI [p( )]
(1)
This loss function can be computed in closed form Malinin & Gales
2018
Charpentier et al. 2020).
As the most straightforward method in this class, consider modification of GNN that is referred to as EN (Evidential Network) (Sensoy et af.) 2018) while exploiting the same architecture, is trained predict the Dirichlet parameters via Loss
There are also more advanced methods based on the Dirichlet distribution which induce the behavior of the underlying model by estimating the density function in the latent space using Normalizing Flows (Kingma et al.) 20166] Huang et al. 2018). These methods can be united within the recently proposed framework Posterior Networks (Charpentier et al., 2020 2021). It was applied the node- level problems in (Stadler et al.) 2021). In this paper, we provide a detailed study of this framework and consider different variations depending on how the density estimation is performed and how the graph information is used. The description of these components can be found in Appendix E
4.3 ENSEMBLES
In our study, we also consider ensembling techniques that proved to increase the predictive per- formance of models and provide instruments for estimating uncertainty. Among the methods that predict the parameters μ4, of categorical distributions P (4). there is a widely used approach for uncertainty estimation introduced by Lakshminarayanan et al. (2017). It can be formulated as an
Foor GNNs, we formally have μt; = fo (A, X, hut we write fo(m;) for simplicity of notation and consis- tency with non-graph methods.

---

Under review as a conference paper at ICLR 2023
empirical distribution of model parameters q(0) (uain) that can be obtained after training several instances of the model with different random seeds for initialization:
(y mx)) Ba(0
Given this, we can split the total uncertainty into data and knowledge uncertainty through the fol- lowing expression (Malinin & Gales 2018):
tocal ui
H
PP (y [x:)] THE Be( P(Hi)ps)))
01 ( H [IP[yvl):)]]
lotal
U1
data
We apply this approach to GNN models and denote the obtained ensemble as EnsGNN. As for the Birichlet-based approaches, we follow CCarpennier et al. (202T) and define an ensemble of models that predict the parameters of posterior Dirichlet distribution as the mean the parameters in ensemble. Here, uncertainty is estimated in the same way as for a single Dirichlet-based model.
5 EXPERIMENTS AND ABLATION
Setup As discussed in Section 3.4 we provide the comparison of methods in 4 different problems using their associated metrics. In particular, report the standard Accuracy for general classifica- tion performance, PRR @ @TU for consistency of toial unceriainty estimates o with the prediction errors, AUPRC@TU TU for the joint classification and confidence performance, and AUROC @ KU for the OOD detection using knowledge uncertainty estimates u. setup can be found in Appendix G
The details of our experimental
Our code and benchmark are publicly available at httpt:t/////// /ttp////nnnnommsmmmmo/ 4open. scissciies ciceece/ #:revisiting-- wong-unneetaaie it ests immtiiiinnn ioni F4B6, together with a framework for evalu- ating variety of baseline models, Dirichlet-based methods, and ensembling techniques.
We run each method 5 times with different random seeds and report the mean and standard deviation. To compare a pair of methods and analyze whether one is noticeably better than the other, we report win/tie/ oss counts aggregated over the datasets and, some cases, over different distribution shifts. We say that method wins on particular dataset if it is better than its competitor and the difference is larger than the sum of their standard deviations. If the difference smaller, there is a tie. Note that we do not compute the standard deviation for the ensembles since they combine all 5 models. To compare different methods, we aggregate their results over all the datasets as follows. First, for each dataset, metric, and distribution shift, rank all the methods according to their performance (the smaller, the better). Then, we average the obtained ranks over all the datasets.
Results and insights In this section, we address several research questions. First, we analyze the complexity and diversity of the proposed data split strategies. Then, we demonstrate the performance of the considered methods on our benchmark. After that, we investigate the Dirichlet- based methods to figure out which of them is superior for each task. In conclusion, we find out whether the best- performing models can further be improved via ensembling.
Q1: How complex and diverse are the proposed data splits?
To answer this question, we analyze the predictive performance of the models on Test-In (ID) and Test-Out (OOD) parts. Tssoos this comparison for GNN, and similar results for GPN, EnsGNN, and MLP can be found in Tables 6 in Appendix. We can see that feature-based splits are the easiest for GNN: the difference between ID and OOD parts is sufficiently small, and the performance on the latter can be even better for some datasets. The PageRank splits are noticeably more complicated, but the most difficult split strategy is based on Personalized PageRank performance drop is dramatic on some datasets. This can be explained by the locality of the ID part (see Figure Tp: the clear separation of ID from OOD in terms of the graph structure makes graph- based learning significantly more complicated.
the
Q2: How do existing methods perform in setup, and what is the current status quo?
To answer this question, we compare the following models: GNN as the classic graph processing method, EnsGNN as the standard and universal approach allowing for getting uncertainty estimates,

---

Under review as a conference paper at ICLR 2023
Table Accuracy of GNN on ID vs OOD test subsets. Diff. % is the difference between the accuracy scores on the OOD and ID parts divided by the accuracy score on the ID part.
Feature
PageRank
PPR
ID
OOD
Diff, %
ID
OOD
Diff.
ID
OOD
Diff, %
AmazonComputerrs
84.88
84.86
0.02
87.86
82.36
6.27
86.40
58.77
33.97
AmazonPhoto
94.90
92.23
2.82
97.07
88.94
8.38
92.68
43.89
52.64
CoauthorCS
92.44
93.
+1.41
94.81
91.33
33.66
92.67
91.63
- -1.12
CoauthorPhysies
96.57
96.06
0.54
97.52
94.44
-3.15
97.72
93.68
-4.14
CoraML
83.68
86.96
+3.92
89.90
82.89
7.80
86.76
75.51
- 12.96
CiteSeer
70.87
71.50
+0.88
74.47
71.42
4.10
74.11
59.25
20.06
PubMed
87.58
86.01
-1.79
88.69
83.39
55.98
85.88
84.53
-1.57
Table 2: Average ranks of standard uncertainty estimation methods over graph datasets.
Accuracy
PRR@TU AUPRC@TU AUROC@KU
Random
GNN
2.3
1.7
2.3
n/a
GPN
2.6
3.0
2.6
n/a
EnsGNN
1.1
1.3
1.1
n/a
Feature
GNN
1.6
2.1
GPN
2.6
2.9
2.7
1.0
EnsGNN
1.1
1.6
1.1
2.7
PageRank
GNN
2.3
1.9
2.1
GPN
2.4
2.7
2.7
1.0
EnsGNN
1.3
1.4
1.1
2.3
PPR
GNN
2.4
2.3
2.6
GPN
2.1
1.7
2.0
1.0
EnsGNN
1.4
2.0
1.4
2.1
and GPN which is known to be state-of-the-art for the OOD detection (Stadler et al. 2021). Table 2 compares these methods over all datasets. One can see that, according AUROC @ KU, the best OOD detection performance is achieved by GPN for all types of shifts, which consistent with previously reported results (Stadler et al. 2021). Unsurprisingly. EnsGNN shows the best predictive performance, as measured by Accuracy. Moreover, has the most consistent uncertainty estimates in context of PRR @ TU and provides the best joint performance via AUPRC @ @UU. In summary, ensembles are the best for all tasks but OOD detection, where the superior method is GPN.
Q3: Which Dirichlet-based methods are the best for each prediction task?
The simplest method is EN. which represents standard GNN trained with the Loss (T. Further, consider the methods using Normalizing Flows, and compare Standard vs Natural density estimation and Graph Encoding vs Graph Propagation. Based on Table 3 one can make the following conclusions. First, GPN still the best method for OOD detection. Second, NatPN is the best according to all other tasks. Third, EN is strong baseline that shows competitive results Accuracy. PRR TU, and AUCPRC@TU often staying close to NatPN. To show whether the difference is statistically significant, we report win/ts. losss counts for some pairs of models. See Table 4 for aggregated results and Tables 9 11 in Appendix for more details.
Here, we provide a detailed analysis of the Dirichlet-based framework.
Q4: ensembles consistently improve the performance of complex models?
Ensemble are known to consistently improve the model performance in various tasks, so we aim to confirm that this result holds the transductive node classification. For this purpose, we compare the ensembles of the two most promising methods GPN and NatPN. The aggregated results are shown in Table (also, see Tables 10 in Appendix for more details). One can see that ensembling via EnsGPN consistently improves GPN for OOD detection, but the difference is mostly insignificant. In contrast, EnsNatPN is noticeably better than NatPN, and this gain is especially significant for Accuracy and the joint performance measured by AUPRC.
8

---

Under review as a conference paper at ICLR 2023
Table Average ranks of Dirichlet-based methods over datasets, including EN, PN, Na(PN, GPN and NatGPN, which are obtained for every considered split strategy and prediction task.
Accuracy PRR@TU AUPRC@TU AUROC@KU
Random
EN
2.0
1.7
2.0
Te/a
PN
3.7
3.7
3.4
n/a
NatPN
1.4
1.9
n/a
GPN
3.0
4.0
3.3
n/a
NatGPN
4.4
4.1
4.4
n/a
Feature
EN
2.3
1.6
2.0
3.3
3.4
3.6
3.9
NatPN
1.9
1.7
1.7
4.6
GPN
2.9
4.4
3.3
1.4
NatGPN
4.4
3.9
4.4
1.9
PageRank EN
2.1
2.0
2.1
4.4
PN
3.9
3.4
3.6
3.1
NatPN
1.7
1.9
1.7
4.4
GPN
2.7
3.7
3.3
1.1
NatGPN
4.0
4.3
1.9
PPR
EN
3.0
3.0
2.9
3.3
PN
3.7
3.9
3.9
3.4
NatPN
1.7
1.9
5.0
GPN
2.1
2.4
2.1
1.0
NatGPN
4.1
4.0
4.3
2.3
Table Win/tie// /loss counts for some pairs of Dirichlet- methods across all the consid- ered graph datasets and split strategies (except Random).
Accuracy
PRR @TU AUPRC@TU AUROC@KU
NatPN vs
3/18/0
6/14/1
8/12/1
3/3/15
GPN vs NatGPN
14/5/2
8/11/2
15/2/4
12/8/1
NatPN vs GPN
12/2/7
14/5/2
13/3/5
0/0/21
Table 5: Win/tie/ loss counts for ensembles vs the corresponding single models across all the considered graph datasets and split strategies (except Random).
Accuracy
PRR@TU AUPRC@TU AUROC@KU
EnsGPN vs GPN
9/11/1 14/7/0
6/14/1
14/7/0
4/17/0
EnsNaIPN vs NatPN
6/15/0
15/6/0
5/9/7
To summarize our findings, we refer to Table 15 for the comparison of all the methods in terms of ranks and to Tables 11-14 for the detailed comparison of their win/t elloos counts. One can conclude that GPN the best single-pass method for OOD detection, while NatPN is the best one for all other tasks. Morcover, the performance of the latter can be further improved by ensembling, so EnsNatPN achieves the best results in terms of Accuracy, PRR, and AUPRC.
CONCLUSION
In this work, propose new benchmark for evaluating robustness and uncertainty estimation in transductive node classification tasks. For this, we design a universal approach to creating data splits with distribution shifts: it be applied to any graph dataset and allows for generating shifts of various nature. Using our benchmark, we show that the recently proposed Graph Posterior Network (Stadler et al. 2021) consistently the best method for detecting the OOD inputs, while the best results for the other tasks are achieved using Natural Posterior Networks (Charpentier et al.) 2021). Our experiments also confirm that ensembling allows one improve the model performance. Thus, we believe that our benchmark will be useful for future studies of node-level uncertainty estimation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Posterior network: timation without ood samples via density-based pseudo-counts. Advances in Neural Information Processing Systems, 33:1356-1367, 2020.
Uncertainty es-
Bertrand Charpentier, Oliver Borchert, Daniel Zügner, Simon Geisler, and Stephan Günnemann. Natural posterior network: Deep bayesian predictive uncertainty for exponential family distribu- tions. In International Conference on Learning Representations, 2021.
Yarin Gal. Uncertainty in deep learning. University of Cambridge, Cambridge, 2016.
Lise Getoor. Link-based classification. In Advanced methods for knowledge discovery from complex data, pp. 189-207. Springer, 2005.
C Lee Giles, Kurt D Bollacker, and Steve Lawrence. Citeseer: An automatic citation indexing
system. In Proceedings of the third ACM conference on Digital libraries, pp. 89-98, 1998.
Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. GOOD: A graph out-of-distribun bench-
mark. arXiv preprint arXiv:2206. 08452, 2022.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs Advances in Neural Information Processing Systems, 30, 2017.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078-2087. PMLR, 2018
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X2:6142..698. 2014.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997, 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
Andrey Malinin. Uncertainty estimation in deep learning with application to spoken language assessment. PhD thesis, University of Cambridge, 2019.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark JF Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, et al. Shifts: A dataset of real distributional shift across multiple large-scale tasks. arXiv preprint arXiv:2107.07455 2021.
Andrey Malinin, Andreas Athanasopoulos, Muhamed Barakovic, Meritxell Bach Cuadra, Mark JF Gales, Cristina Granziera, Mara Graziani, Nikolay Kartashev, Konstantinos Kyriakopoulos, Po- Jui Lu, et al. Shifts 2.0: Extending the dataset of real distributional shifts. arXiv preprint arXiv:2206.15. 25l.:22206.14477. 2022.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based rec- ommendations styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 3(2):127-163, 2000.
10

---

Under review as a conference paper at ICLR 2023
Galileo Namata, Ben London, Lise Getoor, Bert Huang, and U Edu. Query- -riren active surveying for collective classification. In 10th Interaational Workshop on Mining and Learning with Graphs, volume 8. pp. 1, 2012.
Lawrence Page. Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. Al magazine, 29(3):93- 93, 2008
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifica- tion uncertainty. Advances in neural information processing systems, 31, 2018.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868, 2018.
Maximilian Stadler, Bertrand Charpentier, Simon Geisler, Daniel Zügner, and Stephan Günnemann. Graph posterior network: Bayesian predictive uncertainty node classification. Advances in Neural Information Processing Systems, 34: 18033- 18048, 2021.
Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware supi-super learning on graph data. Advances in Neural Information Processing Systems,
2020.

---

Under review as a conference paper at ICLR 2023
A VISUALIZATION OF DISTRIBUTION SHIFTS
In Figures 28 provide the visualization of different split strategies for all the datasets. Some
graphs have multiple connected components - that case, we keep only the largest one.
(a) Feature
(c) Personalized PageRank
(b) PageRank
Figure 2: Visualization of data splits for CoraML dataset: ID is blue, OOD red.
(a) Feature
(b) PageRank
(c) Personalized PageRank
Figure 3: Visualization of data splits for CiteSeer dataset: ID is blue, OOD is red.
Feature
(b) PageRank
Personalized PageRank
Figure 4: Visualization of data splits for PubMed dataset: ID is blue, OOD is red.
12

---

Under review as a conference paper at ICLR 2023
(a) Feature
(h) PageRank
(c) Personalized PageRank
Figure 5: Visualization of data splits for AmazonComputer dataset: ID is blue, OOD is red.
(a) Feature
(b) PageRank
(c) Personalized PageRank
Figure 6: Visualization data splits for AmazonPhoto dataset: ID is bluc, OOD is red.
(a) Feature
(b) PageRank
(c) Personalized PageRank
Figure 7: Visualization of data splits for CoauthorCS dataset: ID is blue, OOD is red.
(a) Feature
(b) PageRank
(c) Personalized PageRank
Figure 8: Visualization of data splits for CoauthorPhysics dataset: ID is blue, OOD is red.
13

---

Under review as a conference paper at ICLR 2023
B PROPERTIES OF DISTRIBUTION SHIFTS
This section provides a detailed analysis and comparison of the proposed distribution shifts. For this purpose, consider three representative real-world datasets AmazonComputers, CoauthorCS, and CoraML, and discuss how different distribution shifts affect the basic properties of data: class balance, degree distribution, and graph distances between nodes within ID and OOD subsets.
0.35
6925
815
825
(a) Feature
PageRank
(c) Personalized PageRank
Figure 9; Visualization of class balance for AmazonComputers dataset.
0:31
0CD
0.20
000
0.10
6.5
900
Feature
(b) PageRank
(c) Personalized PageRank
Figure 10: Visualization of class balance for CoauthorCS dataset.
0.30
19:10
1443
Iebo
0.30
9.00
8.35
645
(a) Feature
(b) PageRank
(c) Personalized PageRank
Figure 11: Visualization of class balance for CoraML dataset.
Class balance Class balance directly affects the amount of evidence acquired by the graph pro- cessing model and used for estimating uncertainty and making predictions. It is especially important for evaluating Dirichlet-based models which exploit Normalising Flows, as their density estimates can become irrelevant due a significant change in class balance.
one can see that neither feature-based nor PageRank- based split makes a notable difference the class balance between the ID and OOD subsets. At the same time, the PPR-based split leads to significant changes for some classes. This shows that the split strategies based on the structural locality in graph can very challenging as they also affect such crucial statistics as class balance. Interestingly, the PageRank-based split does not lead to significant shifts of class balance (for the datasets under consideration), i.c., the more important and less important nodes have, on average, the same probability of belonging particular class.
In Figures
Degree distribution The node degree distribution is one the basic structural characteristics of a graph that describes the local importance of nodes. Degrees are especially important for such graph
14

---

Under review as a conference paper at ICLR 2023
Aewoeetrepeeee opin
desseaaorpptee Regelask
opée
(a) Feature
(h) PageRank
(c) Personalized PageRank
Figure 12: Visualization of degree distribution for AmazonComputerr dataset.
Cataure Owwttertt opit: Paysioak
deciee Dowchert' sp P98
100
337
107
Ougree
Degres
(a) Feature
(b) PageRank
(c) Personalized PageRank
Figure 13: Visualization of degree distribution for CoauthorCS dataset.
PegeRock
(a) Feature
(b) PageRank
(c) Personalized PageRank
Figure 14: Visualization of degree distribution for CI dataset.
processing methods as GNNs, since they describe how many channels around the considered node are used for message passing and aggregation.
In Figures 12 14 one can see that the most significant change in the degree distribution appears when the ID and OOD subsets are separated based on PageRank: the ID part contains more high- degree nodes. This is expected since PageRank is graph characteristic measuring node importance (a. centrality), and node degree is the simplest centrality measure known to be correlated with PageRank. For PPR-based splits, the difference in degree distribution is smaller but still significant since PPR selects nodes by their relative importance for a particular node, so some high-degree nodes can be less important in terms of PPR. Finally, for features- based splits, the degree distribution also changes, but the shift level is much less significant.
Distribution of pairwise distances The distance between two nodes in graph is defined as the length of the shortest path between them. Here, we compute such distances between the nodes in the ID or OOD subset within the original graph, i.c., we consider the whole graph when searching for the shortest path. The distribution of distances shows how easily messages can be passed between the nodes. Therefore, we expect that larger pairwise distances create more complicated tasks.
In Figures 15
one can observe that the based split leads to the most significant changes in distances, making the OOD nodes nearly twice as far from each other the ID ones. At the same time, the PageRank- based split does not lead to such a difference, revealing almost the same distri- butions on ID and OOD subsets. This means that the popularity bias in graph does not prevent one
15

---

Under review as a conference paper at ICLR 2023
deesoeDrepprrg Regelask
100
10
330
10
10
19
10l
2
10
(a) Feature
(h) PageRank
(c) Personalized PageRank
Figure 15: Visualization of pairwise distance distribution for AmazonCompulee dataset.
CoustbertS apdi: Deattre
Catami: CountbartS sptt: Depelazk
danaadd: ConsibeeSS sput.
TD
107
ID
10
10
161
339
(a) Feature
(b) PageRank
(c) Personalized PageRank
Figure 16: Visualization of pairwise distance distribution for CoauthorCS dataset.
dasaeen Cooldi can Rottore
dasaer Caraiii uoi PapeBock
ditase Docattl can
33"
110
107
107
107
10P
199
100
(a) Feature
(b) PageRank
(c) Personalized PageRank
Figure 17: Visualization of pairwise distance distribution for CoraML dataset.
from covering the less popular periphery nodes since the most popular nodes may be widespread Finally, the feature- split preserves the distances within the subsets.
C COMPARISON WITH GOOD BENCHMARK FROM GUI ET AL. (2022)
Our work complements and extends the GOOD benchmark recently proposed by Gui et al. (0022) However, there are several important differences that we discuss in this section.
One of the main properties of the GOOD benchmark is its theoretical distinction between two types of distribution shifts, which are represented through a graphical model. In particular, the authors consider covariate shifts, in which the distribution of features changes while the conditional distri- bution of targets given features remains the same, and concept shifts, where the opposite situation occurs, i.e., the conditional target distribution changes, while the feature distribution is the same. Although this distinction might be very helpful for understanding the properties of particular GNN models, such exclusively covariate or concept shifts rarely happen in practice where both types of shifts are present at the same time.
To create pure covariate or concept shifts, Gui et al. (2022) introduce different subsets of variables that either fully determine the target, create confounding associations with the target, are com- pletely independent of the target. This has to be properly handled and makes it non-trivial to create distribution shifts new datasets with this approach. Indeed, the distribution shifts in the GOOD benchmark can be properly implemented only for synthetic graph datasets or via appending syn- thetic features that either describe various domains completely independent variables or create
16

---

Under review as a conference paper at ICLR 2023
Lerpage,
(a) Concept
(h) Covariate
Figure 18: Visualization of class balance for GOOD-Twitch dataset.
ditauri domala, Occagt
Caca, denaks Conoritt
000
OCD
CLA
(a) Concept
(b) Covariate
Figure 19: Visualization of class balance for GOOD-Cora dataset.
Abeasun WHOLE Cemare Uoivenityy <kdb Occagee
S.tavat WiLER dicuta Jiseents datb Dovanate
0.8
0.0
Claa
(a) Concept
(b) Covariate
Figure 20: Visualization of class balance for GOOD-WebKB dataset.
the necessary concepts by inducing some spurious correlation with the target. Moreover, the au- thors claim that, in the case of real-world datasets, one has to perform screening over the available node features to create the required setup of domain or concept shift. This fact implies numerous restrictions on how the data splits be prepared.
In contrast, our method does not make distinction between covariate and concept shifts and thus can be universally applied any dataset and does not require any dataset modifications. Importantly, the type of distribution shift and the sizes of all split parts are easily controllable. This fiexibility is the main advantage of our approach.
Finally, Gui et al. 2022) confirm the importance of using both node features and graph structure. Still, their nove-level distribution shifts are usually based on node features such the number of words the year of publication in citation network, the language of users in social network, or the name of organizations used in some citation networks. In contrast, ate significant distribution shifts using a very simple technique that requires computing some node property in the graph that should be chosen depending use PageRank to create distribution shifts by the structural popularity of instances or Personalized PageRank take into account the locality and distinguish between the core and periphery nodes.
webpage network. As the graph properties, only node degrees are
propose to use the graph structure directly and cre-
specific problem. For instance, may
Further in this section, we compare our benchmark with GOOD in terms distribution shift statis- ties discussed in Appendix B
17

---

Under review as a conference paper at ICLR 2023
Langugee
ego
33'
(a) Concept
(h) Covariate
Figure 21: Visualization of degree distribution for GOOD-Twitch dataset.
dataurr Caca, dunaser Degire, Docrpe
Daci, domuse Deginn, de Coveriite
18
10
10P
(a) Concept
(b) Covariate
Figure 22: Visualization of degree distribution for GOOD-Cora dataset.
ditavet: WesTTD comats Uniivesity stitt Ceccept
dessatt MeakC. Sennee: Usiveityy sith Conuritt
(a) Concept
(b) Covariate
Figure 23: Visualization of degree distribution for GOOD-WebKB dataset.
Class balance In our benchmark. the PPR-based shift significantly changes the class balance be- tween ID and OOD subsets, while, for feature- and PageRank- -based shifts, the class balance does not change significantly. In GOOD, the level of distribution shift depends the dataset and the type of shift. For instance, Figures 19 and 20 show a significant change for both types of shifts.
Degree distribution Comparing to our distribution shifts, the GOOD data splits have much less impact on the degree distribution: this graph property changes dramatically only when the covariate shift is constructed using the degree domain, as in the case of GOOD-Cora dataset (see Figure 22
Graph distance distribution In contrast to our benchmark, the GOOD approach does not lead to a significant change in pairwise distance distribution between ID and OOD parts: the distance distribution hardly changes for concept shifts, and only covariate shifts make the difference between ID and OOD subsets somehow notable. This proves the necessity of considering the graph structure for inducing challenging distribution shifts.
18

---

Under review as a conference paper at ICLR 2023
(a) Concept
(b) Covariate
Figure 24: Visualization of pairwise distance distribution for COOD-Twitch dataset.
Austa Casa, coosai Bugrees cluli Cueapt
ditanc Outa, Jaxase Dagine, dalb Gocrats
(a) Concept
(b) Covariate
Figure 25: Visualization of pairwise distance distribution for GOOD-Cora dataset.
cxtort Weld domate: Delventty stitt Chenipt
diteuet: Ne55ZL dreats Detwerits dbtt Coverbite
10
(a) Concept
(b) Covariate
Figure 26: Visualization of pairwise distance distribution for GOOD-WebKB dataset.
D ADDITIONAL RESULTS
In this section, we provide additional experimental results.
Tables 6-8 are similar to Table in the main text. They show how difficult our splits are for different models. These results are consistent: the feature-based split is the easiest for all the methods, while the PPR-based one is the hardest. Interestingly, it holds for the graph- agnostic MLP, which means that this graph-based shift also implies noticeable shift in the feature space;
Table is is a detalization of Table where we consider different distribution shifts sepa- rately and add random partition. Similarly, Table I0 detalizes Table
Tables
14 aggregates win/tie/ loss counts for all pairs of methods;
Table 15 compares all the methods in terms of their ranks averaged over the datasets;
provide the results for all the methods on all the datasets. Note that all other aggregated results can be deduced from these tables.
Tables 16 22
19

---

Under review as a conference paper at ICLR 2023
Table 6: Accuracy of MLP on ID vs OOD test subsets for every split strategy.
Feature OOD
PageRank OOD
PPR
Dift,
ID
Diff.
OOD
Diff.
AmazonComputers
75.80
77.69
+2.50
81.09
75.49
6.91
81.18
50.23
38, 13
AmazonPhoto
89.44
84.42
-5.61
92.84
83.50
10006
89.67
39.69
-55.44
CoauthorCS
92.02
92.52
+0.54
93.97
92.95
-1.9
92.39
89.98
2.61
CoasthorPhysies
95.36
95.58
+0.23
96.72
93.98
2284
96.55
87.25
-9.63
CoraML 75.18
70.78
5..85
75.18
68.85
-8.88
72.04
52.50
-27.12
CiteSeer
65.35
64.54
1.24
68.35
65.36
-4.77
68.17
52.37
-23.18
PubMed
85.29
84.50
0.93
86.15
85.10
-1.21
84.43
84.04
-0.46
Table Accuracy of GPN on ID OOD test subsets for every split strategy.
Feature OOD
PageRank OOD
PPR
ID
Diff,
ID
Diff.
ID
OOD
Diff, %
AmazonComputers
88.05
89.19
+ +1.30
89.83
83.58
6.95
88.63
65.84
25.77
AmazonPhoto
92.89
91.32
1.69
94.77
87.48
7.69
89.78
65.25
27.32
CoauthorCS
89.02
91.06
+2.29
92.14
87.99
4.51
89.80
87.93
-2.8
CoautsorPhysics
95.45
95.03
0.44
96.31
92.68
3.77
95.93
88.90
-7.32
CoraML
84.01
84.72
+0.85
86.96
83.11
-4.43
84.48
75.81
10.22
CiteSeer
67.63
68.10
+0.70
72.25
70.49
2.44
74.95
51.60
- 331.16
PubMed
88.95
86.38
2.89
88.45
85.46
-3.37
86.12
86.03
-0.10
Table 8: Accuracy of EnsGNN ID OOD test subsets for every split strategy.
Feature OOD
PageRank OOD
PPR
ID
Diff, %
ID
Diff. %
ID
OOD
Diff, %
AmazonComputers
86.63
85.91
-0.83
88.08
83.53
-5.17
86.63
59.97
30.77
AmazonPhoto
95.16
92.68
2.61
97.65
89.77
-8.07
92.68
45.03
51.41
CoauthorC'S
92.58
94.00
+1553
94.65
91.40
33.4
92.80
91.86
-1.01
CoauthorPhysies
96.52
96.07
0..17
97.56
91.49
3.15
97.74
93.79
-1.04
CoraML
84.62
87.48
+3.38
90.30
83.31
7.75
87.63
77.05
- 12.07
CiteSeer
72.07
72.50
+0.60
74.77
71.90
3.84
74.77
61.91
17.21
PubMed
88.64
86.78
2.10
89.20
84.06
-5.76
86.66
85.61
-1.22
Table 9: Win/tie/lose counts for some pairs of Dirichlet-brag based methods across all the consid- ered graph datasets.
Accuracy
PRR@T TU AUPRC@TU AUROCOKU
NaPPN vs EN
Random
0/7/0
1/6/0
2/4/1
n/a
Feature
1/6/0
2/5/0
2/4/1
1/1/5
PageRank
1/6/0
1/6/0
3/4/0
2/1/4
PPR
1/6/0
3/3/1
3/4/0
0/1/6
GPN vs NatGPN Random
4/2/1
1/5/1
4/2/1
n/a
Feature
5/1/1
1/6/0
5/0/2
2/5/0
PageRank
5/2/0
3/3/1
5/1/1
5/1/1
PPR
4/2/1
4/2/1
5/1/1
5/2/0
NatPN vs GPN
Random
5/0/2
6/1/0
5/1/1
n/a
Feature
4/1/2
7/0/0
5/1/1
0/0/7
PageRank
5/0/2
5/1/1
5/1/1
0/0/7
PPR
3/1/3
2/4/1
3/1/3
0/0/7
20

---

Under review as a conference paper at ICLR 2023
Table 10: Win/tie/ loss counts for ensembles the corresponding single models across all the considered graph datasets.
Accuracy
PRR @ TU
AUPRC@TU AUROC@KL KU
EnsGPN GPN
Random
6/1/0
1/4/2
6/1/0
n/a
Feature
2/4/1
3/4/0
6/1/0
0/7/0
PageRank
4/3/0
1/5/1
4/3/0
4/3/0
PPR
3/4/0
2/5/0
4/3/0
0/7/0
EnsNatPN vs NatPN Random
3/4/0
2/5/0
4/3/0
n/a
Feature
3/4/0
2/5/0
5/2/0
1/3/3
PageRank
5/2/0
3/4/0
6/1/0
3/2/2
PPR
6/1/0
1/6/0
4/3/0
1/4/2
Table 11: Pairwise wwn/tii//iii tiss counts for Accuracy across all graph datasets and split strate- gies (except Random).
MLP
GNN
EN
PN
NaPP
GPN
NatGPN
EnsGNN
EnsNarPN
EnsGPN
MLP
0/21/0
2/0/19
2/1/18
10/2/9
2/1/18
5/1/15
6/6/9
2/0/19
2/0/19
5/0/16
GNN
19/0/2
0/21/0
1/20/0
17/1/3
0/18/3
12/2/7
16/5/0
0/6/15
1/5/15
13/0/8
EN
18/1/2
0/20/1
0/21/0
16/1/4
0/18/3
12/3/6
14/7/0
0/7/14
0/6/15
12/1/8
PN
9/2/10 3/1/17
4/1/16
0/21/0
3/2/16
0/8/13
10/5/6
3/1/17
3/1/17
1/3/17
NarPN
18/1/2
3/18/0
3/18/0
16/2/3
0/21/0
12/2/7
17/4/0
3/6/12
0/7/14
12/2/7
GPN
15/1/5 7/2/12
6/3/12
13/8/0
7/2/12
0/21/0
14/5/2
6/1/14
6/1/14
1/11/9
NatGPN
9/6/6
0/5/16
0/7/14
6/5/10
0/4/17
2/5/14
0/21/0
0/1/20
0/1/20
4/2/15
EnsGNN
19/0/2 15/6/0
14/7/0
17/1/3
12/6/3
14/1/6
20/1/0
0/21/0
9/0/12
14/0/7
EnsNaPP
19/0/2 15/5/1
15/6/0
17/1/3
14/7/0
14/1/6
20/1/0
9/0/12
0/21/0
14/0/7
EnsGPN
16/0/5 8/0/13 8/1/12
17/3/1
7/2/12
9/11/1 15/2/4
7/0/14
7/0/14
0/21/0
Table 12: Pairwise wwn/ counts for PRR @TU across all graph datasets and split strate- gies (except Random).
MLP
PN
NaPPN
GPN
NatGPN
EnsGNN
EnsNatPN
EnsGPN
MLP
0/21/0
0/2/19
0/2/19
2/8/11
1/2/18
3/5/13
3/9/9
1/0/20
1/0/20
6/0/15
GNN
19/2/0
0/21/0
0/21/0
7/11/3
0/15/6
10/8/3
13/7/1
1/17/3
0/12/9
14/1/6
EN
19/2/0
0/21/0
0/21/0
8/10/3
1/14/6
10/7/4
14/6/1
7/8/6
3/8/10
12/4/5
PN
11/8/2
3/11/7
3/10/8
0/21/0
0/11/10
8/6/7
7/10/4
3/8/10
0/9/12
8/5/8
18/2/1
6/15/0
6/14/1
10/11/0
0/21/0
14/5/2
14/6/1
8/7/6
0/15/6
15/3/3
GPN
13/5/3
3/8/10
4/7/10
7/6/8
2/5/14
0/21/0
8/11/2
5/3/13
2/4/15
1/14/6
NatGPN
9/9/3
1/7/13
1/6/14
4/10/7
1/6/14
2/11/8
0/21/0
1/4/16
1/4/16
5/5/11
EnsGNN
20/0/1
3/17/1
6/8/7
10/8/3
6/7/8
13/3/5
16/4/1
0/21/0
9/0/12
14/0/7
EnsNarPN
20/0/1
9/12/0
10/8/3
12/9/0
6/15/0
15/4/2
16/4/1
12/0/9
0/21/0
16/0/5
EnsGPN
15/0/6
6/1/14 5/4/12
8/5/8
3/3/15
6/14/1
11/5/5
7/0/14
5/0/16
0/21/0
Table 13: Pairwise win/tie/losa counts for AUPRC@TU across all graph datasets and split strategies (except Random).
MLP
GNN
EN
PN
NaPN
GPN
NatGPN
EnsGNN
EnsNaPP
EnsGPN
MLP
0/21/0
0/1/20
0/2/19
7/4/10
1/0/20
3/2/16
6/6/9
0/0/21
0/0/21
3/0/18
GNN
20/1/0
0/21/0
1/20/0
17/3/1
0/15/6
13/3/5
16/5/0
0/6/15
0/4/17
13/1/7
EN
19/2/0
0/20/1
0/21/0
16/1/4
1/12/8
13/3/5
18/3/0
0/4/17
0/2/19
13/0/8
PN
10/4/7
1/3/17
4/1/16
0/21/0
2/2/17
3/6/12
10/5/6
1/3/17
1/3/17
2/6/13
NatPN
20/0/1 6/15/0
8/12/1
17/2/2
0/21/0
13/3/5
17/4/0
6/2/13
0/6/15
13/2/6
GPN
16/2/3 5/3/13
5/3/13
12/6/3
5/3/13
0/21/0
15/2/4
4/2/15
5/0/16
0/7/14
NaGPN
9/6/6 0/5/16
0/3/18
6/5/10
0/4/17
4/2/15
0/21/0
0/2/19
0/1/20
5/0/16
EusGNN
21/0/0
15/6/0
17/4/0
17/3/1
13/2/6
15/2/4
19/2/0
0/21/0
12/0/9
15/0/6
EusNaIPN
21/0/0
17/4/0
19/2/0
17/3/1
15/6/0
16/0/5
20/1/0
9/0/12
0/21/0
16/0/5
EnsGPN
18/0/3 7/1/13
8/0/13
13/6/2
6/2/13
14/7/0
16/0/5
6/0/15
5/0/16
0/21/0
21

---

Under review as a conference paper at ICLR 2023
Table 14: Pairwise win/tie/loss counts for AUROC@KU across all graph datasets and split strategies (except Random).
MLP
GNN
EN
PN
NarPN
GPN
NatGPN
EnsGNN
EnsNatPN
EnsGPN
MLP
0/21/0
10/3/8
10/3/8
8/3/10
15/1/5
3/1/17
4/0/17
10/1/10
14/1/6
3/0/18
GNN
8/3/10
0/21/0
1/18/2
6/6/9
15/3/3
0/3/18
1/3/17
5/4/12
15/0/6
0/1/20
EN
8/3/10
2/18/1
0/21/0
5/7/9
13/3/3
0/3/18
1/6/14
7/4/10
15/1/5
0/1/20
PN
10/3/8
9/6/6
9/7/5
0/21/00
17/3/1
0/1/20
1/3/17
10/3/8
17/1/3
0/0/21
NatPN
6/1/15
3/3/15
3/3/15
1/3/17
0/21/0
0/0/21
0/3/18
4/1/16
7/9/5
0/0/21
GPN
17/1/3
18/3/0
18/3/0
20/1/0
21/0/0
0/21/0
12/8/1
20/1/0
21/0/0
0/17/4
NatGPN
17/0/4
17/3/1
14/6/1
17/3/1
18/3/0
1/8/12
0/21/0
17/2/2
19/1/1
3/6/12
EnsGNN
10/1/10
12/4/5
10/4/7
8/3/10
16/1/4
0/1/20
2/2/17
0/21/0
15/0/6
0/0/21
EnsNarPN
6/1/14 6/0/15
5/1/15
3/1/17
5/9/7
0/0/21
1/1/19
6/0/15
0/21/0
0/0/21
EnsGPN
18/0/3
20/1/0
20/1/0
21/0/0 21/0/0
4/17/0 12/6/3
21/0/0
21/0/0
0/21/0
Table 15: Average ranks of the considered methods across all datasets.
Accuracy
PRR@ TU
AUPRC@TU AUROC@ KU
Random
MLP
8.4
8.7
8.9
n/a
GNN
4.4
4.0
4.6
n/a
EN
4.6
3.6
4.9
n/aa
PN
6.7
6.9
n/a
NatPN
4.6
3.0
4.4
n/a
GPN
6.6
7.7
6.9
n/a
NatGPN
8.7
8.1
8.7
1/a
EnsGNN
3.1
2.3
n/a
EnsNatPN
2.3
1.9
n/a
EnsGPN
5.6
7.7
5.7
n/a
Feature
MLP
8.7
8.3
9.1
4.9
GNN
4.3
4.4
5.7
EN
5.4
2.6
4.7
6.3
PN
7.4
6.4
7.0
7.1
NatPN
4.7
3.3
4.4
8.0
GPN
8.4
6.9
2.4
NatGPN
7.3
8.4
3.4
EnsGNN
1.7
4.3
2.0
6.9
EnsNaIPN
3.0
2.7
2.1
8.1
EnsGPN
5.3
7.4
5.9
2.1
PageRank MLP
6.9
9.0
7.9
6.4
GNN
5.0
4.3
4.7
7.9
EN
5.3
3.3
5.0
7.9
PN
8.0
6.6
7.3
4.7
NatPN
4.3
3.4
4.3
7.7
GPN
6.0
7.0
7.0
1.9
NatGPN
9.0
7.6
8.6
2.7
EnsGNN
2.9
3.1
2.1
6.9
EnsNaIPN
3.6
2.0
EnsGPN
5.1
7.1
6.1
1.4
PPR
MLP
8.6
9.0
9.0
6.9
GNN
5.3
5.6
EN
5.6
6.1
5.6
PN
7.4
7.3
7.6
6.4
NatPN
4.9
3.7
4.3
9.0
GPN
5.0
5.0
4.9
1.6
NatGPN
7.6
8.4
EnsGNN
3.0
4.9
3.1
5.0
EnsNatPN
2.4
3.1
2.6
9.3
EnsGPN
3.7
3.6
3.4
1.4
22

---

Under review as a conference paper at ICLR 2023
Table 16: Experiment results on AmazonComputers dataset.
Accuracy
PRR@TU
AUPRC@TU
AUROC@KU
Random
MLP
78.44 + 0.57
58.44 + 1.88
88.32 + 0.27
n/a
GNN
86.33 0.56
63.26 = 1.49
93.80 0.27
n/a
EN
83.73 1.20
67.42 = 3.81
92.93 = 0.28
n/a
88.18 0.23
65.19 + 2.06
94.98 + 0.22
n/a
NatPN
85.75 1.45
65.22 + 3.38
93.73 + 0.45
n/a
GPN
87.70 + 0.14
59.35 + 0.25
94.10 0.08
n/a
NatGPN
82.83 + 1.31
56.45 + 2.69
90.85 0.88
n/a
EnsGNN
86.67 + n/a
64.22 = n/a
94.09 n/a
n/a
EnsNatPN
86.67 n//
64.52 = n/a
94.12
n/a
EnsGPN
87.99 1.99::1 1
58.91 = n/aa
94.22 = n/a
n/aa
Feature
MLP
77.31 + 0.41
63.17 1.23
88.39 0.17
44.38 + 0.26
GNN
+4.87
68.37 = 3.64
93.66 = 0.23
47.45 = 0.52
EN
82.83 - 0.76
72.01 = 2.81
93.07 0.33
47.38 = 0.40
PN
88.39 +: 0.50
64.29 1: 2.33
94.98 0.44
51.53 1 0.33
NaPP
85.34 1.26
65.52 + 4.06
93.56 0.20
51.19 + 0.96
GPN
88.97 0.21
56.81 + 1.16
91.54
53.33 + 0.31
NatGPN
81.87 1.34
63.70 = 6.12
91.35 0.58
53.46 - 0.57
EnsGNN
86.05 + a
67.15 = n/a
94.11=n/a
48.36 = 11
EnsNatPN
86.29 + n
64.76 = n/aa
93.95 = n/a
51.92 = n/a
EnsGPN
89.34 n/a
56.23 n/a
94.70 n/a
53.46 n//a
PageRank
MLP
76.61 + 0.42
55.61 + 1.97
86.58 0.27
59.34 + 0.34
GNN
83.46 + 0.62
66.38 = 0.91
92.62
56.53 = 0.68
EN
81.72 1.14
68.72 = 2.71
91.99
56.32 = 1.33
84.70 0.69
65.96 1 0.97
93.25 0.37
65.04 1:
NatPN
83.03 + 1.28
66.84 + 2.34
0.41
48.01 + 2.04
GPN
84.83 - 0.22
59.26 + 1.69
92.46
90.58 + 0.49
NatGPN
1.
57.22 = 3.61
88.10
87.31 = 2.33
EnsGNN
84.44
65.98 = n/a
9
61.00 = n/a
EnsNatPN
8//4
65.73 = n/a
93.08 = n/a
44.68 = n/a
EnsGPN
85.05 + n/a
58.39 + n/a
92.48 + n/a
91.59 + n//a
PPR
MLP
56.42
54.18 + 3.29
69.68 + 3.26
79.23 + 0.76
GNN
64.30 0.93
70.18 + 3.24
40.41
79.69 - 1.07
EN
63.44 0.84
66.23 + 3.19
78.79 1.23
81.23 = 0.83
PN
68.11
61.39 1 4.21
81.44 0.86
69.79 1 2.28
NaPPN
64.10 1.26
69.25 + 2.63
80.03 1.21
53.22 1 6.98
GPN
70.40 0.31
+ 73.80
85.78 0.30
93.20 +
NatGPN
58,25 - 4.36
51.29 + 8.28
70.60 5.99
85.60 - 3.98
EnsGNN
65.30 + n/a
71.37 = n/a
81.48 n/a
79.78 = n/a
EnsNatPN
65.30 + n
69.48 = n/a
$1.05
51.25 = n/a
EnsGPN
70.77 /
74.65 n/a
86.21 n//
93.38 n/a
23

---

Under review as a conference paper at ICLR 2023
Table 17: Experiment results on AmazonPhoto dataset.
Accuracy
PRR@TU
AUPRC@TU TU
AUROC@KU
Random
MLP
89.49 + 0.25
73.44 + 0.65
96.40 + 0.06
n/a
GNN
94.34 0.24
83.81 = 1.38
98.82 = 0.10
n/a
EN
94.25 0.19
83.18 1.53
98.76 0.10
n/a
93.01 0.27
81.37 + 1.28
98.30 0.08
n/a
NatPN
0.15
83.96 + 1.53
98.75 + 0.05
1/aa
GPN
92.67 + 0.13
67.44
97.25 + 0.08
n/aa
NatGPN
89.09 4.56
73.52 3.20
96.10 2.29
n/a
EnsGNN
94.61 n/a
85.78 = n/a
98.99 n/a
n/a
EnsNatPN
94
84.87 n/a
98.91 = n/a
n//a
EnsGPN
92.92 n/a
67.50 = n/a
97.36 = n/a
n/a
Feature
MLP
65.43
72.02 1.41
94.40 0.14
57.38 0.27
GNN
0.63
80.34 = 1.63
98.16 0.14
59.36 0.39
EN
91.60 +..0
82.66 + 1.75
97.96 = 0.22
58.41 2.29
PN
91.63 0.45
77.16 + 2.06
97.55 1.25:...0 0.21
57.29 1.10
NaPPN
0.33
80.57 1.29
97.98 - 0.16
49.08 + 1.57
GPN
91.63
74.08 + 0.85
97.31 0.09
61.70 0.49
NatGPN
87.83 1.83:133
77.90 3.96
96.12 0.91
59.80 0.32
EnsGNN
9//18
81.62 = n/a
98.37 = n/a
60.81 = n/a
EnsNatPN
9//
81.86 n/a
98.16 n/a
46.46 = n/aa
EnsGPN
9//
75.26 n//a
97.45 n/a
61.75 n/a
PageRank
MLP
85.36 0.10
66.95 + 0.59
93.73 + 0.10
62.74 0.25
GNN
90.37
75.05 = 0.75
96.98 0.13
63.09 0.77
EN
0.37
75.18 1.10
96.90 0.13
62.20 0.95
88.26
73.56 1.40
95.88 0.20
1.18
NatPN
0.38
75.31 + 1.00
96.79 + 0.22
47.23 + 2.53
GPN
0.61
67.40 + 0.86
95.57 + 0.26 0.26
1.76
NatGPN
6.149
70.93 = 1.25
3.66
82.22 4.95
EnsGNN
nn/a
74.81 n/a
97.26 n/a
65.54
EnsNatPN
9//a
74.69 n/a
97.02 n/a
4
EnsGPN
t/a
66.27 + n/a
95.85 + n/a
92.39 + 1/a
PPR
MLP
49.68
45.47
78.29 1.27
GNN
2..6
54.98 10.32
67.28
86.94 1.93
EN
02.89
47.70 6.44
6..74
88.03 + 1.88
PN
60.22 1.811
50.42 11.67
72.29 3.33
82.47 1 2.45
NaPN
1.99
62.85 4.67
68.78
49.36 = 11.49
GPN
70.16 0.36
33.12 16.84
77.10
91.90 3.24
NatGPN
48.62 2.94
6.97
2.60
91.17 = 1.60
EnsGNN
54.56 1/aa
68.28
09.01nn/a
89.05 + n/a
EnsNatPN
55.24
62.02
70.58 n//a
43.71 n
EnsGPN
70.30 +
34.13
77.43 n/a
92.21 n/a
24

---

Under review as a conference paper at ICLR 2023
Table 18: Experiment results on CoauthorCS dataset.
Accuracy
PRR@TU
AUPRC@TU
AUROC@KU
Random
MLP
93.28 + 0.11
64.47 + 1.27
97.32 + 0.05
n/a
GNN
93.41 0.08
73.37 = 1.09
97.93
0.05
n/a
EN
98.33 0.12
74.24 0.84
97.95 0.05
n/a
90.29 0.78
75.42 = 0.98
96.90 0.29
n/a
NatPN
93.42 + 0.08
77.37 + 0.34
98.18 0.0M
n/a
GPN
90.51 + 0.15
65.11 + 0.84
96.10
n/a
NatGPN
88.91 0.28
63.57 + 1.44
n/a
EnsGNN
93.61 + n/a
73.36 = n/a
98.00 n/a
n/o
EnsNatPN
93.50 2 n/a
77.84 n/a
98.23 n/a
n/a
EnsGPN
90.89 1 n/a
64.94 = n//a
96.27 n/a
n/aa
Feature
MLP
92.42 0.20
62.88 + 1.39
96.83 0.08
64.00 + 0.40
GNN
98.48 1.11
71.83 = 0.70
97.86 0.04
50.67 = 0.82
EN
98.44 0.07
72.68 = 0.63
97.90
51.47 0.95
PN
89.72 :: 0.52
76.57 + 1.28
66.78:0.22
50.25 1 0.68
NaPP
93.60 + 0.09
75.68 + 0.31
98.13 0.03
44.86 + 1.22
GPN
90.65 0.18
66.68 + 0.39
96.30 0.08 0.08
+ 60.59
NatGPN
88.07 - 0.43
67.63 = 2.67
95.18 0.20
61.80 = 0.87
EnsGNN
93.72 : 1
71.61 = n/a
97.93 = n/a
49.91 = T1 n/a
EnsNatPN
93.64 n/a
76.41 = n/aa
98.19 n/a
43.61 = 1/aa
EnsGPN
91.00 n/a
67.16 - n/a
96.50 n//a
60.47 - 1//a
PageRank
MLP
93.15 + 0.25
60.50 + 1.14
97.01 + 0.08
68.00 + 0.38
GNN
92.03 0.12
70.03 = 0.58
97.17 0.06
52.17 = 0.29
EN
92.01 0.15
70.48 = 0.83
97.19 0.05
0.05
52.92 + 0.50
88.46 0.66
74.08 1 1.62
96.02 0.29
60.17 1:
NatPN
74.94 0.37
97.56 0.03 0.03
48.79 + 1.89
GPN
88.82
67.52 + 0.64
95.53
82.05 + 2.57
NatGPN
86.51
61.35 = 1.43
93.67
79.99 = 1.09
EnsGNN
92.05 n/a
70.57 = n/a
1.21=n/6
51.54 = n1
EnsNatPN
92.22 1
75.38 = n/a
97.63 n/a
48.16 = nl
EnsGPN
89.12 n/a
68.56 + n/a
95.77 n/a
80.38 + n/aa
PPR
MLP
90.46 + 0.35
61.93 + 1.70
95.81 0.10
73.96 + 1.06
GNN
91.84 0.13
67.13 - 0.82
96.87 0.08
66.72 = 1.59
EN
91.77 + 0.09
67.66 + 0.81
96.88
67.42 + 2.19
PN
85.21 0.79
73.32 1 1.56
M..410.556
66.03 1: 2.59
NaPN
92.29 + 0.08
74.19 1.79
97.57
42.36 1 2.24
GPN
88.31 0.38
71.15 + 0.29
95.65
88.38 + 0.78
NatGPN
85.07 0.48
63.26 + 2.25
93.10 0.39
80.75 - 1.98
EnsGNN
92.05 + 1
66.82 n/a
96.94 n/a
71.42 = n/a
EnsNatPN
92.41 n/a
74.70 =/a n/a
97.65 = n/a
40.08 = n/a
EnsGPN
88.74 n/a
72.05 n/a
95.94 n/a
88.81 1/a
25

---

Under review as a conference paper at ICLR 2023
Table 19: Experiment results on CoauthorPhysies dataset.
Accuracy
PRR@TU
AUPRC@TU
AUROC@KU
Random
MLP
95.53 + 0.08
75.21 + 0.34
98.74 0.02
n/a
GNN
96.32 + 0.06
81.17 = 0.47
99.20 0.01
n/a
EN
96.35 0.04
81.01 = 0.92
99.20 0.03
n/a
95.28 0.26
80.14 1.01
98.88 0.10
n/a
NatPN
96.31 0.05
82.37 + 0.61
99.21 0.02
n/a
GPN
95.32 + 0.08
79.13 + 0.44
98.85 0.03 0.03
n/a
NatGPN
94.37 0.32
79.53 + 1.17
0.15
n/a
EnsGNN
96.33 + n/a
81.47 = n/a
99.21
n/a
EnsNatPN
96.30 n/a
82.77 = n/a
99.25 n/a
n/a
EnsGPN
95.45 - v
79.41 = n1/a
98.90 = n/a
n/aa
Feature
MLP
95.54 0.03
75.55 + 0.27
98.76 0.02
52.04 + 0.08
GNN
96.16 0.05
81.91 = 0.73
99.18 0.03
51.82 = 0.25
EN
96.13 0.05
82.17 = 1.06
99.19 0.04
51.73 = 0.46
PN
95.31 + 0.16
80.88 +: 1.07
98.93 1 0.08
51.48 1 0.57
NaPN
96.09 0.06
84.92 + 0.27
99.28 0.02
48.32 1 0.54
GPN
95.12 0.07
82.13 + 0.57
98.93 0.02
54.36 +
NatGPN
93.92 0.36
80.34 = 0.79
98.50
54.30 = 0.31
EnsGNN
96.16 12166:1 1
82.27 = n/a
99.20 n/a
51.11 = n/aa
EnsNatPN
96.10 n
85.17 = n
99.29 n/a
47.89 = 1/aa
EnsGPN
95.15 n/a
82.75 n/a
98.97 n//a
54.50 n//a
PageRank
MLP
94.53 + 0.09
72.39 + 0.40
98.27 0.03
73.99 + 0.28
GNN
95.06 + 0.07
76.52 = 0.27
98.65 0.02
60.23 = 0.20
EN
95.07 0.05
76.70 = 0.82
98.66 0.03
61.02 = 1.03
93.43 0.39
76.39 1: 1.30
98.12 0.16
65.12 1
NatPN
95.08 0.06
78.15 + 0.69
98.73 0.03
47.06 + 3.39
GPN
93.41 + 0.08
75.97 + 0.33
98.09 + 0.04
78.31 + 0.88
NatGPN
91.68 0.58
72.12 1.44
97.17
82.95 = 3.02
EnsGNN
95.11 n/a
76.61 = n/aa
98.67 n/a
58.45 = n/a
EnsNatPN
9
78.39 = n/a
98.75 98.75
45.49 = n/a
EnsGPN
93.56 + n/a
76.09 + n/a
98.14 n/aa
78.72 + n/a
PPR
MLP
89.11
45.53 + 2.19
93.53 0.29
91.39 + 0.19
GNN
94.49
57.61 + 2.31
9.14
90.71 = 0.12
EN
94.26 0.34
54.63 + 2.40
97.2
90.69 + 0.34
PN
88.58 1: 1.89
59.44 1: 4.95
94.60
81.37 1: 4.48
NaPN
94.53
68.02 1 0.75
98.05
31.41 1 6.64
GPN
90.31 0.71
67.05 + 0.76
96.17 0.37
91.95 + 1.21
NatGPN
88,29 1.37
49.35 = 2.15
0.90
86.99 = 1.45
EnsGNN
94.58 + n/a
57.81 = n/a
9
90.42 n//a
EnsNatPN
94.67 n/a
67.86 = n/a
18.09In//
27.08 = n/a
EnsGPN
90.63 n/a
68.17 n/a
96.42
92.58 n/a
26

---

Under review as a conference paper at ICLR 2023
Table 20: Experiment results on CoraMI. dataset.
Accuracy
PRR@TU
AUPRC@TU
AUROC@KU
Random
MLP
72.08 + 0.65
57.70 + 1.27
83.69 + 0.41
n/a
GNN
86.95 0.33
70.63 = 1.64
94.97 0.12
n/a
EN
87.01 0.33
70.14 2.07
n/a
78.81 0.79
62.59 + 3.10
89.26 0.71
n/a
NatPN
87.08 0.66
70.83 + 2.21
95.06:0...
n/a
GPN
85.42 + 0.17
65.41 + 0.94
93.57
n/a
NatGPN
86.28 0.18
63.90 + 1.44
9318
n/a
EnsGNN
87.31 + n/a
71.37 = n/a
9//
n/a
EnsNatPN
n//a
71.43 = n/a
95.29
n/a
EnsGPN
85.64 /
64.25 = n//a
93.54 n/aa
n/aa
Feature
MLP
71.66 1.08
54.74 1.24
0.60 82.78
62.42 + 0.49
GNN
86.31 = 0.44
66.29 = 2.42
94.15
52.56 - 0.55
EN
86.25 + 0.33
66.77 = 1.59
94.17
52.70 = 0.38
PN
77.61 +: 1.77
62.83 + 3.97
38.5211.22
49.55 + 0.67
NaPP
86.09 0.32
67.00 + 2.20
91.12
43.51 1 1.20
GPN
84.58 0.22
61.94 +: 1.00
92.66
51.97 + 0.88
NatGPN
85.16 1 0.40
63.19 + 1.60
9321
55.16 = 0.80
EnsGNN
86,91 + 1
66.17 = n/a
9/.44
50.77 = n/aa
EnsNatPN
86.57 + n/a
66.70 = n/aa
94.33
40.67 = n/aa
EnsGPN
84.30 44.30 n/a
62.47 - n/a
92.57 + n/a
54.24 n//a
PageRank
MLP
70.14 0.66
56.09 + 1.64
+1.99
56.16 + 0.61
GNN
66.28 = 1.00
93.07
53.23 = 0.65
EN
84.57 0.36
65.17
93.07
53.27 = 0.59
74.96
56.41 1: 3.92
85.53 1.57
59.91 1 1.52
NatPN
64.28 + 1.05
0.10
59.95 + 2.45
GPN
83.87 0.28
65.16 + 1.11
0.26
82.48 + 2.87
NatGPN
83.74 0.28
66.33 = 1.04
92.77
74.32 = 1.97
EnsGNN
84.70 n/a
67.09 = n/a
9/a
54.17 = n/a
EnsNatPN
1.97:n/
65.42 = n/a
n/a
65.41 = n/a
EnsGPN
83.77 + n/a
64.28 + n/a
92.51 + n/a
81.42 + n/a
PPR
MLP
56.41 1.41:00
37.24 + 0.92
65.56 0.47
63.84 0.53
GNN
77.76
58.02 = 3.22
8.57
69.36 - 0.82
EN
59.21 = 2.36
87.71
68.21 0.94
PN
59.39 2.66
51.36 1 6.21
1.57
72.71 1: 4.29
NaPPN
78.40
60.01 + 1.57
88.56
58.37 1 4.00
GPN
77.54 1.46
59.74 + 1.79
87.93 1.04
92.77 + 0.77
NatGPN
77.49 + 1.37
56.85 + 3.25
+ 1.24
84.86 - 1.65
EnsGNN
79.16 1116:10 n
56.34 = n/a
88.45 = n/a
73.52 = n/a
EnsNatPN
79.23 + n/a
59.96 = n/a
89.09 = n/a
63.61 = n/a
EnsGPN
78.62 n/a
60.95 n/a
88.87 1//a
92.32 n/a
27

---

Under review as a conference paper at ICLR 2023
Table 21: Experiment results on CiteSeer dataset.
Accuracy
@RR@TU
AUPRC@TU
AUROC@KU
Random
MLP
66.57 + 0.33
53.58 + 0.56
78.50 + 0.27
n/a
GNN
72.39 + 0.44
56.19 = 1.65
8 0.16
n/a
EN
72.34 0.53
56.17 = 1.67
83.58 + 0.20
n/a
61.44
48.78 + 1.91
75.61 + 0.86
n/a
NatPN
72.52 +: 0.49
55.19 + 1.83
83.52 + 0.41
n/a
GPN
69.94
50.41 + 1.42
80.54
n/a
NatGPN
70.88 + 0.58
50.44 + 0.64
81.29 0.56
n/a
EnsGNN
73.50 n/a
55.15 = n/a
84.24 n4/24=n/aa
n/o
EnsNatPN
73.20
53.89 = n/a
83.77 n/a
n/a
EnsGPN
70.43
49.69 = 1//a
80.78 + n/a
n/aa
Feature
MLP
64.70 0.36
50.93 = 1.11
76.33 0.17
55.84 + 0.19
GNN
71.37 0.23
49.62 = 1.14
81.51 0.26 0.26
51.09 - 0.91
EN
71.26 - 0.39
49.82 = 1.60
8
50.96 + 1.27
PN
62.21 + 0.49
48.65 + 2.11
7.445
50.70 1 0.24
NaPP
71.50 0.56
49.10 + 0.67
81.51
47.09 + 1.81
GPN
68.00 0.46
42.05 + 1.82
0.11
51.76 + 0.37
NatGPN
69.15 1155=0.0 0.32
44.22 + 1.07
78.58
50.30 + 1.40
EnsGNN
47.84 n/a
81.97
50.18 = 11
EnsNatPN
72.12
48.70 = n/aa
81.91
45.65 = 1/aa
EnsGPN
68.27
42.19 n/a
77.41 n/aa
51.62 - n//a
PageRank
MLP
65.96 0.34
46.36 + 0.54
76.37
54.33 + 0.23
GNN
72.03 0.31
51.64 1.39
0.19
50.82 = 0.43
EN
71.79
52.28 =
82.38
50.73 = 0.78
62.51
46.12 1 2.34
0.65 73.32
56.33 1
NatPN
7.11
50.76 + 1.02
0.17
54.78 + 1.55
GPN
0.46
41.23 +: 2.81
79.36 0.226
71.78 + 1.42
NatGPN
71.65
42.54 = 1.84
0.34
59.46 = 3.89
EnsGNN
72.48
52.17 = n/a
n//a
52.31 = n/a
EnsNatPN
51.35 = n/a
8/.04
57.63 = n/a
EnsGPN
70.91
42.07 + n/a
79.59 n/a
77.27 + n/aa
PPR
MLP
55.53
39.66 + 1.44
65.32
69.37 + 0.39
GNN
62.22
44.67 = 2.44
7.67
81.83 + 0.54
EN
62.18 0.91
44.78 + 1.08
72.70
80.35 + 0.77
PN
46.30 1.00
33.99 1 8.13
5..74
80.83 1 1.27
NaPN
63.09 0.58
41.88 1 1.41
7.384
47.42 1 6.69
GPN
56.27 1.23
40.11 3.87
6.714
93.34 +
NatGPN
64.00
36.33 0.48
7.17
88.78 - 4.75
EnsGNN
64.48 n/a
43.41
7//4
85.18 n/a
EnsNatPN
64.54 n/a
7//22 4/a
44.31 = n/a
EnsGPN
59.38 n/a
37.66 + n/a
68.46 n/a
92.86 1/aa
28

---

Under review as a conference paper at ICLR 2023
Table 22: Experiment results on PubMed dataset.
Accuracy
TU
AUPRC@TU
AUROC@KU
Random
MLP
85.59 + 0.20
58.82 + 0.43
92.85 + 0.06
n/a
GNN
86.74 0.22
64.45 = 0.74
94.15 0.12
n/a
EN
86.90 0.28
64.78 = 0.77
94.27 0.12
n/a
85.89 0.38
57.80 + 1.45
0.17 92.89
n/a
NatPN
86.58 0.31
63.84 + 0.84
#.11
n/a
GPN
87.31 0.23
63.46 + 0.41
0.41
0.11
n/a
NatGPN
85.08 0.81
60.22 + 1.96
92.71
n/a
EnsGNN
87.43 n/aa
66.12=11 n/a
94.70=n/aa
n/o
EnsNatPN
n
65.21 n/a
94.45
n/a
EnsGPN
87.39 +: 1
64.30 = n//a
94.48 - n/a
n/a
Feature
MLP
84.66 0.14
60.50 = 0.34
92.52 0.04
48 + 0.09
GNN
86.32 0.17
65.27 = 0.39
94.03 0.06
50.50 = 0.24
EN
86.36 + 0.30
65.81 = 1.03
94.11
50.27 = 0.31
PN
85.46 + 0.19
58.47 + 1.87
92.73 + 0.24
49.03 + 0.84
NaPP
86.15 + 0.17
64.99 + 0.40
93.91 + 0.06
50.73 + 0.28
GPN
86.89 0.29
62.46 0.73
94.00
52.47 + 0.37
NatGPN
84.76 1.06
60.53 = 1.60
92.56 0.84
50.83 = 0.28
EnsGNN
87.15 + 1
66.24 n/a
94.57 n/a
50.33 = n/a
EnsNatPN
86,67 + n/aa
66.59 = n/aa
94.36 = n/a
51.47 = 1/aa
EnsGPN
87.17 n/a
62.69 n/a
94.18 n/a
52.77 1//a
PageRank
MLP
85.31 + 0.11
58.78 + 0.43
92.68 + 0.05
51.32 + 0.09
GNN
84.45 + 0.20
60.85 = 0.86
0.19
55.99 = 0.39
EN
84.77 0.33
60.97 = 0.37
0.211
56.80 = 0.38
83.12 0.25
52.37 1: 1.29
90.47 0.23
66.37 1:
NatPN
84.29 - 0.38
61.00 + 0.40
+.20
56.87 + 1.97
GPN
86.06 + 0.18
63.96 + 0.42
03.73
82.67 + 0.59
NatGPN
82.66 + 0.79
59.50 = 2.85
91.17
74.55 = 1.56
EnsGNN
85.09 n/a
63.43 = n/a
n
60.77 = nl
EnsNatPN
84.88 + n/a
62.66 = n/a
92.92 n/a
61.38 = n/a
EnsGPN
86.41 + n/a
64.25 + n/a
93.95 + n/a
83.77 + n//a
PPR
MLP
84.12 + 0.05
53.29 + 0.34
91.24 0.05
56.44 + 0.24
GNN
84.80 - 0.20
60.82 = 0.70
9.16
58.41 = 1.03
EN
85.04 + 0.31
61.21 + 0.65
92.83
66.78 + 2.97
PN
83.31 1: 0.40 0.40
48.27 1: 1.77
90.02
72.08 1 1.95
NaPPN
84.50 -:00 0.43
60.66 + 1.06
9.21
61.40 1 4.36
GPN
86.05 0.14
62.12 + 1.35
93.50
74.88 + 1.93
NatGPN
82.85 1.88
57.71 = 2.81
91.01
68.86 = 0.67
EnsGNN
85.82 + n/a
62.38 = n/a
9
62.02 = n/a
EnsNatPN
85.18 n/a
62.19 n/a
n/a 93.03 n/a
64.73 = n/a
EnsGPN
86.18 n//
62.76 n/a
93.65 n/a
74.62 1//a
29

---

Under review as a conference paper at ICLR 2023
E COMPONENTS OF DIRICHLET-BASED FRAMEWORK
E.1 STANDARD OR NATURAL DENSITY ESTIMATION
One property that vary in these methods is the number of Normalizing Flows and how we use their density estimates, In particular, we consider a Standard approach (Charpentier et al. 2020). where distinct Normalizing Flows P kk) are used per each class & to predict the corresponding Dirichlet parameter ffen based on the representation 2, (ax):
Pik Pyi
(z,, p
k) n (2, P(k) 7i (2,
where the probability P(k) of class k is approximated by the ratio of train observations from class k, n is usually equal to the dataset size (but can, in general, be a (yper-parameter). and my = n p(k)..
Another approach is the Natural version of Posterior Network that is proposed by Charpentier al. (2021). It exploits a single Normalizing Flow (2,) to predict the evidence S,, while the normal- ized Dirichlet parameters μ4, obtained using one-layer linear transformation (2,). The final predictions are obtained as follows:
S. = n (=)))), pe; = (w.(z)) =
Bik S₁ pssk
E.2 GRAPH ENCODING OR GRAPH PROPAGATION
Another aspect that analyze is how the graph structure be combined with Posterior Networks. Here, we consider two approaches - use the graph for the encoding to obtain z; or use graph propagation smooth the predicted parameters the post-processing step.
In the case of graph encoding, we use a two-layer SAGE convolution to produce the representations z; and then estimate the density as usual via Normalizing Flows. Thus, the graph structure used for the pre- pre-processig step.
The second approach is adopted by Stadler al. (2021). In this case, the initial representations z; are obtained using a graph-agnostic two-layer MLP, while graph propagation is applied to smooth the Dirichlet parameters Gfeai Graph propagation is performed in several steps via some transformation RuxC gaxC follows:
= 1=+1= (1-)) aB, +aB)
where B B E R=xC formed by the parameters 3rea and a a hyperparameter that controls the smoothing effect of the propagation step. Similarly to Stadler et al. (202T), use Personalized Propagation scheme (Klicpera et al. 2018). which takes into account the mutual importance of nodes and is defined as π (B) D 1/2
1-2 AA AD-1/2B.
We refer to the method with graph propagation as GPN (Graph Posterior Network) and with Graph Encoding as PN (Posterior Network). Further, if the Natural version of the Posterior Network is used instead of the Standard one, we denote the models NatGPN and NatPN, respectively.
F DATASET DETAILS
Table 23: Description the considered graph datasets for node classification task.
#Nodes
#Edges
#Classes
#Features
AmazonComputers
13,381
259,159
10
767
AmazonPhoto
7,484
126,530
745
CoauthorCS
18,333
163,788
6,805
CoauthorPhysies
34,493
495,924
5
8,415
CoraML
2,995
16,316
7
2,879
CiteSeer
3,327
4,732
6
3,703
PubMed
19,717
44,338
3
8,415
30

---

Under review as a conference paper at ICLR 2023
G EXPERIMENTAL SETUP
We evaluate all the methods discussed in Section using the benchmark proposed in Section Some of the considered methods require specific training procedure consisting of several stages. Methods without Normalizing Flows, including MLP. GNN, EnsGNN, and EN, have only one training stage when the corresponding models train the standard end-to-end mode. Methods with Normalizing Flows have at least three phases of training warm-up up of exclusively Flow neural layers, end-to-end training of the entire model, and finetuning the same Flow layers. In this regard, we follow the setup of Stadler et al. (021). Further details can be found in Appendix G
In our experiments, one training stage (i.e., warm-up, main training stage, or finetuning) takes 200 epochs, while the best loss value on the Valid-In. part serves as criterion for saving the model checkpoint. We exploit the standard Adam optimizer (Kingma & Ba 2014) with a learning rate of 0.001 for Normalizing Flows and 0.0003 for other neural modules. For all the considered models, we utilize weight decay of 0.00001 and set = 0.001 in Expected Cross-Entropy.
As for model configurations, set the hidden size of linear layers to 64, the number of layers in Normalizing Flows to 8 and the latent space dimension to 16. Also, Graph Propagation is performed in steps with 0.2.
31


Reviews:
[{"review_id": "s-FQtJxYLe", "reviewer": "Reviewer_zVhc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a benchmark for evaluating the uncertainty estimation in node classification tasks. The benchmark consists of seven common datasets, four problems for evaluating the performance of existing methods, the associated metrics, and a universal data-splitting approach. The key idea of the data-splitting approach is to use node characteristics depending on features or graph structures as splitting factors, leading to different distribution shifts. Experiments show that GPN [1] and NatPN [2] are the state-of-the-art methods for the OOD detection and the misclassification detection, respectively.\n\n[1] Stadler, Maximilian, et al. \"Graph posterior network: Bayesian predictive uncertainty for node classification.\" Advances in Neural Information Processing Systems 34 (2021): 18033-18048.\n[2] Charpentier, Bertrand, et al. \"Natural Posterior Network: Deep Bayesian Predictive Uncertainty for Exponential Family Distributions.\" International Conference on Learning Representations. 2021.\n", "summary_of_the_review": "This paper proposes a benchmark for evaluating the uncertainty estimation in node classification tasks. The proposed new data-splitting method in the benchmark accounts for the graph structures that existing benchmarks ignore. In the complex distribution shift scenario generated by the approach, most existing methods fail to maintain a high classification performance and consistency of uncertainty estimates with prediction errors.\n\nHowever, the paper is below the acceptance bar of ICLR because the proposed benchmark is not very helpful and insightful for real applications of the uncertainty estimation (please refer to concerns 1-3). Besides, there are some carless mistakes, e.g., Figures 6 and 7 are the same. If the authors can properly address my concerns, I am willing to raise my score.\n", "strength_and_weaknesses": "Strength:\n\nThe proposed data-splitting approach accounts for the graph structures that existing benchmarks ignore. In the complex distribution shift scenario generated by the proposed approach, most existing methods fail to maintain a high classification performance and consistency of uncertainty estimates with prediction errors.\n\nHowever, my major concerns are as follows.\n1. The authors may want to provide references of realistic datasets which use the proposed PR and PPR splits. Otherwise, the proposed splits may amplify the locality-based and popularity-based biases in real-world applications.\n\n2. Please add some large-scale datasets to the proposed benchmark. The seven datasets in the benchmark have only 2,000 to 35,000 nodes, but the graphs in real-world applications may be large-scale, e.g., ogbn-products [3] and ogbn-mag [3] have more than 1,000,000 nodes.\n3. The authors may want to further analyze or discuss why the Dirichlet-based methods that the benchmark evaluates perform well or poorly on different tasks. This could provide insights for developing new Dirichlet-based methods.\n\n4. The description of ${\\rm PRC}\\_{\\rm oracle}$ needs improvements. In the case that uncertainties of all misclassified samples are different from each other, the value of ${\\rm PRC}\\_{\\rm oracle}$ may not be unique. Which value of ${\\rm PRC}\\_{\\rm oracle}$ do we choose to compute ${\\rm PRR}$?\n\n[3] Hu, Weihua, et al. \"Open graph benchmark: Datasets for machine learning on graphs.\" Advances in neural information processing systems 33 (2020): 22118-22133.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eudF3UBIZA", "reviewer": "Reviewer_fvq7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a benchmark consisting of several graph datasets with different distributional shifts to evaluate the robustness and uncertainty estimation of models for node-level tasks.  Furthermore, a series of experiments have been conducted to show the ID/OOD performance gap.  ", "summary_of_the_review": "In summary, this paper introduces distribution shifts in data splits, but the lack of technical novelty as well as evidence makes the current paper not strong enough for acceptance.", "strength_and_weaknesses": "Strengths:\n- Distribution shift problem is important to the graph learning community\n- Various distributional shifts are considered \n- Comprehensive experiments have been conducted to show that ID/OOD performance graph and SOA models performance. \n\n Weaknesses: \n- I appreciate the extensive experimental efforts, however, the technical contribution of the paper is limited. \n-  The paper does not spend much time discussing the distribution shifts which is the main contribution of the paper. I would like to see more discussion, motivation, and limitations of the Personalized PageRank (PPR).  Furthermore, the process of computing PPR is not explained well. \n- The majority of the paper is dedicated to uncertainty estimation of nodes, metrics for evaluation, and methods which have been well documented in the literature and are not the main contribution here.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "WNE1AbC9OMv", "reviewer": "Reviewer_6NVp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the Uncertainty Estimation for Node Classification problems and proposes a benchmark together with a technique for the controllable generation of data splits with various types of distributional shift. The proposed benchmark consists of several graph datasets equipped with various distributional shift on which we evaluate the robustness of models and uncertainty estimation performance.\n", "summary_of_the_review": "See *Strength And Weaknesses*\n\n------------ after rebuttal ----------------\n\nThe authors' response addressed most of my concerns. But I still suggest adding the MC-Dropout as a baseline in the future version as it is a benchmark paper. I will raise my score from 5 to 6.", "strength_and_weaknesses": "Strengths:\n\n1. This paper proposes a new benchmark for evaluating robustness and uncertainty estimation in transductive node classification tasks.\n2. Using the proposed benchmark, the authors evaluate the robustness of various models and their ability to detect errors and OOD inputs\n\n\nWeaknesses:\n\n1. Similarly to the OOD detection in the image classification task, it is not necessary to use the Valid-out samples. I suggest conducting more experiments without Valid-out samples.\n2. One important baseline (Zhao et al., 2020) is missing, which is introduced in Section 1. It is better to compare this baseline as well.\n3. Since this paper focuses on the benchmark dataset, why not consider the MC-Dropout uncertainty estimation method as a baseline to compare? \n4. For Table 1, what is Diff. %? Is that possible to construct different Diff. % for each dataset?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject