Uncertainty-Aware Prediction for Graph Neural Networks
Authors: Xujiang Zhao
Feng Chen
Shu Hu
jin-Hee Cho
Venue:  (2020)
Keywords: Uncertainty
Graph Neural Networks
Subjective Logic
Bayesian

Abstract: Thanks to graph neural networks (GNNs), semi-supervised node classification has shown the state-of-the-art performance in graph data.  However, GNNs do not consider any types of uncertainties associated with the class probabilities to minimize risk due to misclassification under uncertainty in real life. In this work, we propose a Bayesian deep learning framework reflecting various types of uncertainties for classification predictions by leveraging the powerful modeling and learning capabilities of GNNs. We considered multiple uncertainty types in both deep learning (DL) and belief/evidence theory domains. We treat the predictions of a Bayesian GNN (BGNN) as nodes' multinomial subjective opinions in a graph based on Dirichlet distributions where each belief mass is a belief probability of each class. By collecting evidence from the given labels of training nodes, the BGNN model is designed for accurately predicting probabilities of each class and detecting out-of-distribution.  We validated the outperformance of the proposed BGNN, compared to the state-of-the-art counterparts in terms of the accuracy of node classification prediction and out-of-distribution detection based on six real network datasets.

Under review as a conference paper at ICLR 2020
UNCERTAINTY-AWAR PREDICTION FOR GRAPH NEU- RAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Thanks to graph neural networks (GNNs), semi-supervised node classification has shown the state-of-the-art performance in graph data. However, GNNs have not considered different types of uncertaintics associated with the class probabilities to minimize risk increasing misclassification under uncertainty in real life. In this work, we propose Subjective Bayesian deep learning framework reflecting vari- ous types of uncertainties for classification predictions by leveraging the powerful modeling and learning capabilities of GNNs. We considered multiple uncertainty types in both deep learning (DL) and belief/evidence theory domains. We treat the predictions of Subjective Bayesian GNN (S-BGNN) as nodes' multinomial subjective opinions in a graph based on Dirichlet distributions where each belief mass a belief probability of each class. By collecting evidence from the given labels of training nodes, the S-BGNN model is designed for accurately predict- ing probabilities of each class and detecting out-of-distribution. We validated the outperformance of the proposed S-BGNN, compared to the state-of-the- counterparts in terms the accuracy of node classification prediction and out-of- distribution detection based on six real network datasets.
INTRODUCTION
Inherent uncertainties introduced by different root causes have emerged as serious hurdles to find effective solutions for real world problems. Critical safety concerns have been brought due to lack of considering diverse causes of uncertainties, resulting in high risk due to misinterpretation of uncertainties (e.g., misdetection misclassification of an object by autonomous vehicle). Graph neural networks (GNNs) (Kipf & Welling, 2016: Veličković et al.) 2018) have gained tremendous attention in the data science community. Despite their superior performance in semi-supervised node classification and/or regression, they didn't it allow to deal with various types of uncertainties. Predictive uncertainty estimation (Malinin & Gales] 2018) using Bayesian NNs (BNNs) has been explored for classification prediction or regression in the computer vision applications, with well- known uncertainties, aleatoric and epistemic uncertainties. Aleatoric uncertainty only considers data uncertainty derived from statistical randomness (e.g., inherent noises in observations) while epistemic uncertainty indicates model uncertainty due to limited knowledge or ignorance in collected data. On the other hand, in the belief or evidence theory, Subjective Logic (SL.) (Josang et al., (2018) considered vacuity (or lack of evidence) as uncertainty in an subjective opinion. Recently other uncertainties such as dissonance, consonance, vagueness, and monosonance (Josang et al. 2018) are also introduced.
This work is the first that considers multidimensional uncertainty types in both DL and belief theory domains to predict node classification and distiiititttiiionnnnnnooo (OOD) detection. To this end, incorporate the multidimensional uncertainty, including vacuity, dissonance, aleatoric uncertainty, and epistemic uncertainty selecting test nodes for Bayesian DL in GNNs. We perform semi-supervised node classification and OOD detection based on GNNs. By leveraging the modeling and learning capability of GNNs and considering multidimensional uncertainties in SL, we propose a Bayesian DL framework that allows simultaneous estimation of different uncertainty types associated with the predicted class probabilities of the test nodes generated by GNNs. We treat the predictions of Subjective Bayesian GNN (S-BGNN) as nodes' subjective opinions in a graph modeled as Dirichlet distributions on the class probabilities, and learn the S-BGNN model by collecting the evidence from the given labels of the training nodes (see Figure [}. This work has the following key contributions:
A Subjective Bayesian framework to predictive uncertainty estimation for GNNs. Our pro- posed framework directly predicts subjective multinomial opinions of the test nodes in a graph,

---

Under review as a conference paper at ICLR 2020
Loss = = + kL(pll))
+
l pll
1
G
Belli
sample
sample
Vacuity (From Subjective Logic)
Dissonance
Input
(SL Opinion))
Dirichiet Distriuation
Categorical Dipribution
Prediction Label
Knowledge
sample A
Aleatoric
Epistemic
Prior Distribution
Deterministo
GNN
Training Label
(From Bayeslon Framework)
[a] Subjective Bayesion GNN
(b) Graph-Based Kernei
(c) Teacher Network
(d) Square Loss
(e) Multiple Uncertointies
Dirichlet Estimation
Figure I: Method Overview. our proposed framework is hased on (a) Subjective Bayesian GNN designed for estimating the different types of uncertainties including (e) vacuity, dissonance, aleatoric, and epistemic uncertainties for the applications in graph data. The loss function includes (d) square to reduce bias and (b) (c) two KL components to reduce error predicting uncertainty.
with the opinions following Dirichlet distributions with each belief probability as a class probability, Our proposed framework is generative model, so it cal be highly applicable across all GNNs and allows simultaneously estimating different types of associated uncertainties with the class probabilities.
Efficient approximate inference algorithms: We propose a Graph-based Kernel Dirichlet distri- bution Estimation (GKDE) method to reduce error in predicting Dirichlet distribution. We designed an iterative knowledge distillation algorithm that treats deterministic GNN teacher network while considering our proposed Subjective Bayesian GNN model (a realization of our proposed framework for specific GNN) distilled network. This allows the expected class probabilities based on the predicted Dirichlet distributions (i.e., outputs of our trained Bayesian model) to match the predicted class probabilities of the deterministic GNN model, along with uncertainty estimated in the predictions.
Comprehensive experiments for the validation of the performance of our proposed frame- work. Based on six real graph datasets, we compared the performance of our propose framework with that of other competitive DL algorithms. For a fair comparison, we tweaked the DL algorithms to consider various uncertainty types in predicted decisions.
RELATED WORK
Epistemic Uncertainty in Bayesian Deep Learning (BDL): Machine/deep learning (M/DL) re- search mainly considered aleatoric uncertainty (AU) and epistemic uncertainty (EU) using BNNs for computer vision applications. AU consists of homoscedastic uncertainty (i.e., constant errors for different inputs) and heteroscedastic uncertainty (i.e., different errors for different inputs) (Gal 2016). A BDL framework
presented to estimate both AU and DU simultancously in regression settings (c.g., depth regression) and classification settings (c.g., semantic segmentation) Kendall & Gal 2017). Later, new type of uncertainty, called distributional uncertainty (DU), is defined based on distributional mismatch between the test and training data distributions (Malinin & Gales 2018). Dropout variational inference (Gal & Ghahramani] 2016) is used as one of key approximate inference techniques in BNNs. Other methods (Eswaran et al. 2017 Zhang et al.) (018)) measure overall uncertainty in node classification but didn't consider uncertainty decomposition and GNNs.
Uncertainty Quantification in Belief/Evidence Theory: In the belief/evidence theory domain, un- certainty reasoning has been substantially explored, such as Fuzzy Logic (De Silva 2018). Dempster- Shafer Theory (DST) (Sentz et al. 2002), or Subjective Logic (SL) (Jøsang) 2016). Belief theory focuses on reasoning of inherent uncertainty in information resulting from unreliable, incomplete, deceptive, and/or conflicting evidence. SL considered uncertainty in subjective opinions in terms of vacuity (i.c., lack of evidence) and vagueness (i.e., failing in discriminating a belief state) (Jøsang 2016). Recently, other uncertainty types have been studied, such as dissonance (due to conflicting evidence) and consonance (due to evidence supporting composite states) (Josang et al. 2018).
In deep NNs, SL. considered to train deterministic NN for supervised classification in computer vision applications (Sensoy et al. 2018). However, they didn't t consider a generic way of estimating multidimensional uncertainty using Bayesian DL for GNNs used for the applications in graph data.
2

---

Under review as a conference paper at ICLR 2020
3 PROPOSED APPROACH
Now we define the problem of uncertainty-awar semi-supervised node classification and then present a Bayesian GNN framework to address the problem.
3.1 PROBLEM DEFINITION
Given an input graph - (4, Et,,,r, where V = fl,.... ,N is ground set of nodes, IE C V V V is ground set of edges, r = [r1, ... [N]" E R N Xxd is a node-level feature matrix, r; E R is the feature vector of node i, Yr. {y i E L} are the labels of the training nodes L C V, and
y: E (I..... K is the class label of node i. We aim to predict: (1) the class probabilities of E V L.]; and (2) the associated multidimensional
the testing nodes: PVLL = (P: E [0, 1) uncertainty estimates introduced by different root causes: uyyL = {u; E 11mh i v V\ L). where Pi, is the probability that the class label y/ l and m is the total number of uncertainty types.
3.2 MULTIDIMENSIONAL UNCERTAINTY QUANTIFICATION
Multiple uncertainty types may be estimated, such aleatoric uncertainty, epistemic uncertainty, vacuity, dissonance, among others. The estimation of the first two types of uncertainty relies on the design of an appropriate Bayesian DL model with parameters, O. Following (Gal 2016), node i's aleatoric uncertainty is: Aleatoric [P:] = P (g) [H(y,)r: [r: 0)] where H(-) is Shannon's entropy
of Prob(p; rr; 0). The epistemic uncertainty of node is estimated by:
Epistemic(pi [p:] = [Emobto))) [(y O)]] Eprob(0(a) 1M((y,1r;0))
(1)
where the first term indicates entropy (or total uncertainty).
Vacuity and dissonance can be estimated based the subjective opinion for each testing node
i (Josang et al., 2018). Denote i's subjective opinion as [b.1.... dbk u.]. where b;x(2 (2 0) is the belief mass of the k-th category, v2(2 0) is the uncertainty mass (i.e., vacuity), and K is the total number of categories, where ΣK bis + 02 = 1. i's dissonance is obtained by:
K
duk Σx b big Bay (aee,, bas)
w(b,) Σ k=1
(2)
Σラ bij
where the relative mass balance between a pair of belief masses b.j and bu is expressed by Bal(bij, biz) = - [b.j - buxl/(bs, + ba). To develop Bayesian GNNs framework that pre- dicts multiple types of uncertainty, we estimate vacuity and dissonance using a Bayesian model. In SL, a multinomial opinion follows a Dirichlet distribution, Dir(p, a.), where a E [1, oo] represents the distribution parameters. Given S; = >E 2 can be obtained by bak = (ass - 1)/S, and v, K/S,
k=1 O'kk, belief mass b, and uncertainty mass
3.3 PROPOSED BAYESIAN DEEP LEARNING FRAMEWORK
Let p IP . PN] T N denote the class probabilities of the node in V, where P. [pa, p Pik. T refers the class probabilities of specific node i. As shown in Figure our proposed Bayesian GNN framework can be described by the generative process:
Sample 0 from a predefined prior distribution, i.e.. N/(. I).
For each node E V: (1) Sample the class probabilities P. from Dirichlet distribution: Dir(p, la)). where a; (r; 0) is parameterized by a GNN network a f(r 0) Nxd + [1, 20 ] NxK that takes the attribute matrix r as input and directly outputs all the node-level Dirichlet parameters
a [ar,. xx.. and O refer to the hyper- parameters of the GNN network; and (2) Sample y/ ~ Cat(ys/p,) p,), a categorical distribution on P.-
In this design, the graph dependencies among the class labels in yL and yv are modeled via the GNN network (r; 9). Our proposed framework is different from the traditional Bayesian GNN network (Zhang et al. 2018) in that the output of the former are the parameters of node-level Dirichlet distributions (ox), but the output of the latter are directly node-level class probabilities (p). The conditional probability of p. Prob(p)r; 0), can be obtained by:
Prob(plr: 0) II N
1=1 Dir(p, a, fr(
(3)
where the Dirichlet probability function Dir(p, /ox)) is defined by:
T(S:)
-1 1 S = Σ K
Dir(p,la:) IT I'(ax)
Pik
(4)
k=
3

---

Under review as a conference paper at ICLR 2020
Based on the proposed Bayesian GNN framework, the joint probability of conditioned on the input graph G and the node-level feature matrix r can be estimated by:
/
Prob(y r; (r
Prob(ylp) Prob(( Probbb(
(5)
where Prob(Olg) is the posterior probability of the parameters @ conditioned on the input graph g. which are estimated in Sections (34 and (3.6
The aleatoric uncertainty and the epistemic uncertainty can be estimated using the equations de- scribed in Section 3.2 The vacuity associated with the class probabilities (p:) of node i can be
estimated by: Vacuity (p)) [u:] (poo(())))
K Σ Ei=1 Osk) The dissonance of node i
K
estimated as: Disso.[pi [p:] Ep (bb.) where w(b)) defined in Eq.
3.4 BAYESIAN INFERENCE WITH DROPOUT
The marginalization in Eq. (5) is generally intractable. dropout technique used to obtain an approximate solution and use samples from the posterior distribution of models (Gal & Ghahramani 2016). Due to this reason, we adopt a dropout technique (Gal & Ghahramani 2015) for variational inference in Bayesian CNNs where Bernoulli distributions are assumed over the network's 's weights. This dropout technique allows us to perform probabilistic inference over our Bayesian DL framework using GNNs. For Bayesian inference, we identify a posterior distribution over the network's ' weights, given the input graph 9 and observed labels yr. by Prob 9). where = (WI...., WL, L, b1, by}: L is the total number of layers and W; refers to the GNN's weight matrices of dimensions P. X P,-1. and b; is a bias vector of dimensions P. for layer = 1,... LL.
Since the posterior distribution is intractable, we use variational inference learn (),,)), distribution over matrices whose columns are randomly set to zero, approximating the intractable posterior by minimizing the Kullback-Leibbe (KL.)-iivergence between this approximated distribution and the full posterior, which is given by:
min (L(q(O, ()][Prob(01gg)
(6)
where y= (M Mr. m. m............................... m are the variational parameters, where M. RPxP-- and m, E Rd, We define W. in (0, ()(0)))))) by:
w, =,=MM M,diag((22))]=1)
2i ~ (Beeolll(()) for i 1,...,,,,;
P-1 (7)
where =dd....,ddz) ] is the dropout probabilities with 24j of Bernoulli distributed random variables. The binary variable Σi = 0 corresponds to unit j in layer i being dropped out as an input to layer i. We can obtain the approximate model of the Gaussian process from (Gal & Ghahramani] 2015). The dropout probabilities, d, ss can be optimized or fixed (Kendall et al. 2015). it is beyond the scope of our study. In Gal &
For simplicity, we fix d, 's in our experiments, Ghahramani 2015), the minimization of the cross entropy (or square error) loss function is proven to minimize the KL-divergence (see Eq. (6)). Therefore. training the GNN model with stochastic gradient descent enables learning of an approximated distribution of weights, which provides good explainability of data and prevents overfitting.
For the dropout inference, we performed training GNN model with dropout before every weight layer and dropout at test time to sample from the approximate posterior (i.e., stochastic forward passes, a.k.a. Monte Carlo dropout; see Eq. { Eq. (5) by:
At the test stage, we infer the joint probability
Prob(y|r; ~ M m= / Prob(pl 0 ((m))) (m)) ~ g()),
1
which can infer the Dirichlet parameters a as: a 2 1 ΣM frr ((m)). ((m) q(0).
As our model is generative model to predict Dirichlet distribution parameters, use loss function to compute its Bayes risk with respect to the sum of squares loss lly 2l by:
((>) = - Ily. 1. Prob(p, 7)dp. = Σ - Σ j-1 (ye - Elpss) Var(p:s) iEL.
Eq. ( aims to minimize the prediction error and variance, leading to maximizing classification accuracy of each training node by removing excessive misleading evidence (Sensoy et al. 2018).
4

---

Under review as a conference paper at ICLR 2020
3.5 GRAPH-BASEE KERNEL DIRICHLET DISTRIBUTION ESTIMATION
To better learn the Dirichlet distribution from our Bayesian GNN framework, we proposed Graph- Based Kernel Dirichlet Distribution Estimation (GKDE). The key idea of GKDE is estimating prior Dirichlet distribution parameters for each node based on training nodes (see Figure (b)). And then, we leave prior Dirichlet distribution in the training process to learn two trends: (i) nodes with high vacuity (due to lack of evidence) will be shown far from training nodes; and (ii) nodes with high dissonance (due to conflicting evidence) will be shown in the class boundary.
Based SI., let each training node represent of evidence estimation for target node j from node by h(y;, dis(i, = [,,,...,/,, h hx] E [0, 1] and hy (yi, dis(i, j)) is obtained by:
evidence for its class label. Denote the contribution
y/ + k
hg (i)
(10)
oV ((dis(i,
y4 = k
d is the Gaussian kernel function to estimate the distribution effect between nodes and j. and dis(i, means the node distance (shortest path between nodes i and j). and a is the bandwidth parameter. The prior evidence estimation based GKDE: é, EisL hyy, dis(i,j)), and the prior Dirichlet distribution & êú + 1. During training process, we minimize the KL-divergence between model predictions of Dirichlet distribution and prior distribution: min (Di((i((a)) (I)))(())))
where g(dis(i.j) ov2n
3.6 TEACHER NETWORK FOR REFINED INFERENCE
Our key contribution is that the proposed Bayesian GNN model is capable of estimating various uncertainty types to predict existing GNNs. As one of the preferred features, the expected class probabilities generated by our Bayesian GNNs model should be consistent with the predicted class probabilities of the GNN model. In addition, our Bayesian GNN model is generative model and may not necessarily always outperform GNN models (i.e., discriminative models) for the task of node classification prediction when uncertainty-basee prediction is not fully benefited.
To refine the inference of our proposed model, we leverage the principles of Knowledge Distillation in DL (Hinton et al.) 2015). In particular, we consider our proposed model as distilled model and a deterministic GNN model as a teacher model, as shown in Figure I train our proposed model imitate the outputs of the teacher network on the class probabilities while minimizing the loss function of our proposed model. We observed that the modeling of data uncertainty in our proposed model provides useful information to further improve the accuracy of the deterministic GNN model. Therefore, we consider propagating the useful information back to the teacher model help train itself.
(c). The key idea is to
Let us denote Prob(y r, r) as the joint probability of class labels via a deterministic GNN model, where 33 refers to model parameters. The probability function Prob(y r: 7, g) estimated based on Eq. X using the variational parameters 7. We measure the closeness between Prob(y | r: B3) and Prob[y G)), with KL-divergence to minimized while minimizing their own loss functions based on the labeled nodes. This leads to solving the following optimization problem:
miny,3 [(v) + C(B) A (KL/Prob(s) (x7,9) || Prob(y r;; B)] KL(Dir())lDir(aa)
(11)
where L(B) is the loss function (i.e., cross entropy) of the deterministic GNN model and À is trade-off parameter. Our inference algorithm using backpropagation is detailed in the Appendix.
EXPERIMENTS
In this section, we describe our experimental settings and demonstrate the performance of our proposed model based semi-supervised node classification. For the performance comparison and analysis of our model and other existing counterparts, we demonstrate and analyze the obtained results in terms of the overall classification accuracy.
4.1 DATASETS
We use six datasets, including three citation network datasets (Sen et al. 2008) (i.e., Cora, Citeseer, Pubmed) and three new datasets (Shchur et al.) 2018) (i.e., Coauthor Physics, Amazon Computer, and Amazon Photo). We summarize the description and experimental setup of the used datasets in Table I For all the used datasets, we deal with undirected graphs with 20 training nodes for each
5

---

Under review as a conference paper at ICLR 2020
Table 1: Description of datasets and their experimental setup for the node classification prediction.
Cora
Citeseer
Pubmed
Co. Physies
Ama.Computer
Ama.Photo
#Nodes
2,708
3.327
19.717
34, 493
13, 381
487
#Edges 5,429
4,732
44,338
282, 455
259, 139
126, 530
#Classes
#Features
1,433
3,703
500
8,415
767
745
#Training
100
#Validation nodes
500
500
500
500
500
500
#Test nodes
1,000
1,000
1,000
1000
1,000
1000
category. We chose the same dataset splits as in (Yang et al. 2016) with additional validation node set of 500 labeled examples for the hyperparameter obtained from the citation datasets, and followed the same dataset splits in (Shchur al. 2018) for Coauthor Physics, Amazon Computer, and Amazon Photo datasets, for fair comparison.
4.2 COMPARING SCHEMES
We conduct the extensive comparative performance analysis based our proposed models and number of other state-of-the-art counterparts. Our proposed Subjective GNN models are: (1) S-GNN (Subjective GNN) with vacuity and dissonance estimation, which outputs subjective opinion (Dirichlet distribution) instead of softmax probability; (2) S-BGNN (Subjective Bayesian GNN), Subjective Graph Neural Network with Bayesian framework with multiple type uncertainty estimation; (3) S-BGNN-T (S- B-BGNN with Teacher network), help improve the expected class probability estimation; (4) S-BGNN-T-K (S- B-NN-N with GKDE), help improve the Dirichlet distribution estimation. Here
use two popular GNN models: GCN (Kipf & Welling 2016) and GAT (Veličković et al. 2018).
Our proposed models are compared against number of the state-of-the-ar counterparts. For evaluating three citation datasets (i.e., Cora, compared our mod- els with: (1) GCN (Kipf & Welling (016);;
Table 2: Semi-supervised node classification accuracy.
Cora
Citeseer
Pubmed
DeepWalk ICA
67.2
43.2
65.3
Citeseer, Pubmed),
75.1
69.1
73.9
Planctoid GCN
75.7
64.7
77.2
(2) GAT (Veličković al.] (2018); (3) non- parametric Bayesian GCNN (BGCNN) (Pal
81.5
70.3
79.0
Bayesian GCN
81.2 - 0.8
72.2
76.6
BGCNN
80.3 1: 0.6
72.6 1: 0.6
79.2 1: 0.5
al. 2019); (4) Bayesian GCN (Zhang al.
GCN-Drop
70.9
79.0
S-GCN
81.5 1.0
0.6
79.0 0.2
2019; ; (5) MC-dropout for Bayesian GNN
S-BGEN
81.2
7.6
79.0 0.2
(GoN-Drop. GAT-Drop) (Ryu af.]
2019);
S-BGCN-T
82.2 3.6
71.3.006
79.2 0.4
(6) skip-gram gram based graph embeddings (Deep-
S-BGCN-T-K
82.0 0.6 0.6
71.0 1: 0.6
79.3 0.3
Walk) (Perozzi et al. 2014);; (7) iterative classi-
GAT
83.0
72.5::0..7
79.0 1: 0.3
fication algorithm (ICA) (Lu & Getoor 2003);
GAT.Drop
79.0
S-GAT
72.6
79.0 0.3
and (8) Planetoid (Yang et al. (016)) We se-
S-BATT
82.9 0.7
72.4 1: 0.7
78.9
lected these for the comparison with mod- els based on (Veličković et al., (018) for fair comparison with the latest comparable models. Using Coauthor Physics, Amazon Computer, and Amazon Photo, we compared the perfor- mance of our models with that of GCN and GATgShchur et al., 2018), we can not show S- GAT (S-BSA (S-B due to memory limited, More details of model setup refer to the Appendix
S-BGAT-T
83.7 0.6
73.2 1 0.5
79,1 0.2
S-BGAT-T-K
83.8 0.7
73.0 + 0.7
79.1 + 0.2
Co.Physies
Computerr
Ama.Photo
GAT
92.5 + 0.9
78.0 + 19.0
85.7 + 20.3
GCN
92.8 1.0
82.6
91.2 1.2
GCN
93.0 0.8
79.7 + 1.3
91.6 1.2
GCN-Drop
S-GCN
93.1 0.8
78.9
- 1.6
90.4 + 1.4
S-BGCN
93.1 1 0.7
78.3
1.6
90.2
1.6
S-BGNN-
93.2 1 0.8
84.1
1.3
92.3
1.2
B-BCN-T-K
93.0 - 0.8
84.0
1.2
92.0 1 1.3
GCN' and GAT are implemented from (Shchur et al.
2018
4.3 EXPERIMENTAL RESULTS & ANALYSIS
In Table we summarized the mean percentage of classification accuracy with standard deviation of each model compared in this experiment. The results prove that our model achieves the best accuracy result across all datasets except Citeseer. To be specific, our proposed S-BGCN-T is able to improve over GCN by a margin of 0.7%, 1.2%, 0.2%, 0.2%, 4.5% and 0.7% on Cora, Citeseer, Pumbed, Coauthor Physics, Amazon Computer, and Amazon Photo, respectively. In addition, our proposed S-BGAT-T model improves 0.8% for both Cora and Citeseer datasets over GAT. Notice that S-BGNN-T even outperforms S-B S-BGNN particularly the Cora and Citeseer dataset (i.e., 1% 1.3% increase). These results prove that the teacher network can prevent overfitting, leading to a further improvement in classification prediction.

---

Under review as a conference paper at ICLR 2020
0.5
Foncon
Vacaly
Vicuty
Disponanoo
Dooe sance
0.0
Aluutot::
Abotoroo
0.0
Epictenia
Preceson
Tratwels
Crampy.
Popp
Rander
Vacalty
Alabort:
Epclonid
Recal
Recall
Recal
(a) PR curves on Cora
(b) PR curves on Citeseer
(c) PR curves on Pubmed
Figure 2: PR curves for Node Classification Prediction: S-BGCN-T-K is used to predict node classification where test nodes are selected based the low extent of uncertainty. The PR curves of all compared models can be found in the Appendix.
5 UNCERTAINTY EXPERIMENT AND ANALYSIS
In Section 4 we showed that our S-BGNN-T improves prediction performance. In this section, we study the effectiveness of prediction based on different types of uncertainty. We studied the different types of uncertainty-
node classification and -utto--distributionn in terms of the area under the ROC (AUROC) and Precision-Resed
(AUPR) curves in both experiments as in (Hendrycks & Gimpell 2016) for three citation network datasets. For the OOD detection, randomly selected 1-3 categories as OOD categories and trained the models only based on training nodes of the other categories. Due to the space constraint, we summarize the description of datasets and experimental setup for the OOD detection in the Appendix.
To better evaluate our multiple uncertainties, we compare our model with two baseline models: (1) GCN which uses GCN (Kipf & Welling 2016) with the softmax probability entropy measuring uncertainty; and (2) GCN-Drop, where one of the two uncertainty types (i.e., aleatoric, or epistemic
uncertainty) adapts Monte-Carlo Dropout (Gal & Ghahramani] (0166) into the GCN model (Ryu et al.
2019). In the OOD, we also consider Distributional uncertainty (Malinin & Gales 2018
5.1 QUALITY OF UNCERTAINTY METRICS
In Figure 2 we used S-BGCN-T-K to pre-
Table 3: Node classification prediction in AUPR.
dict node classification when test nodes are selected based on the lowest uncer- tainty for given type. First of all, all uncertainty types show decreasing pre- cision recall increases. This implies that all uncertainty types are to some ex- tent the indicators of prediction accuracy because low uncertainty increases predic- tion accuracy. In Figure 2 we can ob- serve almost 100% performance of pre- cision when recall is close to zero on Cora and over 95% on Pubmed. Further, the outperformance of Dissonance uncer- tainty is obvious among all. This indicates that low uncertainty with few conflicting evidence is the most critical factor to enhance classification prediction accuracy, compared to low extent of other uncertainty types. In addition, although epistemic uncertainty was very low, epistemic uncertainty performs the worst among all. This also indicates that epistemic uncertainty is not necessarily helpful to enhance prediction accuracy in semi-supervised node classification. Lastly, we found that vacuity is not as important as dissonance because accurate prediction is not necessarily dependent upon large amount of information, but is more affected by less conflicting (or more agreeing) evidence to support single class.
AUPR
Data
Model
Dis.
Al.
Ep.
S
90.4
95.4
92.6
88.0
93.4
S-BGCN-T
89.5
94.4
91.9
85.3
92.2
Cora
GCN-Drop.
92.7
90.0
93.6
GCN
94.1
S-BGCN-TKK
79.8
85.6
82.2
75.2
83.5
B.BCNN
80.0
85.6
82.4
76.6
83.6
Citeseer
GCN-Drop.
82,3
77.8
83.7
GCN
83.2
S-BGCN-T-K
85.6
90.9
88.9
86.0
89.3
Pubmed
S-BGCN-T
84.8
90.2
88.8
85.
OCN-Drop.
88.6
85.6
89.0
GCN
89.2
Va.: Vacuity, Dis.:
Dissonance, AL:: Alcatoric, Ep: Epistemic, En.: Entropy
In Table although all BGCN-T models with the five different uncertainty types do not necessarily outperform all the existing models (i.e.. GCN Entropy and variants of GCN-Drop.), the outperfor- mance of Dissonance is fairly impressive. This result confirmed that low uncertainty caused by dissonance is the key to maximize node classification prediction accuracy. When compare S-BGCN-T and S-BGCN-T-K, we found GKDE can only help improve performance little. To better understand different uncertainty types, we used (-SNE (Maaten & Hinton, 2008) to represent the computed
7

---

Under review as a conference paper at ICLR 2020
Figure 3: Graph embed- ding representations of the Cora dataset for classes and the extent of uncer- tainty: (a) shows the rep- resentation of seven differ- ent classes; (b) shows our model prediction: and (c)- (f) present the extent of uncertainty for respective uncertainty types, includ- ing vacuity, dissonance, aleatoric, epistemic.
In Ground Trush
@u Prediction
Vicurty
Diseoaance
feature representations of pre-trained S-BGCN-T-K model's first hidden layer on the Cora dataset Figure
5.2 DUITTF--DSTTIBBTION DETECTION
In this section, we discuss how different uncertainty types can prove the performance in the out- of-distribution (ODD) detection. In Table we considered uncertainties with 3 models for our performance comparison. Note that Distributional uncertainty is the the most recent model showing the best performance in the OOD detection. Across the three citation network datasets, particularly S- BGCN-T-K Vacuity showed significantly better performance, strikingly outperforming Distributional uncertainty. Notice that S-BGCN-T-K outperforms S-BGCN-T (i.e., 4% 7% increase), especially the improvement of vacuity. These result prove that the GKDE can improve the Dirichlet distribution estimation, leading to a better uncertainty estimation.
Table 4: AUROC and AUPR for the OOD detection.
Data
Model
AUROC
AUPR
Dis.
Al.
Ep.
D.En
En.
Vi
Dis.
AL
Ep.
D.Ei.
En.
S-BGCN-I-K
87.6
73.5
85.3
70.8
85.1
84.8
78.4
49.0
75.3
44.3
73.8
73.1
S-BGCN-T
83.5
71.7
Cora
GCN Gro.Drop.
81.9
70.5
80.9
69.7
44.2
67.2
OCN
80.7
66.9
S-BGCN-I-K
848
55.2
78.4
55.1
74.0
86.8
35.8
81.3
74.0
Citeseer
S-BGCN-T
78.6
59.6
73.9
56.1
75.1
69.3
79.8
57.4
76.4
57.8
78.3
69.3
Drop.
GCN
70.8
70.2
S-BGCN-T-K
74.6
67.9
71.8
59.2
69.7
72.2
69.6
52.9
63.6
44.0
64.8
56.5
Pubmed
S-BGCN-T
70.0
70.8
65.7
62.9
55.1
GCN-Drog
68.7
60.8
66.7
59.7
46.7
54.8
Va.: Vacuriy, Dis.: Dissomance, AL: Aleatoric, Ep:: Epistemic, D.Em: Differential Entropy, En.: Entropy
6 CONCLUSION
In this work, proposed Subjective Bayesian GNNs framework for uncertainty semi- supervised node classification and out-of--istribun (OOD) detection for GNNs. Our proposed framework provides effective, efficient way of predicting node classification and detecting OOD considering multiple uncertainty types. We leveraged the estimation various types of uncertainty from both DL and evidence/belief theory domains. In addition, We leveraged the Teacher network to help refine the classification probability and GKDE to accuractly estimate Dirichlet distribution.
The key findings from this study include:
For the overall classification prediction, our proposed S S-BGNN-T outperformed the competitive baselines on most datasets. The key role to improve the accuracy is Teacher Network.
For the node classification prediction considering various uncertainty types, we found that disso- nance (i.e., uncertainty derived from conflicting evidence) played a significant role improve classification prediction accuracy.
For the OOD detection, vacuity uncertainty played key role when S-BGCN-T-K is used to detect OOD. This means that less information and/or more randomness (or less predictability) enables detecting OOD more effectively. More impressively, GKDE can indeed help to estimate Dirichlet distribution accurately so that enhance the vacuity performance. Also vacuity outperformed the most recent counterpart, Distributional uncertainty.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Clarence W De Silva. Intelligent control: fuzzy logic applications. CRC press, 2018.
Dhivya Eswaran, Stephan Günnemann, and Christos Faloutsos. The power of certainty: A dirichlet- multinomial model for belief propagation. Proceedings of the 2017 SIAM International Confer- ence on Data Mining, pp. 144-152. SIAM, 2017.
Yarin Gal. Uncertainty in deep learning. University of Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with bernoulli approxi- mate variational inference. arXiv preprint / 1506.02/55. 02888, 2015.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In ICML, pp. 1050-1059, 2016.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pp. 249-256, 2010.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02116, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/ 2015.
Audun Jøsang. Subjective logic. Springer, 2016.
Audun Josang, Jin-Hee Cho, and Feng Chen. Uncertainty characteristies of subjective opinions. In FUSION, pp.
IEEE, 2018.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? NIPS, pp. 5574-5584. 2017.
Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla. Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. arXiv preprint arXiv:1511.0268% 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
Thomas Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 1oX:::160900200,, 2016.
Qing Lu and Lise Getoor. Link-based classification. In ICML, pp. 496-4096333 2003.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(79v):557.-2655 2008.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. arXiv preprint 1rXiv:1802.10001 2018.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In SIGIR, pp. 43-52. ACM, 2015.
Soumyasundar Pal, Florence Regol, and Mark Coates. Bayesian graph convolutional neural networks using non-parametric graph learning, arXiv preprint arXiv: 12132, 2019.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representa- tions. In KDD, pp. 701-710. ACM, 2014.
Seongok Ryu, Yongchan Kwon, and Woo Youn Kim. Uncertainty quantification of molecular property prediction with bayesian neural networks, arXiv preprint arXiv: 190300855 2019.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Ree Radd. Collective classification in network data. Al magazine, 29(3):93, 2008.

---

Under review as a conference paper at ICLR 2020
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In NIPS, pp. 3183-3193, 2018.
Kari Sentz, Scott Ferson, et al. Combination of evidence in Dempster-Shafer theory, volume 4015. Citeseer, 2002.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint orXiv:/8/100886 2018.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liö, and Yoshua Bengio. Graph Attention Networks. ICLR, 2018.
Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. arXiv preprint arXiv: 1603.088000. 2016.
Yingxue Zhang, Soumyasundar Pal, Mark Coates, and Deniz Ustebay. Bayesian graph convolutional neural networks for semi- supe-supereieed classification. arXiv preprint arXiv: 2018.
Yingxue Zhang, Soumyasundar Pal, Mark Coates, and Deniz Ustebay. Bayesian graph convolutional neural networks for semi-supervised classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5829-58295508 5836, 2019.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1
SOURCE CODE
For review purpose, the source code and datasets are accessible at https: /ww. dropbox com/ sh/ whwcssgshiiiiiccce 1umbSsss u BWww91ryk35gicggBBe..k...
DESCRIPTION OF DATASETS
Cora, Citeseer, and Pubmed (Sen et al. 2008): These are citation network datasets, where a network is directed network where node represents document and an edge is a citation link, meaning that there exists an edge when document cites document, or vice-versa with a direction. Each node's s feature vector contains a bag-of-wordd representation of document. For simplicity, we don't 't discriminate the direction links and treat citation links as undirected edges and construct a binary, symmetric adjacency matrix A. Each node is labeled with the class to which it belongs.
Coauthor Physics, Amazon Computers, and Amazon Photo (Shchur et al., 2018): Coauthor Physies is the dataset for co-authorship graphs based the Mierosoft Academic Graph from the KDD Cup 2016 [hallengd] In the graphs, a node is an author and an edge exists when two authors co-author paper. A node's features represent the keywords of its papers and the node's class label indicates its most active field of study. Amazon Computers and Amazon Photo are the segments of an Amazon co-purchase graph (McAuley et al., 2015), where node is good (i.e., product), an edge exists when two goods are frequently bought together. A node's features are bag-of-wort of-words representation of product reviews and the node's 's class label is the product category.
In semi-supervised node classification task,the training and test nodes are selected based on (Sen et al., 2008) for the citation network datasets and are randomly selected for the Coauthor Physies, Amazon Computers, and Amazon Photo.
A.2 EXPERIMENTAL SETUP FOR OUT-OF-DISTTRBUTION (OOD) DETECTION
For OOD detection, we summarize the experiment setup for the use of the three citation network datasets (i.e., Cora, Citeseer, and Pubmed) in Table 5 In this setting, we still focus on the semi- supervised node classification task, but only part of node categories are not using for training. Hence, suppose that model only outputs partial categories (as we don't know the OOD category). For example, Cora dataset, we train the model with 80 nodes (20 nodes for each category) with the predictions of 4 categories. Positive ratio is the ratio of -itto--distribttion nodes among all test nodes.
Dataset
Cora
Citeseer
Pubmed
#Number of training categories
3
2
#Training nodes
80
60
40
#Test nodes
1000
1000
1000
#Positive ratio
38%
55%
40.4%
Table 5: Description of datasets and their experimental setup for the OOD detection.
A.3 CALCULATION OF AUPR AND AUROC
For the calculation of precision, recall, TPR, and FPR, we select a certain & of nodes out of test nodes to label them as positive (correct) based the extent of uncertainty, the lowest uncertainty for classification prediction and the highest uncertainty for OOD detection. And the remaining test nodes (i.e.. 100 o %) are labeled as negative. Each test node's prediction is checked with its ground truth to derive AUPR and AUROC.
KDD Cup 2016 Dataset: Online Available httpst: azurewebs net/
11

---

Under review as a conference paper at ICLR 2020
TIME COMPLEXITY ANALYZE
BGCN has a similar time complexity with GCN while BGCN-T has the double complexity of GCN. In the revised paper, we will add a table showing Big-O complexity of all schemes considered. For a given network where [V] is the number of nodes, [E] is the number of edges, C' is the number of dimensions of the input feature vector for every node, and F is the number of features for the output layer, the complexity of the compared schemes are: O E CF) for GCN, O CF) for S-BCGN, ((IF for S-BCGB-T and S-BCGB-T-A T-K, O(V)CC E F) for GAT, and O for S-BGAT-T and S-BGAT-T-K.
A.4 MODEL SETUPS FOR SEMI-SUPERVISED NODE CLASSIFICATION
Our models are initialized using Glorot initialization Glorot & Bengio 2010) and trained to minimize loss using the Adam SGD optimizer (Kingma & Ba 2014). As our proposed models (i.e., S-BGCN- T-K, S-BGAT-T-K) need discriminative model to refine inference, we use standard GCN and GAT models as teacher networks for S-BGCN-T-K and S-BGAT-T-K, respectively. For the S-BGCN-T-K model, we use the early stopping strategy (Shchur et al.] 2018) on Coauthor Physics, Amazon Computer and Amazon Photo datasets while non-early stopping strategy is used in citation datasets (i.e., Cora, Citeseer and Pubmed). We set bandwidth = for all datasets in GKDE, and set trade off parameters À min(1, 1/200), (where is the index of current training epoch); and other hyperparameter configurations are summarized in Table The $ S-BGAT-T-K model has two dropout probabilities, which are dropout on features and dropout on attention coefficients, as showed in Table 7 We changed the dropout attention coefficients to 0.4 at the test stage and set trade off parameters À = min(1, t/50), using the same early stopping strategy (Veličković et al. 2018) 1). Note that lack of memory (we used one Titan X GPU, 12 GB memory), we could not obtain the result for GAT (also S-BGAT) on Coauthor Physics, Amazon Computer and Amazon Photo, which are very dense datasets.
For semi-supervised node classification, we use 50 random weight initialization for our models on Citation network datasets. For Coauthor Physics, Amazon Computer and Amazon Photo datasets, we report the result based on 10 random train/validation/tesl splits. In both effect of uncertainty on classification prediction accuracy and the OOD detection, we report the AUPR and AUROC results in percent averaged over 50 times of randomly chosen 1000 test nodes in all of test sets (except training or validation set) for all models tested on the citation datasets. For S-BGCN-T-KK model in these in Table except S-BGCN-T-K Epistemic
tasks, we use the same hyperparameter configurations using 20,000 epochs to obtain the best result. For baseline models, GCN-Drop. models use the same hyperparameters in Table to achieve the best performance, also using 20,000 training epochs for GCN-Drop. Epistemic. GCN Entropy uses the same hyperparameter configurations in (Kipf & Welling 2016
Cora
Citeseer
Pubmed
Co.Physics
Ama.Computer
Ama.Photo
Hidden units
16
16
16
64
64
64
Learning rate
0.01
0.01
0.01
0.01
0.01
0.01
Dropout
0.5
0.5
0.5
0.1
0.2
0.2
Le reg.strength
0,0005
0,0005
0,0005
0.001
0,0001
0.0001
Monte-Carle samples
500
500
500
100
100
100
Max epoch
200
200
200
100000
100000
100000
Table 6: Hyperparameter configurations of S-BGCN-T-K model.
12

---

Under review as a conference paper at ICLR 2020
Cora
Citeseer
Pubmed
Hidden units
64
64
64
Learning
0.01
0.01
0.01
Dropout
0.6/0.6
0.6/0.6
0.6/0.6
La reg.strength
0,0005
0,0005
0.001
Monte-Carle samples
100
100
100
Max epoch
100000
100000
100000
Table 7: Hyper- paraeepas of S-BGAT-T-K model.
A.5 ALGORITHM FOR OUR ALGORITHM
Algorithm 1: S-BGGN-T with jointly train for Teacher network
Input: EV r) and y1
Output: PvyL- uvy.
Set hyper-parametens:
Initialize the parameters 9: Calculate the prior Dirichlet distribution Dir(á);
repeat
Forward pass to compute a, Prob(p,(r; ②), Prob(y, 9) for E V;
Compute joint probability Prob(y r; 9), Prob(y|r;
Backward pass via the chain-rule the calculate the sub-gradient gradient: g = Ve((O)
Update parameters using step size via (6+11) = ( gg
f (= + 1:
11 until convergence
12 Calculate PVLL. UyyL
13 return Pvi. MyyL
Algorithm 2: S-BGNN-T-K with pre-train for Teacher network
Input: G (V,E, r) and Y1
Output: Pvy.. uvyr
Set hyper-parameter;;
Initialize the parameters S;
Calculate the prior Dirichlet distribution Dir(á);
Pre-train the Teacher Network get Prob(y
repeat
Forward pass to compute a. (p, for i E V;
Compute joint probability Prob(y r, 9);
Backward pass via the chain-rule the calculate the sub-gradient gradient: g = Ve((9)
10 Update parameters using step size via (é-1) ( ng
+++1. 1;
12 until convergence
13 Calculate PYLL- uyyL
14 return PVi. MYYL
B ADDITIONAL EXPERIMENT RESULTS
Further experiment have been run in addition to the uncertainty analysis in section 5. First, show the ablation experiment for each compents we proposed. Second, we show more uncertainty visualization result in network node classification for Citeseer dataset. To better understand the performance of uncertainty quality clearly for each uncertainty, we show the AUROC and AUPR curves for all models and uncertainties.
13

---

Under review as a conference paper at ICLR 2020
B.1 ABLATION EXPERIMENTS
We conducted an additional experiment to ensure the benefit of the teacher network. We anticipate that the graph kernel prior will improve the estimation accuracy of Dirichlet distribution. However, due to the space constraint, we didn't t show the classification results without using the graph kernel prior. In the revised version, we added a detailed ablation study in the revised paper in order to clearly demonstrate the contribution of the key technical components, including a teacher Network, Graph kernel Dirichlet Estimation (GKDE) and subjective Bayesian framework. The key findings obtained from this experiment (1) The teacher Network can further improve node classification accuracy (i.e., 0.2% 1.5% increase, as shown in Table 2); and (2) GKDE (graph kernel prior) using the uncertainty estimates can enhance OOD detection (i.e., 4% 7% increase, shown in Table 9).
Table 8: Ablation experiment for Node classification prediction in AUPR.
AUPR
Data
Model
Va.
Dis.
Al.
Fp.
En.
S-BGCN-T-K
90.4
95.4
92.6
88.0
93.4
Cora
S-BGCN-T
89.5
94.4
91.9
85.3
92.2
S-BOCN.
89.4
94.3
91.6
85.3
92.0
S GCG
89.4
94.4
S-BGCN-T-K
79.8
85.6
83.5
S-BGCN-T
80.0
85.6
82.4
76.6
83.6
Citesoer
S-BGCN.
79.7
85.2
82.2
76.2
83.2
S-GUN
79.8
85,3
S-BGCN-T-K
85.6
90.9
88.9
86.0
89.3
S-BGCN-T
84.8
89.3
Pubmed
S BGCN.
84.4
90.1
88.5
85.1
89.0
S-GCN
84.6
90.2
Va.: Vacuity, Dis.; Dissonance, Al.: Aleatoric, Ep: Rpistemic, Fi..; Eatropy
Table 9: Ablation exepriemnt on AUROC and AUPR for the OOD detection.
AUROC
AUPR
Data
Model
D.E..
En.
Va.
Ep.
D.En.
S-BGCN-T-K
87.6
75.5
85.5
70.8
85.1
84.8
78.4
49.0
75.3
44.5
73.8
73.1
S B
84.5
81.2
83.5
71.8
84.1
83.5
74.4
53.4
75.8
46.8
70.8
71.7
Cora
S-BGCN
84.1
81.4
83.3
71.9
84.0
83.3
74.1
53.8
75.4
47.0
70.3
71.6
S-GCN
84.2
81.2
83.9
74.2
53.9
70.6
B-BGCN-TKK
78.4
79.1
86.8
81.3
B
78.6
59.6
73.9
56.1
75.1
69.3
79.8
57.4
76.4
$7.8
78.3
69.3
Citeseer
S BBBGNN
78.5
59.7
73.2
56.5
75.4
69.1
79.6
57.8
76.0
57.9
78.1
69.0
S-GCN
78.6
60.0
75.7
79.9
57.8
78.4
S B-C
74.6
67.9
71.8
59.2
69.7
72.2
69.6
52.9
63.6
44.0
64.8
56.5
Pubmed
S-BGCN-T
71.8
70.0
68.0
70.8
65.7
53.9
61.8
46.0
62.9
55.1
S-BGCN
71.9
68.3
70.2
60.01
68.2
70.5
65.9
53.6
61.9
46.4
62.6
55.0
S-GCN
71.7
68.8
68.8
65.8
53.9
63.0
Va.: Vacuity, Dis.: Dissonance, AL.: Aleatoric, Ep.: Epistemic, D.Ei.: Differential Entropy, En.: Entropy
B.2 GRAPH EMBEDDING REPRESENTATIONS OF DIFFERENT UNCERTAINTY TYPES
To better understand different uncertainty types, we used t-SNE (t-Distributed Stochastic Neighbor Embedding (Maaten & Hinton 2008)) to represent the computed feature representations of a pre- trained BGCN-T model's first hidden layer on the Citeseer dataset.
Six Classes on Citeseer Dataset: In Figure 4 (a), node's color denotes a class on the Citeseer dataset where 6 different classes are shown in different colors. Figure (b) is our prediction result.
For Figures (c)-(f), the extent of uncertainty is presented where blue color refers to the lowest uncertainty (i.e., minimum uncertainty) while red color indicates the highest uncertainty (i.e., maximum uncertainty) based on the presented color bar. To examine the trends of the extent of uncertainty depending on either training nodes or test nodes, we draw training nodes as bigger circles than test nodes. Overall we notice that most training nodes (shown as bigger circles) have low uncertainty (i.e., blue), which is reasonable because the training nodes are the ones that are already observed. Now we discuss the extent of uncertainty under each uncertainty type.
Vacuity: In Figure (c), although most training nodes show low uncertainty, we observe majority of test nodes in the mid cluster show high uncertainty as appeared in red.
14

---

Under review as a conference paper at ICLR 2020
Max
(a) Citeseor Ground Truth
(b) Citeseor Prediction
(c) Citeseor Vacuity
Min
(d) Citeseer Dissonance
(c) CitescerAAlentrrie
(f) Citeseer Epistemic
Figure 4: Graph embedding representations of the Citeseer dataset for classes and the extent of uncertainty: (a) shows the representation of seven different classes, (b) shows our model prediction and (c)-(f) present the extent of uncertainty for respective uncertainty types, including vacuity, dissonance, and alcatoric uncertainty, respectively.
1.0
CGH
3.90
Dic Aualeo
0.55
2.35
Voaily
0.00
Predision
0:00
Dec cnbero
0.80
Vaady
080
0.5
0.5
(a) PR curves on Cora
(b) PR curves on Citeseer
(c) PR curves on Pubmed
Figure 5: PR curves of node classification prediction for all models and uncertainties.
Dissonance: Figure 4 (d), similar to vacuity, training nodes have low uncertainty. But unlike vacuity, test nodes much less uncertain. Recall that dissonance represents the degree of conflicting evidence (i.e., discrepancy between each class probability). However, in this dataset, we observe a fairly low level of dissonance and the obvious outperformance of Dissonance in node classification prediction.
Aleatoric uncertainty: In Figure 4 (e), lot of nodes show high uncertainty with larger than 0.5 except a small amount training nodes with low uncertainty. High aleatoric uncertainty positively affects, showing high performance in OOD detection.
Epistemic uncertainty: In Figure (f), most nodes show very low epistemic uncertainty because uncertainty derived from model parameters can disappear they trained well. Therefore, non- distinctive low uncertainty for most nodes do not help much to select good test nodes to improve performance in node classification.
15

---

Under review as a conference paper at ICLR 2020
Faroer Per Foleey Eico Cootei
Crop
Necten Dinp barigy Chac deral
Rocioe
Diix Lahlog
Precsion
Precision
Prectsion
04
Recall
Rocall
Recal
(a) PR curves on Cora
(b) PR curves on Citeseer
(c) PR curves on Pubmed
Figure 6: PR cuves of OOD detection for all models and uncertainties.
Poeilive
Posilvve
Posdive 04
Gon Coloer Depael
O04 Drmpp
Becoed Sycoes
62-
33
Vitaay
22
60
False Positive Rate
False Postive Rote
False Postlive Rate
(a) ROC curves on Cora
(b) ROC curves on Citeseer
(c) ROC curves on Pubmed
Figure 7: ROC curves of OOD detection for all models and uncertainties.
B.3 PR AND ROC CURVES
AUPRC for the OOD Detection: Figure the AUPRC for the OOD detection when S-BGCN- T-K is used to detect OOD in which test nodes are considered based on their high uncertainty level, given a different uncertainty type, such as vacuity, dissonance, aleatoric, epistemic, or entropy (or total uncertainty). Also to check the performance of the proposed models with a baseline model, we added S S-BGCN-T-K with test nodes randomly selected (i.e., Random).
Obviously, in Random baseline, precision was not sensitive to increasing recall while in S-BGCN-T-K (with test nodes being selected based on high uncertainty) precision decreases as recall increases. But although most S-BGCN-TA models with various uncertainty types used to select test nodes shows sensitive precision to increasing recall (i.e., proving uncertainty being an indicator of improving OOD detection), S-BGCN-T-K Epistemic even performed worse than the baseline (i.e., S-BGCN-T-K Random). This is because epistemic uncertainty cannot distinguish the flat Dirichlet distribution (a = (1, ..... 1)) from sharp Dirichlet distribution (a (10, ........ 10)), which leads to effective selection of test nodes for improving the performance in OOD detection. In addition, unlike AUPR in node classification prediction, which showed the best performance in S-BGCN-T-K Dissonance (see Figure S- S-BGCN-T-K Dissonance showed the second worst performance among the proposed S-BGCN-T-K models with other uncertainty types. This means that less conflicting information does not help OOD detection. On the other hand, overall we observe Vacuity performs the best among all while S-BGCN-T-K Entropy also performs fairly well as the third best. From this finding, we can claim that to improve OOD detection, more randomness with high aleatoric uncertainty and less information with high vacuity can help boost the accuracy of the OOD detection. Although the uncertainty level observed from aleatoric uncertainty and entropy was quite similar, the performance in OOD detection not necessarily similar, as shown in Figures(6 (b) and (c) on Citeseer and Pubmed. The reason is that BCGN-T Aleatoric provides test nodes with more distinctive uncertainty levels while BCGN-T Entropy docsn't. 't. This is because BCGN-T Entropy combines the alcatoric and epistemic uncertainty where epistemic uncertainty is mostly highly low, ultimately leading to poor distinctions of nodes based on different uncertainty levels.
AUROC for the OOD Detection: First, we investigated the performance of our proposed S-BGCN- T-K models when test nodes are selected based on seven different criteria (i.e., uncertainties and random). Like AUPR in Figure 5 based on S-BGCN-T-K, we considered baseline by selecting
16

---

Under review as a conference paper at ICLR 2020
test nodes randomly while five different uncertainty types used to select test nodes based on the order of high uncertainty. For AUROC in Figure 7 we observed much better performance in most S-BGCN-T-K models with all uncertainty types except epistemic uncertainty. Although epistemic uncertainty is known to be effective to improve OOD detection (Kendall & Gal 2017) in computer vision applications, our result showed fairly poor performance compared to the other uncertainty types are used. This is because our experiment is conducted with a very small of training nodes (i.e., 3% on Cora, 2% on Citeseer, 0.2% on Pubmed) which is highly challenging observe high performance particularly with epistemic uncertainty. Recall that we used 200 epochs to train nodes for all models except BCGN-T Epistemic which was trained with 20,000 epochs. In this experiment, even S-BGCN-T-K Vacuity performed the best although S-BGCN-T-K Dissonance, S-BGCN-T-K Aleatoric, or S-BGCN-T-K Entropy performs comparably. But on Citeseer and Pubmed datasets, we also observed relatively low performance with S- S-BGCN-T-K Dissonance. This finding is also well aligned with what we observed in Table 4 (in paper). S-BGCN-T-K Vacuity performs the best on all three datesets. Obviously S-BGCN-T-K Vacuity outperform S-BGCN-T-E Distributional in OOD detection.
B.4 ANALYZE FOR EPISTEMIC IN OOD DETECTION
In OOD detection, epistemic uncertainty performed the worst because it cannot distinguish the flat Dirichlet distribution (a (1,..., 1)) from sharp Dirichlet distribution (a (10,..., 10)), resulting poor performance in OOD detection. Unlike AUPR in node classification prediction with outperformance in -BGCN---K Dissonance (see Figure 2 S-BGCN-T-K Dissonance showed the second worst performance among the proposed S-BGCN-T-K models with other uncertainty types. This implies that less conclusive belief mass does not help OOD detection.
Although epistemic uncertainty is known to be effective to improve OOD detection (Kendall & Gal 2017) in computer vision applications, our result showed fairly poor performance compared the case other uncertainty types are used. This is because our experiment is conducted with very small of training nodes (i.e., 3% on Cora, 2% on Citeseer, 0.2% on Pubmed) which highly challenging to observe high performance particularly with epistemic uncertainty.
C DERIVATIONS FOR UNCERTAINTY MEASURES AND KL DIVERGENCE
This appendix provides the derivations and shows how calculate the uncertainty measures discussed in section 3 for BGCN. Additionally, it describes how to calculate the joint probability, Dirichlet parameters and KL-divergence between Prob(y) rr (3) and Prob(y)r; (r,
C.I UNCERTAINTY MEASURES
Vacuity uncertainty of Bayesian Graph neural networks for node d:
Vacuity/p.] = (prob(01
K
= Ep))(0))))
K/
OAkk
k=1
Ey(0)
K/
Olik
M
K
K
(m) ik
a (m)
= =f(r (())) g(m) ~ (0)
M
m==
k=
l
17

---

Under review as a conference paper at ICLR 2020
Dissonance uncertainty of Bayesian Graph neural networks for node i:
[w(bb)]
Disso.[p.]
(0) (bbb))
M
@ g(0) w(bb))
M
and
K bik Znj=l biz Bal bose)
((b;)
ΣK Σj=1 bij
where the relative mass balance between pair of belief masses by and bis is expressed by (al(bi d1x) lhi bakl (bxj + bx))-
Aleatoric uncertainty of Bayesian Graph neural networks for node i, followed (Malinin & Gales 2018
Aleatoric
( [H(y, |r 0)] Egge) Ml(y, r; 0)]
H [[y: Ir: (( g(m) ~ ((0)
M
M Prob(y: r; (())) ((m))log log (Probly: j|r; ((m))) ((m) (0))
Epistemic uncertainty of Bayesian Graph neural networks for node i, followed GGal 2016
Epistemic/p.] = H Epra (rr 0)] - ( [ (r: 0)
~ H [( 0)] Ey(or [ 0)]
H Prob(y (obb |r; ((m))]
M Σ ((;
1
g(m) g())
C.2 JOINT PROBABILITY
At the test stage, we infer the joint probability by:
/
p(y|r: 9)
Prob(y)p)Prob((pl:: P)oro)() dpdd
Prob(y)p)Prob((p:::0ppg ()(())))
((m)))
(lm) ((m) ~ (())
M
N
Prob(y, Helly..p,PPPer Prob(p (r; ((m) dp, ((m) q())
M
Prob(y, h.ll..pp.Prrro. Prob(
(r: )dm)), ((m) (())
M
M N
Σ
II
Prob(y, P. Dir(p,aa (m) )dp. (m) = f(r, g(m)), 0 (m)) (()
M
m=Li=1
18

---

Under review as a conference paper at ICLR 2020
where the posterior over class label will be given by the mean of the Dirichlet:
(m)
Prob(y: p) Prob(y: plp,)) |r; ((m) ddp;
ip
K
(m)
k= ik
The probabilistic form for specific node by using marginal probability, Prob(y,(r:()
Σ Prob(y |r: G)
E
Prob(y,lp,) Prob(p, rr Probb(O)))) 1=1
Σ
Prob (obbyy,l,,)Proolee (r:
Σ
Prob(y, Proo(pp ((m)) ddp, ((m) q(0)
/=1
II /=1
Prob y p, Prob((p rr: @(()) )lm))) ydp;
@(m) g(0)
II Prob Prob(y,lr, (()) (r:byy,)r0mm))) @ (
Prob(y,lp, Prob: jr: (m) )m)) @(m) (
specifically for probability of label p,
(m)
Prob(y: = g) M
1
Ap
a [(r, 0(m)) (mm) (()
(m)
km
ik
C.3 KL-DIVERGENCE
KL-divergence between Prob(y 9) and Prob(y rr; 9)
Prob(y rr G)
(L|Prob(y (r:)) ir; B)]]
Eprot
log
Prob(y |r; 3)
Prob(y, 9r:
Bprob(yin
log
Prob(y, |r;
Prob(y, 9)
Eproh(yIns)))
log
Prob(y,
iml
Prob(y, G)
Eprohly
( (n)))
log
Prob(y,Ir::3)
N
K
Prob(y: j|r; 9)
Prob( = j]r: 9)
log
Prob(y; = j 3)
The KL divergence between two Dirichlet distributions Dir(a) and Dir(ā) can be obtained in closed form follows:
K
K
KL(Dir())[Dir(a) = In + Σ
[II((a)))) - InT(())) + > (a. - - (s))
where S >K 0g and $ = DK à.
19


Reviews:
[{"review_id": "S1eQwBJesH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the effect of various uncertainties in bayesian GNNs. They study various uncertainty models such as leatoric, epistemic, vacuity and dissonance.  They study multiple uncertainty types in both\ndeep learning (DL) and belief/evidence theory domains. They treat the predictions\nof a Bayesian GNN (BGNN) as nodes’ multinomial subjective opinions in a graph\nbased on Dirichlet distributions where each belief mass is a belief probability of\neach class. By collecting evidence from the given labels of training nodes, the\nBGNN model is designed for accurately predicting probabilities of each class and\ndetecting out-of-distribution. They show that their proposed Bayesian GNN outperforms state-of-the-art counterparts in terms of the accuracy of node classification prediction and out-of-distribution detection based on six real\nnetwork datasets.\n\nThe main issue with this paper is the motivation of the framework. While the technical details seem correct, the writing is unclear in many places in the manuscript. From an experimental perspective, ablation experiments need to be added. Also, the main motivation of this work is that it is modeling the uncertainty in model predictions. However, I do not see this verified anywhere experimentally.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlMjvBy5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to model various uncertainty measures in Graph Convolutional Networks (GCN) by Bayesian MC Dropout. Compared to existing Bayesian GCN methods, this work stands out in two aspects: 1) in terms of prediction, it considers multiple uncertainty measures including aleatoric, epistemic, vacuity and dissonance (see paper for definitions); 2) in terms of generative modeling, the GCN first predicts the parameters of a Dirichlet distribution, and then the class probabilities are sampled from the Dirichlet. Training/inference roughly follows MC Dropout, with two additional priors/teachers: 1) the prediction task is guided by a deterministic teacher network (via KL(model || teacher)), and 2) the Dirichlet parameters are guided by a kernel-based prior (via KL(model || prior)). Experiments on six datasets showed superior performance in terms of the end prediction task, as well as better uncertainty modeling in terms of out-of-distribution detection.\n\nPros:\n1. This model considers uncertainties in multiple dimensions.\n2. Better predictive performance and OOD detection ability on 6 real datasets.\n\nCons:\n1. Adding an additional layer of the Dirichlet is not well motivated.\n2. Needs ablation studies on modeling choices, e.g., how much does the graph kernel prior help.\n3. In table 2, needs to compare with traditional Bayesian GCN (such as Zhang et al 2018). Besides, is GCN-Drop in table 4 Zhang et al 2018?\n4. In table 2, seems that knowledge distillation helps, since GCN gets similar performance to BGCN, but BGCN-T outperforms. A natural baseline is GCN w/ knowledge distillation.\n5. In table 4, the baseline GCN-Drop gets better uncertainty estimates than the proposed approach in terms of aleatoric, epistemic, entropy which can be evaluated for GCN-Drop. I wonder if it is possible to develop a measure for vacuity and dissonance for GCN-Drop as well. But anyway this table contradicts the motivation for adding an additional layer of the Dirichlet.\n\nMinor details:\n1. Is teacher jointly trained with the model or is it pretrained? And what's teacher's network architecture? Is it much larger? I cannot understand \"choose two graph convolutional layers in which the first layer is 16hidden units for GCN and 64 hidden units for GAT, and removed a softmax layer\".\n\nOverall, this is a very technical work, but the modeling choices need to be better justified/motivated compared to existing works on GCN with MC Dropout. I am inclined to reject this paper.\n\n\n-------updates after reading rebuttal----\nThanks for the response! I have no further questions.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeJX0KpYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a Bayesian graph neural network framework for node classification. The proposed models outperformed the baselines in six node classification tasks. The main contribution is to evaluate various uncertainty measures for the uncertainty analysis of Bayesian graph neural networks. The authors show that vacuity and aleatoric measure are important to detect out-of-distribution and the dissonance uncertainty plays a key role for improving performance.\n\n** Introduction/Conclusion/Contribution\n- Applying MC dropout to graph neural networks is a bit old idea [1-5], it cannot be considered as the “contribution” of this study.\n- The authors should emphasize more on what’s the advancements over existing studies.\n\n** Methodology\n- My understanding is the authors proposed to use multiple uncertainties (vacuity, dissonance, aleatoric, epstemic). Vacuity and dissonance measures can also be implemented with other Bayesian graph neural networks.\n- Ablation study is need to demonstrate the usefulness of each component in the objective function (equation 11). \n- Can “BGAT-T” be considered as the proposed method? It doesn't use the proposed GNN framework but just an extension of original GAT with MC dropout and knowledge distillation.\n\n** Experiments\n- Please provide the details of hyperparameter settings, e.g. optimizer, batch size, learning rates, …\n- The authors should perform experiments with varying numbers of labels per category (e.g. 5, 10, 20), because different methods show different behaviors under label scarcity.\n- There are a number of recently published papers that address node classification based on Bayesian graph neural networks, e.g., see references. They should be used as baselines if available.\n- I think the authors should evaluate BGAT-T with Co.Physics, Ama.Computer, and Ama.Photo datasets, because this method was overall superior to other methods on the first three datasets. \n- Experiments are insufficient in the uncertainty analysis (section 5). The authors can evaluate the performance of BGAT as well as other Bayesian graph neural networks in terms of uncertainty quantification performance. \n- Also, in order to evaluate uncertainty quantification performance, I would suggest to look at the trade-off between classification accuracy and classification rejection based on the uncertainty, like accuracy-rejection curve in [6] \n\n** misc\n- Check the following sentence of subsection 3.6. “our key contribution is that the proposed Bayesian GNN model is capable of estimating various uncertainty types to predict existing GNNs.”\n\nReferences\n[1] Ryu, S., Kwon, Y., & Kim, W. Y. (2019). Uncertainty quantification of molecular property prediction with Bayesian neural networks. arXiv preprint arXiv:1903.08375.\n[2] Zhang, Y., & Lee, A. A. (2019). Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning. arXiv preprint arXiv:1902.00925.\n[3] Pal, S., Regol, F. L. O. R. E. N. C. E., & Coates, M. A. R. K. (2019). Bayesian graph convolutional neural networks using non-parametric graph learning. In Representation Learning on Graphs and Manifolds Workshop, Int. Conf. Learning Representations.\n[4] Akita, H., Nakago, K., Komatsu, T., Sugawara, Y., Maeda, S. I., Baba, Y., & Kashima, H. (2018, December). Bayesgrad: Explaining predictions of graph convolutional networks. In International Conference on Neural Information Processing (pp. 81-92). Springer, Cham.\n[5] Zhang, Y., Pal, S., Coates, M., & Ustebay, D. (2019, July). Bayesian graph convolutional neural networks for semi-supervised classification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, pp. 5829-5836).\n[6] Nadeem, M. S. A., Zucker, J. D., & Hanczar, B. (2009, March). Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option. In Machine Learning in Systems Biology (pp. 65-81).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a way to produce uncertainty measures in graph neural networks. However, the reviewers find that the methods proposed lack novelty and are incremental additions to prior work.
Decision: Reject