Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration
Authors: Xiao Wang
Hongrui Liu
Chuan Shi
Cheng Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Neural Networks
Confidence Calibration

Abstract: Despite Graph Neural Networks (GNNs) have achieved remarkable accuracy, whether the results are trustworthy is still unexplored. Previous studies suggest that many modern neural networks are over-confident on the predictions, however, surprisingly, we discover that GNNs are primarily in the opposite direction, i.e., GNNs are under-confident. Therefore, the confidence calibration for GNNs is highly desired. In this paper, we propose a novel trustworthy GNN model by designing a topology-aware post-hoc calibration function. Specifically, we first verify that the confidence distribution in a graph has homophily property, and this finding inspires us to design a calibration GNN model (CaGCN) to learn the calibration function. CaGCN is able to obtain a unique transformation from logits of GNNs to the calibrated confidence for each node, meanwhile, such transformation is able to preserve the order between classes, satisfying the accuracy-preserving property. Moreover, we apply the calibration GNN to self-training framework, showing that more trustworthy pseudo labels can be obtained with the calibrated confidence and further improve the performance. Extensive experiments demonstrate the effectiveness of our proposed model in terms of both calibration and accuracy.

Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration
Xiao Wang, Hongrui Liu, Chuan Shi; Cheng Yang
School of Computer Science (National Pilot Software Engineering School) Beijing University of Posts and Telecommunications Beijing, China
(xiaowang, liuhongrui, shichuan, yangcheng)@bupt. edu.cn
Abstract
Despite Graph Neural Networks (GNNs) have achieved remarkable accuracy, whether the results are trustworthy is still unexplored. Previous studies suggest that many modern neural networks are over-confident on the predictions, however, surprisingly, we discover that GNNs are primarily in the opposite direction, i.c., GNNs are under-confident. Therefore, the confidence calibration for GNNs highly desired. In this paper, we propose novel trustworthy GNN model by designing topology-aware post-hoc calibration function. Specifically, we first verify that the confidence distribution in a graph has homophily property, and this finding inspires us to design a calibration GNN model (CaGCN) to learn the calibration function. CaGCN is able to obtain unique transformation from logits of GNNs to the calibrated confidence for each node, meanwhile, such transformation is able to preserve the order between classes, satisfying the accuracy- preserving property. Moreover, we apply the calibration GNN to self-training framework, showing that more trustworthy pseudo labels can be obtained with the calibrated confidence and further improve the performance. Extensive experiments demonstrate the effectiveness of our proposed model in terms of both calibration and accuracy.
Introduction
Graphs are ubiquitous in the real world, including social networks, e-commerce networks, traffic networks, and so on. Recently, Graph Neural Networks (GNNs), which are able effectively learn the node representations based on the message- message-passing manner, have attracted considerable attention 34 To date, GNNs have been applied various applications and achieved remarkable accuracy, e.g., node classification [6]] 33), link prediction 41
in dealing with graph data [16] 33 39
44 15
and graph classification
However, it is well established that a model with good accuracy is not the only goal, but trustworthy model is highly desired in many applications, especially safety-critical fields [T] Usually, trustworthy model implies that it should know when it is likely to be incorrect, in other words, the probability, i.e., the confidence, associated with the predicted class label should reflect its ground truth correctness likelihood 12]. For example, in the scene of autonomous driving, the system will adopt the prediction given by the model only when the model has high confidence for its prediction. Otherwise, the decision becision-making power will be returned to the driver or the system adopts other safer strategies. Recently, the confidence calibration has attracted considerable attention in deep learning the
12 40 19. which reveals that many modern neural network models are over-confident predictions, i.e., the prediction accuracy lower than its confidence. However, it has not been studied
'Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

in GNNs on the semi-supervised scenario, which gives rise to one fundamental question: will the current GNNs follow the same over-confident property as other neural networks? A well-informed answer can help us better understand GNNs and enable GNNs to be applied to various areas in more reliable manner.
As the first contribution of this study, we present experiments assessing the relationship between the confidence and the accuracy of Graph Convolutional Networks (GCNs) [[6] and Graph Attention Net- works (GAT) [3 in the node classification task (more details can be seen in Section 2 respectively, Surprisingly, we discover that existing GNNs are far distant from being well-calibrated, and more importantly, GNNs tend to be
in their predictions, which is very different from other modern deep learning models that are often over-confident [9]. GNNs being under-confident means that many predictions are distributed in the low- cow-confidence range, and therefore, fewer predic- tions are available for safety-critical applications. Once the weakness is identified, another natural question is: how can we calibrate the confidence on predictions given by GNNs so as to make them more trustworthy?
Essentially, the confidence calibration is to calibrate the outputs (also known logits) of original models (e.g., GNNs), therefore, a straightforward manner is to employ temperature scaling (TS) 12) OP-families [25] to learn calibration function using a held-out dataset in a post-hoc way. However, when being applied to graphs, they all ignore the effect of topology, which will inevitably make mistakes during calibration. For example, considering that the logits of two nodes a and b are the same, but node a is similar to its neighbors while node is not. Apparently, the predictions of GCNs for a should be more confident than b, while the traditional calibration methods, e.g., TS, will learn the same confidence for a and b, because it does not consider the effect of topology.
Moreover, most of them explore calibration functions only in the linear space 122 18 while it is well known that non-linear space contains more complex function transformation which is able to calibrate networks with complicated landscapes well. Even if some works have explored the non-linear space such as Matrix Scaling [1], they generally degrade the classification accuracy of the original classifier, while good accuracy is still a basic requirement by many applications.
In this paper, introduce topology awaree post-hoc calibration method for GNNs. Specifically, for the logits given by the original classification GNNs, we employ another calibration GCN (CaGCN) to propagate confidence, naturally enabling that the confidence of topologically adjacent nodes becomes similar. CaGCN learns a unique temperature for each node for temperature scaling, thus preserving the accuracy of the original classification GCN. In addition, based on our finding that large numbers of high-accuracy predictions are distributed in the low-confidence range, we design a calibrated self-training model CaGCN-st in which the confidence is firstly calibrated then used to generate pseudo labels with high confidence. The contributions of this paper are three-fold:
We study the trustworthy problem of GNNs, and discover one unique characteristic of GNNs, i.e., the predictions made by GNNs are usually under-confident.
We propose a novel trustworthy GNN model based on the confidence calibration. Our proposed calibration function has three features: topology-aware non-linear, and accuracy-preserving We further design a calibrated self-training GNN model, which can effectively utilize the predictions with high confidence.
Extensive experiments demonstrate the effectiveness of our proposed models in terms of both calibration and accuracy.
2 Notation and Preliminary Study
In this paper; we focus on the calibration of semi-supervised node classification in an undirected attributed graph G = EV,E) with the adjacent matrix A E VxN and the node feature matrix X = [x1,..., Xw] J V is set of nodes and E C V x V is set of edges between nodes. == is the number of nodes. Here we give the definition of perfect calibration of GNNs as follows:
Definition 1. Given random variables A, X, Y C KK} and GNN model fe where θ is the learnable parameters, for node i with label y; E Y, z, - fe(xxA) = [.., 2i,x] is the output of GNNs (i.e., the prediction probability), and &; arg max, Zia and p; = max Zi, are the prediction and the confidence respectively. Then we define fo to be perfectly calibrated as:
P(5; p. p) p. Pp 10.
(I)
2

---

Cora 20
Uscal. Clesseer GCH
Uncal Pubmed- 20
Uscal Cara 20-GCN
Outbetts Expected
Outarts
Expacted
Exuected
Expeciad
9.
0.1
0.8
3.0
0.2
0.8
0.
0.3
0.4
2.0
0.6
0.
6.
1.0
Uncal, Cora- 20- GAT
Uncal,-Cora-
10
Uncal- Cheener 20- GAT
3.0
Uncal Pubrred 1ubmed-20--- GAT
3.
Uncal, Caall--CooFul---- 20 GAT
Expected
Expacted
Expected
Eqactedd
0.6
0.6
0.8
0.4
0.
Figure Reliability diagrams for GCN (top) and GAT (bottom) without confidence calibration. The diagram is expected to plot an identity function of accuracy with respect to confidence. Any deviation from a perfectly diagonal (i.e., the difference between blue and red histogram) represents the miscalibration.
Uncal. Cora 20- GCN
Uncal Cheseer 20 GCN
Uncal- Pubmedd 20- GCN
Uscal. Cora-ull-----aaa 20-GCH
InCorrec
InCornet
InComeet
Intoret
16
0.4
0.6
0.3
0.6
0.8
1.0
0.5
06
6.7
0.8
1.0
0.8
6.2
0.6
0.8
1.3
Uncal- Coro 20 GAT
Uncol: Publedd 28 GAT
Uncal CoraFull
20 G&T
16
InCorrec
Correct
InComrec
InComent
10
0.2
0.4
0.5
0.5
0.8
5.7
16.8
1.00
00.2
1.0
Figure 2: Confidence distribution before calibration.
According to Definition GNN is perfectly calibrated only when the confidence i is exactly equal to the true probability of getting a correct prediction for every node.
Next, we take two representative GNNs (GCN [66] and GAT (33) as examples to analyze whether they are perfectly calibrated. Specifically, we apply GCN and GAT to four widely used datasets Cora 29.. Citeseer [29], Pubmed [29], CoraFull [3], and examine whether their results satisfy Definition
To provide more results, we select three label rates for training set (i.e., 20, 40, 60 labeled nodes per class). All the experimental settings follow [16] 33. Since the true probability cannot be exactly known, we take an approximate way to evaluate the calibration as in [1]. particular, we first partition the [] range of confidence into 20 equal bins and then we group the nodes into corresponding bins according to their confidence. After that we calculate the average accuracy of each bin. We expect the average accuracy is equal to the average confidence of each bin, which means the model is approximately perfectly calibrated. For example, if the average confidence nodes in the bin [0.95, 1.0] is 0.96, and then the classification accuracy in this bin should be 96%.
We illustrate the results of label rate being 20 in Fig. Tusing Reliability Diagrams (3) here, where the x-axis is the confidence in 20 bins of equal size and y-axis is the average accuracy in each bin. The blue represents the classification accuracy of GCN and GAT while the red is our expectation. More results of label rate being 40, 60 and other GNN models can be seen in Fig. 8 Fig. Fig. [2 Fig. 13 and Fig. [4 in the appendix. We can see that in all the datasets, the average accuracy of most bins higher than the average confidence. In other words, these GNNs actually achieve remarkable
3

---

performance, but they all output low confidence, i.e., the GNNs are usually under-confident. Please note that this phenomenon of GNNs very different from other modern neural networks, which are generally known to be over-confident [12 19]. Moreover, as shown in Fig. 2 we also visualize the confidence distribution of test nodes, where the x-axis is the confidence and y-axis is the density [28]. The histogram height multiplied by the width is equal to the frequency. The blue represents the confidence distribution correct predictions while the yellow represents that of incorrect predictions. More results of label rate being 40 and 60 can be seen in Fig. I0 and Fig. I the appendix. We can see that a large quantity of correct predictions are distributed in the low confidence range. The results above indicate that the current GNNs are far from perfect calibration, leading to unreliable confidence.
Confidence Calibration on GCNs
In this section, we propose our method to calibrate current GNNs. Given A and X, for a l-layer GCN [6]. the output of the GCN before the softmax layer can be obtained by:
V Ao - Ao(AXW )
)w()
= MNN"
(2)
where w()) is the weight matrix of 1-th layer in GCN and o(-) is the activation function. For each node i E [1....N ,N). our goal is to learn a calibration function which is fed with v; (often known as the logit of node i) and outputs calibrated confidence using held-out dataset in a post-hoc way. The
calibration function should satisfy three points below: (1) taking the network topology into account (2) non-linear (3) preserving the classification accuracy of the GCNs.
3.1 CaGCN: GCNs as Calibration Function
Table 1: Summary of total variation con-
fidence before and after calibration (Bold: best). Uncal. is short for uncalibrated and
TS is short for temperature scaling.
GCN
Dataset
Uncal.
TS
Ours
Cora
240.267
172.346
164.651
Figure 3: The illustration of the confidence propagation. Different colors indicate differ- classes.
Citeseer
128.
112.212
108.684
Pubmed
1299.33
1266.68
1113.41
CoraFull
6014.32
4698.20
4500.30
We assume that the ground-truth confidence distribution in graph has homophily property, i.e., the confidence of neighboured nodes given by well-calibrated models should be similar, and thus we conduct an experiment to verify this. We employ the classic temperature scaling method [12] as our calibration function and use the total variation [27] of confidence as our evaluation, which sums the difference of confidence between all the neighboured nodes. We compare the total variation of confidence before and after confidence calibration, where the results are shown in Table [ We can find that the total variation of confidence does decrease after temperature scaling, which verifies our assumption. This inspires us that if a GCN model is well-calibrated, then the confidence between neighbors should be more similar than before.
To this end, we find that GCN itself can play the role of calibration function that meets above require- ment since GCN is able propagate node features along the network topology and smooth similar information between neighboured nodes. Therefore, we can employ another l-layer GCN (CaGCN) as our calibration function propagate the confidence along the network topology. Specifically, given the output of the classification GCN, the logit v'; and confidence Pi for node after calibration can be obtained by:
V (o(A:.A) 2 W = |V IVN',
(3)
(VM.) GSM (Vi, P. max Zi,k.
where (VM((',) (v/,.) = Σn exp(Viy) is the softmax operation. Then the total variation of confidence will surely become lower and the original classification GCN will be calibrated. Please note that although
exp(v i..)

---

temperature scaling can be directly applied here, compared with GCN, it does not take the network topology into account, which may cause mistakes mentioned in Section I Moreover, temperature scaling only employs a linear transformation, and GCN is able to learn a non-linear calibration function.
For a comprchensive understanding of confidence propagation, we make a detailed and visible illustration here. As shown in Fig. [ the logits of two nodes a and b are the same, but node is similar to its neighbors while node b is not, Apparently, the predictions of GCNs for should be more confident than b. Suppose that a, and their neighbours are under-confident based on the observation above. If we continue to propagate their logits along the topology using another GCN, the logits of and its neighbors will tend be the same. Therefore, if one or more of these nodes are calibrated during the calibration process, all of them will be calibrated as well. The confidence is propagated in this way. On the other hand, looking at another node b, it as difficult even for manual classification as it is for GCNs. Consequently, the confidence of b should stay still even be lower. However, it will become higher because of the influence from a if we use the traditional calibration method without considering the network topology. Instead, when the network topology is taken into account, the logit will be averaged by its neighboured and each dimension tends to 1/K. will be correctly calibrated when other nodes in the same situation are well-calibrated.
3.2 The Accuracy-Preserte
Property
Until now, we have proposed non-linear calibration model CaGCN which can take the network topology into account, but the accuracy-preserving property cannot be satisfied. To address this problem, we firstly study the general pecrracy-reservvin calibration function.
Proposition 1. Let h IR* + RRK be a calibration function, s: ::R R be a I-D function and = [vi.:". vix]]" be the logit of node i. The calibration function h preserves the classification accuracy of the original model if is a strictly isotonic function and satisfies:
h h(v))
((v..)......(Vvxx)))) Vi (1.
(4)
Proof We set Vi,1 V1.2 ... < ViA without loss of generality. Since 1s....)...... .......sssvv... s(vxk) shares the
same order with v; as a result of the strictly isotonicity of s, the order between classes of the logit va unchanged, hence the accuracy of the prediction is preserved.
Temperature scaling 12] is the simplest accuracy-presert
calibration method using a scalar parameter f called temperature for all classes. Given the logit of node i, the confidence of the prediction is û; = max; OSM (t > 0). temperature scaling, h(v:) [v.1/t,...,,,, /t] is the
calibration function and s(x) = x/1 is the strictly isotonic function.
However, we can find that temperature scaling (TS) 40 only performs the same linear transformation for all the nodes using the same r. As mentioned in Eq. 3 we propose to use CaGCN as our calibration function, while CaGCN is generally not isotonic, i.c., the order between classes of v, and v is not the same, implying that after calibration by CaGCN, the accuracy of original GCN cannot be preserved. Instead, here we propose an improved CaGCN. Given the output of the classification GCN, we firstly use l-layer GCN to learn a unique temperature for each node i, then get a calibrated logit v'; by transforming its original logit v; using I; in temperature scaling way, and finally obtain calibrated confidence Pi as follows:
t= t= (Ao(- Ao(AVW w())) (w(()) [h, IN] TT] 0, Vi (1, ,NJ),
(5)
V1: = ((v...) [v.1/44,.. VL,K /4] 2 - (oSM (eww(V/2)) (/11),, (.),,...... OsM (V,,))) ,A, max 2iA0
where t; E is a scalar greater than zero and o (x) - log(1 + exp(x)) is an element wise softplus activation [ The model proposed in Eq. does not change the order between classes of v: and v'r implying that the accuracy original GCN is preserved. Compared Eq. 5 with Eq. 3 we can find that Bq. | makes the same transformation all the dimensions of v,, which will limit the learnable calibration function space. However, we will prove that actually Eq. 5 is the same with the model proposed in Eq. on confidence calibration using the Proposition 2 Considering that for any logit v;- expectation is in fact that the calibration model can output any confidence p; E (k. 1). Please note that P; 2 1 R. or the prediction will be changed. Since Eq. has no limitation on the learnt calibration model, its output P can take any value from & to 1. Therefore, if we can prove the output P; in Eq. can also traverse the interval (t) 1) for any vi. the equality between Eq. 3 and Eq 5 can be proved.
5

---

Proposition 2. Given the original logit v; = [V...... : v of node i, assume Vij not approaching
infinity for each j E ,K). The calibrated confidence Pi in Eq. can traverse the interval ( 1) for node i.
Proof We set Vi,1 Vi. ...>> >> ViA without loss of generality. For any I2 with the assumption of v; not approaching infinity, we have that
lim P = lim exp(van/ti) lim
exp((vin1-1ap) (v
exp((vi--vv)) (6)
1-0 10 Sy-rexp(visft 1-0 exp((x - v;2)/41)
and
lim exp(v.1/t))
lim P 1++000
expeex((vi//t) K
Obviously, both (vM(()))) and vi/4; are continuous, thus OSM (vre//) is continuous. Therefore, P; = max 4. = max; Osw(Vvxd (Vax/t;) can traverse the interval (1/K,1).
The assumption about V, is easy to be satisfied since the L2-norm in GCN drives the weight matrix W approaching zero matrix and each element in node feature matrix X is not infinity. Therefore, based on Eq. 2 each element Vi, in V cannot approach infinity. From Proposition we know that for any
v, there exactly exists such unique temperature f; that Po can take any value from I/K to In other
words, the model can be perfectly calibrated.
3.3 Optimization Objective
Since NLL loss I0] can decomposed into calibration loss and refinement loss [2I], minimizing
NLL loss benefits for confidence calibration. Therefore, we employ the NLL loss as our objective function with an additional regularization term. We use the prediction probability z, E & in Eq. 5 to calculate the NLL loss. Denote the K-class one-hot label for node yi [y,,,- .Y.x and suppose the size of the validation set is DDvat]. Then the NLL loss over all validation nodes is represented C,n where:
Dual K
Lill = Σ
Σ (islog(Zan).
(8)
Due to the under-confidence of GCNs, our goal is increase the confidence of correct predictions while decreasing that of incorrect predictions. Considering that for incorrect predictions. the NLL loss cannot directly reduce their confidence. therefore, we design a regularization term for NLL loss as follows:
loor Σ
line
Leal
(cor) Zim
+
27,5
(cor)
Z
(inc)
Zi,s
(inc)
(9)
where |cor| and line] are the number of nodes correctly and incorrectly predicted and 2i.m and ži,, are the max and submax prediction probability. Intuitively, the confidence of incorrect predictions is decreased by reducing the gap between the max and the submax value of z; and vice versa. Combining Laul and Lests we have the following overall objective function:
== Lmll + L
(10)
where A is the parameter the regularization term. With the guide of labeled data, can optimize CaGCN via back propagation and learn the calibrated confidence. The overall framework of CaGCN shown in Fig.
4 Self-training with Confidence Calibration
Here we propose practical application of confidence calibration to improve the performance of self-training in GCNs. Self-traie trfinini predict the labels for unlabeled data, and then add them to the training set, as to achieve better performance, When applying self-training to GCN, we firstly obtain the predictions 9; and the confidence P given by GCN and then add the most confident nodes to the training set with pseudo labels y based on Å: We continue to train until convergence. However, existing self-training methods perform not as expected with higher label rates 30) Considering the under-confidence of existing GCNs, motivated by (26), we argue that the perfo-ncrf of

---

Classification GCN
(lugits)
Training
Validation
Sufimas(V)
Test set
Unlaheled mode
CaGCN
Suflmas(V")
Figure 4: The overall framework of CaGCN. Solid lines represent that we can backpropagate gradient here while dashed lines represent we cannot. We firstly train a classification GCN using the training set to obtain the logit V of all the nodes. Then we feed V to CaGCN to get the temperature t and transform using t into V'. Finally, the loss can obtained using V' after softmax according to Eq. 10 and CaGCN can be optimized with the guide of the validation set.
Table 2: ECE (M=20) on different models and citation networks of various label rate (L/C) with and without calibration. Uncal. represents the uncalibrated model, (-) denotes this method cannot converge to meaningful result and bold denotes the best result, the subscript of each result refers to the standard deviation (x 10 while the superscript refers to the results of paired t-test for 0.05 level and ** for 0.01 level).
GCN
GAT
Dataset
L/C
Uncal.
MS
CaGCN
Uneal.
MS
CaGCN
0.1347..3
0.04883.5
0.04145.7
0.04016.7
0.1558g.9
0.07179.8
0.05449.4
0.0450
Cora
40
0.11344.7
0.04177.2
0.03724.6
0.04075.4
0. 0.1340s.4
0.04857.7
0.04916.0
0.03655
0.09374.9 60
0.03555.4
0.03646.1
0.03764.4
0.1201
0.03936.1
0.04115.3
0.031355
20
0, 0,1448
0,06418.7
0,06443.7
0.05951.2
0.15345.0
0.09168.7
0.06339.8
0.05726.
Citeseer
0.09577.7
0.06014.2
0.05385.7
0.05455.5
0.1252s.7
0.07973.1
0.0590,.4
0.05325
60
0.08066.4
0.0559s.0
0.05216.4
0.05463.4
0. 10900.
0.06487.1
0.05199.1
0.05257.6
20
0.05867.7
0.05413.8
0.04764.2
0.0405
0.08353.1
0.06564.6
0.05013.7
0.03565",
Pubmed
40
0.0444..5
0.04466.3
0.04366.
0.04022.0
0.08694.6
0.06586.5
0.05396.u
0.0308::4
60
0.0445..7
0.03676.0
0.03186.4
0.03114.8
0.09934.1
0.06696.3
0.04835.7
O 0.03085:2
20
0.19866.1
0.10136.1
0.0776
0.21193.6
0.11015.1
0. 0.0788...
CoraFull
40
0.2321
0.111765
0.0701
0.24384.2
0.1133g3
0.07382
0,23374.0
0.09813.8
0.076832
0.249718
0.11335.2
0.0849
existing self-training methods originals from large numbers of high-accuracy predictions distributing causing that they cannot be added to the training set.
in low-confidence intervals shown in Fig. 2
Consequently, design self-training model CaGCN-st where confidence is firstly calibrated then employed to generate pseudo labels for unlabeled nodes. Specifically, given an unlabeled dataset Du and labeled dataset D,. which has been divided into three parts D Dval and Dres, we firstly train a classification GCN using Drrain to get the logit of each node. Then all the logits will be fed into a CaGCN to train and we get a calibrated confidence for each node. It should be noted that instead of Dyal we still employ Drrain to train our CaGCN, After that, the most confident predictions of Dy will be adopted as the pseudo labels according threshold and added to the label set. The D is enlarged in this way. The process above will be repeated stages until convergence. Please note that our classification GCN and CaGCN are re-initialized each stage.
5 Experiments
In this section, we evaluate the performance of CaGCN on confidence calibration and CaGCN-st on self-training respectively. We choose the commonly used citation networks Cora (29) Citeseer [2)].
Pubmed (29] and CoraFull B for evaluation, and more detailed descriptions are in Appendix B
7

---

Table 3: Node classification accuracy and the standard deviation on GCN and its self-training variants.
Methods
Dataset
L/C
Orig.
St.
Ct.
Union
Inter.
TS-st
CaGCN-st
20
81. 63.24
82.270.33
81.51o.30
81.850.68
81.41o.29
82.680.20
83.11
Cora
40
83. 0.226
83. 83.590.34
83.660.25
83.330.41
83.380.33
84 84.440.35
84.370.38
60
84. 84.44
84.98n.32
84.630.31
85.030.30
84.880.18
85.600.24
85.790.27
20
71.64 .640.32
73. 73.240.44
74.220.29
74. 44..80
72.250.45
74.20
74.90%.40
Citeseer
40
72.25a.32
74.70
12.12
74.790.3
73.660.32
75.620.19
75,48p,50
60
73. 73.20g.35
75.08n.29
1..2
75.530.30
75.230.23
75.870.24
76.43%.20
20
79.5 39.3
80.32
79.670.32
81.120.29
79.590.29
80.95o.18
81.16
Pubmed
40
80.65
82.20n.32
81.620.40
81.840.23
82.28 82.28n.39 80.466.55
83 83.08
60
83.380.34
83.35p.28
83.400.36
83.320.35
83.31o.17
83. 83.260.39
84.47 84.47
20
60.450.43
60.870.28
60. 12..2
60.520.35
61.010.53
61.730.41
62.19 0.49
CoraFull
40
65. 35.77
65.83n.45
64.22o.35
64.339.42
65. 65.840.37
66. 10.60
66.30p.31
60
66.52..25
66.62n.30
66.649.29
66.780.29
66.820.32
66.950.45
67. 0..40
5.1
Confidence Calibration Evaluation
Baselines. Since our CaGCN is a general calibration model for GNNs, here we choose GCN [16 and GAT 33 as our classification models. For comparison, we choose the classic post-hoc calibration
methods temperature scaling (TS) 112 and matrix scaling with off-diagonal regularization (MS) [18] as our baselines.
Experimental settings. For the base model GCN and GAT, i..., the uncalibrated model, we follow parameters suggested by [16] and [3 and further carefully tune them to get optimal performance. For the post-hoc calibration technique, we follow the official implementation [12 18] For our CaGCN, we train a two-layer GCN with the hidden layer dimension be 16. We set 0.5 for all datasets, weight decay to be 5e-3 for Cora, Citeseer, Pubmed and 0.03 for CoraFull. Other parameters of CaGCN follows [16]. We evaluate the performance of confidence calibration by ECE [22] NLL [10 and Brier Score (BS) 44, which we expect are smaller, and we set the bin number M 20 for ECE (more details can be seen Appendix (A). For all methods, we randomly run 10 times and report the average results. More detailed experimental settings can be seen Appendix B
Table 2 reports calibration results evaluated by ECE (more results on NLL and Brier Score are in Appendix C.I.p We have the following observations: (1) Compared with uncalibrated models and other baselines, CaGCN is statistically significantly better at the * 0.05 level and ** 0.01 level. (2) The ECE values on uncalibrated models are generally the highest, implying that GCN and GAT are poorly calibrated. (3) MS behaves badly on datasets with many classes, e.g.. CoraFull. This is because the number of parameters for matrix scaling scales quadratically with the number of classes while the size of the validation set keeps unchanged. Therefore, it will over-fit to the small validation set when dataset has a great number of classes. However, CaGCN does not have this problem.
Results.
Additional analysis. In Section we visualize the under-confidence problem of existing GNNs using reliability diagrams. Here we utilize the same visualization method to make a comparison before and after confidence calibration. As shown in Fig. [ Fig. 9 Fig. an Fig. I in the appendix, we can find that the confidence is well-calibrated after calibration.
5.2
Classification Evaluation of Self-Training
Baselines. Since self-training can be applied to any models, here we choose GCN and GAT as our base models, i.e., the original models (Orig.) without self-training, and we choose self-training (St.), co-training (Ct.), Union, Intersection (Inter.) methods proposed in (20) for comparison, which are commonly used as the baselines in self-training. Furthermore, we employ TS as the confidence calibration function in CaGCN-st as another baseline and we denote it by TS-st.
Experimental settings. We set the learning rate Ir 0.001 for CaGCN-st and train our CaGCN-st 200 epochs for Cora, 150 epochs for Citeseer, 100 epochs for Pubmed and 500 epochs for CoraFull. We set the threshold t {0.8. 0.85,.5550. 0.9, 0.95, 0.99) and the maximum number of stage s 10. As 0.8.8.855...90999.
8

---

for baselines, all the parameters follow (20] and we further carefully tune them to get optimal performance. For all methods, we randomly run 10 times and report the average results.
Results. Table summarizes the node classification accuracy on GCN and its self-training variants. More results GAT can be seen in Appendix C.2 We have the following observation: (1) CaGCN-st consistently outperforms all the baselines on all the datasets and label rates at the 0,05 level. (2) Compared with the base model, self-training methods generally achieve better results, which proves their effecttiveness (3) Self-training methods with confidence calibration (i.e., TS-st and CaGCN-st) have better performance, which implies that confidence calibration scales more correct predictions to the high confidence range while keeps incorrect predictions basically unchanged, which we believe is beneficial for self-training
Ablation study. CaGCN-st generates pseudo labels based the calibrated confidence. Here we study the effectiveness of the confi-
Table 4: Abaltion study on self-training
dence calibration function CaGCN in CaGCN- st. We propose variant GCN-st of CaGCN- st, where CaGCN removed from CaGCN-st while other parts are kept unchanged. All the experimental settings of GCN-st are the same as CaGCN-st. We report the results in Table 4 and we can observe that CaGCN-st consistently outperforms GCN-st on all the datasets, implying that self-training with cal- ibrated confidence can generate more correct pseudo labels.
GCN
GAT
Dataset
L/C
GCN-st CaGCN-st
GCN-st
CaGCN-st
20
82.28
83.11
84.08
84.08
Cora
40
84.10
84.37
85,50
85,63
60
85.16
85.79
85.57
86.26
20
74.13
74.90
73.73
74.34
Citeseer
40
75.28
75.48
75.07
75.62
60
75.85
76.43
75.13
76.08
20
81.01
81.16
80.34
81.17
Pubmed
40
82.90
83.08
82.75
83.47
60
83.44
84.47
83,46
83,95
20
61.32
62.19
62.09
65.46
CoraFull
40
65.96
66.30
65.92
66.86
Additional analysis. We also investigate the changing trends of accuracy with respect to the threshold t in CaGCN-st in Appendix C.2
60
66.43
67.60
66.54
67.45
and study why GCNs are poorly calibrated Appendix
D
Related Work
Graph Neural Networks. Modern GCNs mimics CNNs to learn the local and global structural patterns of graphs through designed convolution and readout functions. [5] generalizes CNNs to graph signal based the spectrum of graph Laplacian. ChebNet (7] uses Chebyshev polynomials to approximate the K-order localized graph filters and GCN [6 further employs the 1-order simplifica-
33 utilizes attention mechanisms to adaptively learn aggregation weights. GraphSAGE [3 uses various ways pooling for aggregation. [20] introduces self-training to GCNs and 30 proposes multi-stage self-supervised (M3S) self-training algorithm of GCNs.
tion of the Chebyshev filter. GAT
Both 200 and 33 focus on the few-shot learning and neither has ever explored self-training with higher label rates GCNs. More works GNNs can be found in surveys [36] 43), however, the best of our knowledge, current GNNs have not considered the confidence calibration.
Confidence Calibration. Confidence calibration has been studied for a long time in CV and NLP [3] (223]2251444 14 37 40) 42). [12] discovers modern neural networks are poorly calibrated and study factors influencing calibration. Platt scaling [44] is simple post-hoc calibration method for binary models, which transforms the logit using scalar parameters. Temperature scaling is the simplest multi-class extension of Platt scaling and matrix and vector scaling another two extensions of platt scaling. [40] proposes Mix-n-Match calibration strategies which mix parameter methods with non-parameter methods. (25] explores the non-linear space for post-hoc calibration function using
neural network. Moreover, (32 points out GNNs can be miscalibrated in the supervised scenario and mainly focus miscalibration originated from the imbalanced class distribution. However, none of them have considered the confidence calibration in GNNs on the common semi- supervise scenario.
7 Conclusion
Current efforts on advancing GNNs mostly focus on classification accuracy. However, when deploying GNNs to real-world applications, especially safety-critical fields, whether the results of GNNs are

---

trustworthy is another important factor that cannot be neglected. In this paper, we study the confidence calibration problem in GNNs and discover existing GNNs are under-confident on their predictions. To solve this problem, we propose novel trustworthy GNN model CaGCN which respects the homophily property of confidence in GNNs and preserves the classification accuracy. Moreover, we propose a novel self-training method CaGCN-st where confidence is first calibrated by CaGCN and then used to generate pseudo labels. Extensive experiments demonstrate the effectiveness of our proposed model in terms of both calibration and accuracy.
An interesting direction for future work is to extend CaGCN to other graph tasks, but more studies need to be conducted. We take the link prediction as an example, where we can regard the link prediction as a binary classification problem and the output as the confidence. Considering that the ground-trute truth confidence distribution for nodes should have the homophily property as is shown in Section 3.1 edges are likely to have the same property as well. As result, we can employ CaGCN to propagate the confidence between edges by regarding the edges as the nodes. However, more exploration still needs to be conducted for the homophily property of edges.
Broader impact. Current efforts on advancing GNNs mostly focus on classification accuracy. However, when deploying GNNs to real-world applications, especially safety-critical fields, whether the results of GNNs are trustworthy is another important factor. The demands for a trustworthy model are universal and extensive such as in the field of disease prediction [1], traffic states prediction 6 and object detection [1] for autonomous driving, where estimating the true probability of getting correct prediction is necessary. We take the disease prediction [I] as an example, where GNNs are utilized to encode the information of different symptoms, users and diseases. In this scenario, accurately and comprehensively predicting diseases at an early stage will help patients receive prevention treatments in a timely manner. Otherwise, the misdiagnosis and missed diagnosis will endanger the health of patients. Therefore, a trustworthy model is urgently needed. Our CaGCN can make a trustworthy prediction based on its confidence, and as a result, decrease the risk of misdiagnosis and missed diagnosis. We hope our work can provide insights for future improvements in tackling the trustworthiness problem in other saftey-critical fields.
Limitations. One potential issue of this work is that it provides a limited explanation to the under- confidence problem. We advocate peer researchers to look into this, making GNNs more reliable in different domains. Other than that, since this work is mostly on the discovery of the confidence calibration problem in GNNs and the theoretical aspect of improving calibration, we do not foresee any direct negative impacts on the society.
Acknowledgments and Disclosure of Funding
This work is supported in part by the National Natural Science Foundation of China (No. 62172052.
No. U20B2045, U1936104, 61772082, 61 6020299, 62002029).
References
[1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv: 06565, 2016.
[2] Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond frwfrreuency information in graph convolutional networks. arXiv preprint arXiv:2101.00797, 2021.
[3] Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsu- pervised inductive learning via ranking, arXiv preprint arXiv:1707.03815, 2017.
[4] Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
[5] Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv: 13 /2266203 2013.
[6] Zhiyong Cui, Kristian Henrickson, Ruimin Ke, and Yinhai Wang. Traffic graph convolutional recurrent neural network: A deep learning framework for network-scale traffic learning and forecasting. IEEE Transactions Intelligent Transportation Systems, 21(11):4883- 4894,, 2019.
10

---

[7] Michaël Defferrard, Xavier Bresson, and Piere Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. arXiv preprint arXiv:1606.008 09375, 2016.
[8] Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incor- porating second-order functional knowledge for better option pricing. Advances neural information processing systems, pages 472-478, 2001.
[9] Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. fair comparison of graph neural networks for graph classification. arXiv preprint arXiv:1912.0989,, 2019.
[10] Jerome Friedman, Trevor Hastie, Robert Tibshirani, et al. The elements of statistical learning. volume 1. Springer series in statistics New York, 2001.
[11] Jiayuan Gu, Han Hu, Liwei Wang, Yichen Wei, and Jifeng Dai. Learning region features for object detection. In Proceedings of the european conference on computer vision (ECCV), pages 381-395, 2018.
[12] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinherger. On calibration of modern neural networks. International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[13] William I. Hamilton, Rex Ying. and Jure Leskovec. Inductive representation learning large graphs. arXiv preprint vrXiv:170..02216: 2017.
[14] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification. In 9th International Conference on Learning Representations, ICLR 2021.
[15] Weihua Hu, Matthias Fey, Marinka Zitmik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint XxXxx:00050006877 2020.
[16] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 169.09760 2016.
[17] Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In 7th International Conference on Learning Representations, ICLR 2019.
[18] Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Hao Song, Peter Flach, et al. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration. arXiv preprint arXiv: 12656, 2019.
[19] Aviral Kumar, Sunita Sarawagi. and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. International Conference on Machine Learning. pages 22144. PMLR, 2018
[20] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for supi-supe learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 32, 2018.
[21] Allan H Murphy, A new vector partition of the probability score. Journal of Applied Meteorology and Climatology, 12(4):595-600 1973.
[22] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[23] Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625 632, 2005.
[24] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
11

---

[25] Amir Rahimi, Amirreza Shaban, Ching-An Cheng, Richard Hartley, and Byron Boots. Intra order-preserving functions for calibration of multi-class neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[26] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty- pseudo-label selection framework for semi-supervised learning. XrXiv preprint arXiv:2101.06322, 2021.
[27] Stanistaw Saks. Theory of the integral. 1937.
[28] David W Scott. Multivariate density estimation: theory, practice, and visualization. John Wiley & Sons, 2015.
[29] Prithviraj Sen, Galileo Namata, Mustafa Bilgie, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 29(3):93 93, 2008.
[30] Ke Sun, Zhouchen Lin, and Zhanxing Zhu. Multi-stage self superviee learning for graph con- volutional networks on graphs with few labeled nodes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5892-5899, 2020.
[31] Zhenchao Sun, Hongzhi Yin, Hongxu Chen, Tong Chen, Lizhen Cui, and Fan Yang. Disease prediction via graph neural networks. IEEE Journal of Biomedical and Health Informatics, 25 (3):818 12.86 2020.
[32] Leonardo Teixeira, Brian Jalaian, and Bruno Ribeiro. Are graph neural networks miscalibrated? arXiv preprini arXiv: 090.99959909990 2019.
[33] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 17/0. 10903, 2017.
[34] Xiao Wang, Meiqi Zhu, Deyu Bo, Peng Cui, Chuan Shi, and Jian Pei. Am-gen: Adaptive multi- channel graph convolutional networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Duta Mining, pages 1243-1253, 2020.
[35] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning. pages 6861-6871 PMLR, 2019.
[36] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Yu Philip. comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 2020.
[37] Chen Xing, Sercan Arik, Zizhao Zhang, and Tomas Pfister. Distance-based learning from errors for confidence calibration. arXiv preprint arXiv: 192.0119/..0 2019.
[38] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations, ICLR 2019.
[39] Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[40] Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. International Conference on Machine Learning, pages 1 11117-11122 128. PMLR, 2020.
[41] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. arXiv preprint 2018.
[42] Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. arXiv preprint arXiv:2010. /2783, 2020.
[43] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020.
[44] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Be- yond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33, 2020.
12


Reviews:
[{"review_id": "q2f2Yq57qbD", "reviewer": "Reviewer_KZsj", "summary": "This paper proposed a confidence calibration method for graph neural networks (GNNs) for semi-supervised node classification problems. Based on the existing researches such as temperature scaling, the authors attempt to evaluate the confidence problem of GNNs and find a unique phenomenon that GNNs including GCN and GAT are less confident in their predictions which seem to be constant to other neural networks. From this observation, they proposed a confidence calibration procedure being aware of the topology of a graph structure and prove the accuracy-preserving property of their method. They also provided an effective optimization procedure for training their objective. Intensive experiments demonstrated the improvements of their proposed method with regard ECE and classification accuracy.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written paper. The main question \"will the current GNNs follow the same over-confident property as other neural networks?\" of this paper is clear and interesting for not only me but other researchers. I enjoyed reading it. \n\nStrong points:\n1. The authors firstly tackled the confidence calibration problem of the GNNs and report under-confident phenomena of GNN that have not been reported in the context of the confidence calibration of other neural networks.\n2. To fix the under-confidence problem of GNN, they proposed a new calibration GNN model using an idea based on the homophily property. Although its structure is a bit simple and having no significant difference from the existing GCN, they proved the accuracy-preserving property that assures that no decline is caused by their CaGCN.\n3. Intensive experiments showed better performances of CaGCN from other confidence calibration methods such as TS and MS.\n\nWeak points:\n1. In l.131 of p. 4, they employed the total variation to assess the confidence distribution and argued that \"We can find that the total\nvariation of confidence does decrease after temperature scaling, which verifies our assumption.\" It is bit confusing for me. I suppose that if a GCN is well-calibrated, its confidence distribution is also calibrated. And thus a TS operation may not able to decrease the TV. Could you clarify your arguments more?\n2. In Table 3, the authors report the node classification accuracy for four datasets. Although the accuracy-preserving property is proved in the aforementioned section, I cannot understand why accuracy of the TS-st and CaGCN-st were improved from the baseline. Is it caused by the calibration procedure or just the increasing number of parameters and layers of baselines?\n3. In the context of graph neural networks, node classification is one of the fashionable problems but is a sub-class of the GNN task. Is it possible to enlarge the coverage of their proposed method to the other GNN tasks?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "Bh_Nf_xmvd", "reviewer": "Reviewer_TDBE", "summary": "This paper analyzes the GNN prediction confidence, and reveals that the prediction confidence made by GNN is lower than the accuracy derived from the prediction confidence values, i.e., GNNs are under-confident.  This is different from the previously analyzed modern neural network models, which are often over-confident on the prediction (the prediction accuracy is lower than its confidence). To make the confidence values usable to reflect the true prediction confidence in safety-critical applications, a confidence calibration mechanism is proposed to make the confidence value be exactly equal to the true probability of getting a correct prediction for every node. The calibration follows the classic temperature scaling method, which has been used in the calibration of modern neural networks (ref [10]). ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is a novel study on the confidence calibration of GNN models, given that the previous work focused on the calibration of modern neural networks (ref [10]).  However, the paper has to improve on several aspects. First, the description of the model has unclarified facts. Second, the evaluation should be conducted with statistical test to show the effectiveness of the proposed solution. Last, some writing inconsistency should be addressed.  More details can be found below.\n\nIn line 175, “, the order between classes of v_i and v’_i is not the same, implying that after calibration by CaGCN, the accuracy of original GCN cannot be preserved. ” But softmax is a monotonic function. The order between classes of v_i and v’_i should be the same.\n\nComparing to the classic temperature scaling method in the calibration of modern neural networks (ref [10]), the calibration in Eq(5) introduce different temperature t_i for different nodes. This is an interesting idea. As shown in figure 4, the weights W in Eq(5) are shared with the weights in Eq(3) in GCN for getting output V. In this way, t_i is in fact from t_i = \\sigma+(v_i).  However, v_i is a vector. A softplus activation function applied on a vector v_i cannot result in a scalar t_i. Clarification is needed. \n\nRegarding the evaluation results, Table 3 reports the node classification accuracy and the variance (over 10 independent runs) of different models. The variance is quite large and the accuracy difference is small. Statistical test should be done to show if there is a significant improvement between CaGCN-st and baselines.  Table 2 doesn’t show the variance of ECE of different models. Given the small difference between CaGCN and other baselines, it would be necessary to present the variance and run statistical test as well.\n\nWriting to correct: \n1.\tIn Definition 1,  label index in Y is from 0 to K.  But z_i is a vector with elements from 1 to K (should be from 0 to K). Please check also in many other places, the index is sometimes from 1 to K, and sometimes from 0 to K.\n2.\tSame for the index of node i. The index below Eq (2) is from 0 to N, for the total N nodes above Definition 1.  In Eq (2), i is indexed rom 1 to N. But in some other places, i is indexed from 0 to N.\n3.  a l-layer GCN    etc\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "2-HKnmZwhmW", "reviewer": "Reviewer_4YpW", "summary": "This paper aims to propose a novel graph neural network model that is trustworthy via confidence calibration. Graph neural networks (GNNs) attract many researchers’ attention, which has achieved excellent achievements. However, can we trust the predicted results of GNNs? Inspired by previous research works showing that model deep neural networks are over-confident with predictions, the authors study the confidence calibration of GNNs. Through many experiments and analyses, they find that the traditional GNNs are under-confident. This observation is different from previous research works related to the confidence calibration of deep neural networks. Thus, how to do the confidence calibration for GNNs is important and highly desired. The authors propose a new calibration GNN model (CaGCN) to learn the calibration function for improving the confidence of GNNs. Specifically, they first validate the confidence distribution in a graph owns homophily characteristics. Then, they transform logits of GNNs to the calibrated confidence of each node. Owing to GNNs is used to deliver confidential information among different nodes during the transformation process, the process preserves the order between classes. Moreover, they apply the CaGCN to a self-training framework, which generates trustworthy pseudo labels for unlabeled data to improve model performance. Finally, the authors conduct extensive experiments to validate the effectiveness from two perspectives: calibration and accuracy.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Graph neural networks (GNNs) are popular in many domains. But can we trust the predicted result of GNNs? This research problem is interesting and useful. If we have strong and trustworthy GNNs, we can apply these GNNs to many real-world applications, especially safety-critical fields. Thus, from this perspective, we can find that the research topic of this paper is novel and important.\n\nIn addition, from the presentation perspective, this paper is well-organized and readers can follow the logic of the paper easily. Specifically, this paper has two steps: (1) finding the under-confident of GNNs, (2) proposing a calibrated GNNs (CaGNN) to make the predicted results more trustworthy. For the first step, the authors find the under-confident of GNNs via experiments, visualizations, and analyses. For the second step, the authors provide detailed explanations.\n\nMoreover, from the experimental perspective, there are two advantages in this paper: (1) the authors conduct extensive experimental results to validate the effectiveness of their framework. Through these experiments, we find that the authors’ framework is not only more stable but also more accurate compared with other baseline models; (2) for each experiment, the authors provide detailed experimental settings including parameter setting, dataset, and so on, which is beneficial for other readers to reproduce experimental results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are some limitations in this paper: (1). The authors think that the predictive performance of GNNs is under-confident via analyzing the predictive performance of GNN and GAT. But I think the authors should test more graph neural networks to validate the conclusion. (2). The authors utilize GCNs to implement the confidence propagation part. They think the GCNs can preserve topology information and non-linear relations in graphs. But I think the motivation is weak. The authors should provide stronger reasons to support their model choice. For instance, can we utilize other non-linear graph models to capture these characteristics? (3). Figure 3 cannot reflect the confidence propagation, I suggest the authors replot this figure. (4). The authors can provide more application scenarios to highlight the social impact of the research.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores an interesting problem that lies at the intersection of two hot topics: graph neural networks and the reliability of NNs.
The observation that GNNs under-estimate themselves, as opposed to regular NNs, is also interesting, and the technical contributions to solving this problem are also solid, and thus can be expected to be of interest to the NeurIPS community.
Decision: Accept (Poster)