What Makes Graph Neural Networks Miscalibrated?
Authors: Hans Hao-Hsun Hsu
Yuesong Shen
Christian Tomani
Daniel Cremers
Venue: NeurIPS 2022 Accept (2022)
Keywords: Graph Neural Networks
Post-hoc Calibration

Abstract: Given the importance of getting calibrated predictions and reliable uncertainty estimations, various post-hoc calibration methods have been developed for neural networks on standard multi-class classification tasks. However, these methods are not well suited for calibrating graph neural networks (GNNs), which presents unique challenges such as accounting for the graph structure and the graph-induced correlations between the nodes. In this work, we conduct a systematic study on the calibration qualities of GNN node predictions. In particular, we identify five factors which influence the calibration of GNNs: general under-confident tendency, diversity of nodewise predictive distributions, distance to training nodes, relative confidence level, and neighborhood similarity. Furthermore, based on the insights from this study, we design a novel calibration method named Graph Attention Temperature Scaling (GATS), which is tailored for calibrating graph neural networks. GATS incorporates designs that address all the identified influential factors and produces nodewise temperature scaling using an attention-based architecture. GATS is accuracy-preserving, data-efficient, and expressive at the same time. Our experiments empirically verify the effectiveness of GATS, demonstrating that it can consistently achieve state-of-the-art calibration results on various graph datasets for different GNN backbones.

What Makes Graph Neural Networks Miscalibrated?
Hans Hao-Hsun Hsu"
Christian Tomani 1,2 Technical University of Munich, Germany Munich Center for Machine Learning. Germany
Yuesong Shen" 1,2
Daniel Cremers 1,2
(hans.han hsu, guesong.sheee christian. tomani, cremers) @tumm. de
Abstract
Given the importance of getting calibrated predictions and reliable uncertainty estimations, various post-hoc calibration methods have been developed for neural networks on standard multi-class classification tasks. However, these methods are not well suited for calibrating graph neural networks (GNNs), which presents unique challenges such as accounting for the graph structure and the graph-induced correlations between the nodes. In this work, we conduct a systematic study on the calibration qualities of GNN node predictions. In particular, we identify five factors which influence the calibration GNNs: general under-confident tendency, diversity of nodewise predictive distributions, distance to training nodes, relative confidence level, and neighborhood similarity. Furthermore, based on the insights from this study, we design novel calibration method named Graph Attention Temperature Scaling (GATS), which is tailored for calibrating graph neural net- works. GATS incorporates designs that address all the identified influential factors and produces nodewise temperature scaling using an attention-based architecture. GATS is experiments empirically verify the effectiveness of GATS, demonstrating that it can consistently achieve state-of-the-arr calibration results on various graph datasets for different GNN backbones
preserving. data-efficient, and expressive the same time. Our
Introduction
Graph-structured data, such as social networks, knowledge graphs and internet of things, have wide- spread presence and learning on graphs using neural networks has been active area of research. For node classification on graphs, a wide range of graph neural network (GNN) models, including GCN [1]] GAT (2) and GraphSAGE (7), have been proposed to achieve high classification accuracy.
This said, high accuracy not the only desideratum for a classifier. Especially, reliable uncertainty estimation is crucial for applications like safety critical tasks and active learning. Neural networks are known to produce poorly calibrated predictions that are either overconfident or under-confident
(3. To mitigate this issue a variety of post-hoc calibration methods (5) [5 33 33 6 have been
introduced over the last few years for calibrating neural networks on standard multi-class classification problems. However, calibration of GNNs, in the context of node classification graphs, is currently still an underexplored topic. While is possible to apply existing calibration methods designed for multi-class classification GNNs in nodewise manner, this does not address the specific challenges of node classification on graphs. Especially, node predictions a graph are not i.i.d. but correlated, and tackling structured prediction problem (20). A uniform treatment when calibrating node predictions would fail to account for the structural information from graphs and the non i.i.d. behavior of node predictions.
"Equal contribution
SSurcee code available at
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Our contribution. In this work, we focus on calibrating GNNs for the node classification task [ 32]. First, we aim at understanding the specific challenges posed by GNNs by conducting systematic study on the calibration qualities of GNN node predictions. Our study reveals five factors that influence the calibration performance of GNNs: general under-confident tendency, diversity of nodewise predictive distributions, distance to training nodes, relative confidence level, and neighborhood similarity. Second, we develop Graph Attention Temperature Scaling (GATS) approach, which is designed in way that accounts for the aforementioned influential factors. GATS generates nodewise temperatures that calibrate GNN predictions based on the graph topology. Third, we conduct series of GNN calibration experiments and empirically verify the effectiveness of GATS in terms of calibration, data-effieiency, and expressivity.
2 Related work
For standard multi-class classification tasks, variety of post-hoc calibration methods have been proposed in order to make neural networks uncertainty aware: temperature scaling (TS) [5], ensemble temperature scaling (ETS) [35,, multi-class isotonic regression (IRM) 33 Dirichlet calibration [55 spline calibration [6], etc. Additionally, calibration has been formulated for regression tasks [13 More generally, instead of transforming logits after training a classifier, a plethora of methods exists that modify either the model architecture or the training process itself. This includes methods that are based on Bayesian paradigm [10, I
M evidential theory (25). adversarial calibration [29] and
model ensembling [16]. One common caveat of these methods is the trade-off between accuracy and calibration, which oftentimes do not go hand in hand. Post-hoe methods like temperature scaling, on the other hand, are accuracy preserving. They ensure that the per node logit rankings are unaltered.
Calibration of GNNs is currently substantially less explored topie. Nodewise post-hoc calibration on GNNs using methods developed for the multi-class setting has been empirically evaluated by Teixeira et al. [28]. They show that these methods, which perform uniform calibration of nodewise predictions, are unable to produce calibrated predictions for some harder tasks. Wang et al. [33 observe that GNNs tend to be under-confident in contrast to the majority of multi-class classifiers, which are generally overconfident 55) Based on their findings, Wang et al. [3 propose the CaGCN approach, which attaches GCN on top of the backbone GNN for calibration. Some approaches improve the uncertainty estimation of GNNs by adjusting model training. This includes Bayesian learning approaches 37 g and methods based on the evidential theory [38] 27
3 Problem setup for GNN calibration
We consider the problem of calibrating GNNs for node classification tasks: given a graph G (V,E), the training data consist of nodewise input features {ir E X' and ground-truth labels (y.Nic y = [1, ... K) } for subset C V of nodes, and the goal to predict the labels (y. E y for the rest of the nodes U = V \ L. A graph neural network tackles the problem by producing nodewise probabilistic forecasts P.- These forecasts yield the corresponding label predictions ú; := argmax, py(y) and confidences è := max, (y)).. The GNN is calibrated when its probabilistic forecasts are reliable, e.g., for predictions with confidence 0.8, they should be correct 80% of the time. Formally, a GNN is perfectly calibrated (33) if
VC [0, [0, P(u - n/c = c) c.
In practice, we quantify the calibration quality with the expected calibration error (ECE) [1 We follow the commonly used definition from Guo et al. (5) which uses a equal width binning scheme to estimate calibration error for any node subset N' C V: the predictions are regrouped according to M equally spaced confidence intervals, i.e. (Bi.... BM) with Bmn = ti E "M < Bj the expected calibration error of the GNN forecasts is defined as
< M J. and
M
Bml
ECE
Σ
acc(Bmm) - conf( with
(2)
WT
m=1
1
1
Σ c.
acc acc(B,.)
1(y Uz) and conf (Bm.)
(3)
Bm
|Bmll ie
ie Bmn

---

Factors that influence GNN calibration
To design calibration methods adapted to GNNs, we need to figure out the particular factors that influence the calibration quality of GNN predictions. For this we train a series of GCN [T]]
and GAT [32] models on seven graph datasets: Cora [24]. Citeseer [24], Pubmed [18, Amazon
Computers [26], Amazon Photo [26]. Coauthor CS [26]. and Coauthor Physies [26]. We summarize
the dataset statistics in Appendix A.1 Details about model training are provided in Appendix A.2 for
reproducibility. To compare with the standard multi-class classification case, we additionally train
ResNet-20 [9] models on the CIFAR-10 image classification task [12 as reference.
Our experiments uncover five decisive factors that affect the calibration quality of GNNs. In the following we discuss them in detail.
4.1
General under-confident tendency
Cora
Citeseer
Pubmed
Computers
1.0
Accursty
Accuracy
Accuraay
Accuraty
0.5
0.5
0.5
0.0
0.0
0.0
0.0
Confidence
Conficlence
Confidencee Physics
Confidence
Photo
CIFAR-10
1.0
1.0
1.0
1.0
Accuracy
Accuracy
Accurscy
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.5 Confidence
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 Confidence
1.0
Confidence
Confidence
Figure 1: Reliability diagrams of GCN models trained on various graph datasets. We see a general tendency of under-confident predictions (plots above the diagonal) except the Physics dataset. This is in contrast to the overconfident behavior of multi-class image classification using CNNs (in gray).
Starting with global perspective, notice that GNNs tend to produce under-confident predictions. In Figure [w we plot the reliability diagrams [19] for results on different graph datasets using GCN. Similar to Wang al. [33] we see general trend of under-confident predictions for GNNs. This is in contrast to the standard multi-class image classification case which has overconfident behavior. Also, it interesting see that this under-confident trend can be more or less pronounced depending on the dataset. For Coauthor Physics, the predictions are well calibrated and have no significant bias.
Results using GAT models lead to similar conclusions and provided in Appendix B.]
4.2 Diversity of nodewise predictive distributions
Citeseer
Pubmed
Computers
10
Dersily
10-
Dersity
10-
Dersily
Dersity
Entropy Photo
Entropy CS
Entropy Physics
Entro py
CIFAR-10
10
Density
Density
Dersity
Dersity
2
Entropy
Entropy
Entropy
Entrepy
Figure 2: Entropy distributions of GCN predictions on graph datasets. Compared to the standard classification case, GNN predictions tend to be more dispersed, reflecting their disparate behaviors.
3

---

Contrary to the standard multi-class case, GNN outputs can have varying roles depending on their positions in the graph. which means that their oulput distributions could exhibit dissimilar behaviors. This is empirically evident in Figure where we visualize the entropy distributions of GCN output predictions v.s. the standard multi-class results (GAT results are available Appendix [3.2). We see that the entropies of GNN outputs have more spread-out distributions, which indicates that they have distinct roles and behaviors in graphs.
In terms of GNN calibration, this observation implies that uniform node-agnostic adjustments like might be insufficient for GNNs, whereas nodewise adaptive approaches could
temperature scaling [ be beneficial.
4.3
Distance to training nodes
Cora
Citeseer
Pubmed
Photo
0.15
0.15
0.15
0.35
0.35
0.15
0.15
Cat eetr
0.05
0.05
9.05
0.05
0.05
0.05
0.05
Dist: 15 mmmm
Dec. 15 Trahn
Dit. tn mmin
Da. t0 mm
Did. 00 trann
Diet. train
01.. mmn
Figure 3: Nodewise calibration error of GCN results depending on the minimum distance to training nodes. We observe that training nodes and their neighbors tend be better calibrated.
A graph provides additional structural information for its nodes. One insightful feature is the minimum distance training nodes. We discover that nodes with shorter distances, especially the training nodes themselves and their direct neighbors, tend to be better calibrated.
To evaluate the calibration quality nodewise, we propose the nodewise calibration error, which is based on the binning scheme used to compute the global expected calibration error (ECE) [] :] for each node, we find its corresponding bin depending its predicted confidence, and the calibration error of this bin is assigned to be its nodewise calibration error.
Using this nodewise metric, in Figure 3 we visualize the influence of minimum distance training nodes on the nodewise calibration quality (c.f. Appendix [33 for GAT results). We see that nodes close training ones typically have lower nodewise calibration error. This suggests that minimum distance to training nodes can be useful for GNN calibration.
4.4 Relative confidence level
Cora
Citeseer
Pubmed
Computers
Photo
Physics
0.3
0.3
0.3
0.3
0.3
0.3
0.3
Cale. enerr
Cafe: ereer
0.2
0.2
0.2
02
0.
0.2
62
Calls
0.1
Calls
6.1
6.1
0
Rel confidence
Ral. confidence
Rel. confidence
Ral. confidence
Ral. confidence
Bel. can Sdunce
Rd.. confornce
Figure 4: Nodewise calibration error of GCN results depending on the relative confidence level. We observe that nodes which are less confident than their neighbors tend to have worse calibration.
Another important structural information is the neighborhood relation. We find out that the relative confidence level Sē, of node i, i.c., the difference between the nodewise confidence è, and the average confidence of its neighbors
8c, = ć
(4)
(2)
jEn(i)
has an interesting correlation the nodewise calibration quality. In Figure we show the relation between the relative confidence level of node and its nodewise calibration error (c.f. Appendix B.4 for GAT results). Especially, We observe that nodes which are less confident than their neighbors tend to have worse calibration, and it is in general desirable to have comparable confidence level w.r.t. the neighbors. For GNN calibration, the relative confidence level 86, can be a useful node feature to consider.

---

4.5
Neighborhood similarity
Cora
Citeseer
Pubmed
Computers
Photo
C5
Physics
0.2
0.2
0.
0.2
6.2
6.2
Cit. crer
CaI.. enerr
0.1
611
0.0
0.0
0.0
0.0
3.6
0,0
6,0
homophtly
homophly
homophily
homophily
Node hamophily
Node homophily
homoppilly
Figure 5: Nodewise calibration error GCN results depending on node homophily, Nodes with
strongly agrecing neighbors tend to have significantly lower calibration errors.
Furthermore, we find that different neighbors tend to introduce distinct influences. For assortative graphs which are the focus of this work, find out that calibration of nodes are affected by node homophily, i.e., whether node tends to have the same label prediction as its neighbors. For node with na agreeing neighbors and n.z disagreeing ones, measure the node homophily
+ 1
Node homophily log
(5)
where positive values indicate greater ratio of agree neighbors and vice
FigureBjsummarte the variation of nodewise calibration error w.r.t. the node homophily for different graph datasets (c.f. Appendix BB5 for GAT results). We find out that nodewise calibration errors tend to decrease significantly for nodes with strongly agreeing neighbors. This suggests that neighborhood predictive similarity should be considered when doing GNN calibration.
5
Graph attention temperature scaling (GATS)
Based on the findings Section we design a new post-hoc calibration method, named Graph Attention Temperature Scaling (GATS), which is tailored for GNNs.
5.1 Formulation and design of GATS
To obtain calibration method that is adapted to the graph structure G E) and reflects the observed influential factors in Section the graph attention temperature scaling approach extends the temperature scaling (S) method to produce a distinct temperature T; for each node E V. is then used to scale the uncalibrated nodewise output logits and produce calibrated node predictions p.
2 T
Vie V,
p. softmax
Formulation of T;. The nodewise temperature T; should address the five factors discussed in Section
We achieve this via the following considerations:
We introduce global bias parameter To account for the general under-confident tendency;
To tackle the diverse behavior of node predictions, we learn a nodewise temperature contri- bution T, based on the predicted nodewise logits 24:
To incorporate the relative confidence w.r.t. neighbors. we introduce 8é; from Eq. 4 as additional contribution term scaled by learnable coefficient w;
To model the influence of neighborhood similarity, we use an attention mechanism 31 to aggregate neighboring contributions T with attention coefficients 0i.j depending on the output similarities between the neighbors and j;
Distance to training nodes is used to introduce a nodewise scaling factor n adjust the node contribution and the aggregation process. It is learnable for training nodes and their direct neighbors and fixed to for the rest:
if is training node
7
Ynx
if is neighbor of training node N4 No learnable parameters. otherwise
1,
5

---

Diversty of node predictions Distance to traning nodes Neghhartood ambanly
Goneral under confident rendency
Relative confidence lovel
CS
TW
Accurecy
11 soft w d2g
0.5
T,
74 m +
hm1
den())
0.0
0. Confidence
1.0
Figure 6: Illustration of graph attention temperature scaling (GATS) for graph with four nodes where node 2 is a training node and node is the target node which aggregates information from the other three nodes. GATS incorporates the five factors discussed in Section and produces nodewise temperatures 7; using the graph structure and an attention mechanism.
Putting together the above components, the nodewise temperature T; has the following expression:
ViE V. T, E softplus (w 8è; +
Σ jEA(S)
ai n +/ To-
(8)
Here have multi-head formulation where h indicates the fi-th attention head, and inii) denotes the neighbors of node including self-loop. We uses 8 heads (H = 8), which works well in practice. (c.f. Section 6.3
Defining T/a. Nodewise contribution T.¹ are computed as the outputs of parameterized linear layers (=(::) which take transformed nodewise output logits & as input
Vi E V, vh E HJ, Tf (2;; = (Q")T 2;-
The transformed nodewise output logits 2 are produced follows: we first normalize the original logits 2, to range [0, 1], then sort the classwise logits for each node. This makes the linear layers 6² focus on the general logit distributions rather than class predictions. A similar idea has been explored by Rahimi et al. [23], where they show that intra order- preserrins functions improve the model calibration. Here we find out that this sorting-based transformation helps GATS to learn useful representations of nodewise contributions 7/" (c.f. Section 6.3
Defining Oiij- The attention coefficients oi are defined based on the neighbor similarity, which is determined by the inner product between rescaled nodewise logits 2//y; and 2//Y5. Inspired by Veličković et al. 32. we compute the attention coefficients aij as follows:
(OLS))ERRO) = soft (eeakyReLUU ( (__= 2₁ ))
(10)
jEa(()
75
5.2 Calibration properties of GATS
Zhang et al. [35] propose three desiderata for calibration methods: accuracy-preserving, data-efficient, and expressive. GATS fulfills all of them:
GATS is preseeviireeeeeing:: since all node predictions scaled by inverse temperatures 1 /T& which are positive scalars, the order of output logits is preserved;
GATS data-efficient. It is a parametric calibration model with C H learnable parameters (To, w, "e, "n, (ghH))))) According to Zhang et al. 35 parametric methods are already data-efficient;
GATS is expressive, as it produces nodewise temperatures T, adapted to the graph structure.
Experiments in Section 6.2 empirically confirm the data-efficiency and expressivity of GATS.

---

5.3 Comparison with CaGCN
It is interesting to compare our proposed GATS approach to the CaGCN method proposed by Wang et al. [33], since both approaches aim at calibrating GNNs, and both make use of the graph structure to produce nodewise temperatures. While CaGCN uses a GCN generate nodewise temperatures straighforwardll. GATS uses an attention mechanism which differentiates the influence from various neighbors. GATS also integrates a series of careful designs following the insights from the study in shows that GATS tends to produce better calibration results
Section 4 Experiments in Section 6.1
compared to CaGCN.
Experiments
To evaluate the performance of GATS on GNN calibration and understand the effects of its designs, conduct a series of experiments for baseline comparison and ablation study. We use two representative GNNs: GCN and GAT, which are trained the seven aforementioned graph datasets plus larger graph, CoraFull (2). for post-hoc calibration. We display the ECE results with M = 15 bins. We use the expected calibration error (ECE) 21 with M 15 bins as an evaluation metric, and follow an experimental protocol similar to Kull et al. 14); For all the experiments, we randomly split the dabeled/unlabeles (15%/85% data five times, and use three-fold internal vrass-validation of the labeled data to train the GNNs and the calibrators. We also utilize five random initializations, resulting in 75 total runs for each experiment. We provide detailed experimental settings in Appendix
6.1 Performance comparison
We benchmark GATS against existing baselines on a variety of GNN calibration tasks. We compare GATS with the following baselines:
Temperature scaling (TS) S simply uses a global temperature to scale the logits.
Vector scaling (VS) (5) scales the logits separately over the class dimension and additionally introduces a classwise bias for the recalibrated output logits.
Ensemble temperature scaling (ETS) 355 learns a mixture of uncalibrated, TS-calibrated, and uniform probabilistic outputs.
GCN as a calibration function (CaGCN) GNNs. It uses a GCN on top to generate nodewise temperatures.
specifically designed for calibrating
Additionally, we also report the ECEs of uncalibrated predictions as a reference. Among the above baselines, TS, VS, and ETS are calibration methods designed for standard classification cases and operate on nodes uniformly. CaGCN on the other hand performs separate nodewise adjustments and uses the graph structure, similar to our proposed GATS approach.
For the post-hoe calibration experiments, we fix the weight of the trained GNN backbones and adjust the parameters of the calibration methods on the validation set. Negative log- le-likelioood is chosen as the objective for the calibration process. We provide details of method configurations and calibration settings in Appendix A.3 Table summarizes the calibration results.
Overall, we observe that GATS consistently produces well-calibrated predictions for all graph datasets and GNN backbones. Except for the GAT model trained on Pubmed (3rd best) and the GCN model trained Amazon Computers (2nd best), GATS achieves the highest calibration quality in all cases.
Also, it is interesting to see that for all cases the best result is achieved by methods which use the graph structure and produce adapted adjustments for different nodes. This demonstrates the necessity of designing calibration methods that address the unique challenges posed by GNN calibration.
Although CaGCN can get the best results for Pubmed using GAT and Amazon Computers using GCN, we see that its performance is rather unstable for different scenarios, and sometimes it can even produce worse calibration results than the uncalibrated baseline. Using their proposed margin- based loss did not help our settings. We suspect that CaGCN might have an overly complex architecture for the task, and it cannot differentiate neighborhood influences with the common normalized adjacency matrix. Our proposed GATS model does not have this issue. It has consistent and good calibration performance in all cases.
7

---

Table GNN calibration results in terms ECE (in percentage, lower is better) of GATS and other baseline methods on various graph datasets. Overall, GATS achieves state-of-the-an performance, getting the best results in most scenarios. Also, all best results are achieved by methods that consider the graph structure. This shows the need for dedicated methods to tackle GNN calibration.
Dataset
Model
Uncal
TS
VS
ETS
CaGCN
GATS
GCN
13.04:55.22
3.92:1
4.36 1.34
3.79-11.35
5.29:11.77
3.64 1.34
Cora
GAT
23.31
3.69:10.90
3.30=1.12
3.54
4.09
3.18
GCN
10.66 5.15+1.50
4.92=1.44
1.65+1..69
6.86+1.41
4,43+11.30
Citeseer
GAT
22.88+3..53
1.74+1.47
4.25=1.48
4.11=1.64
5.75+1.31
3.86+1.56
GCN
7.18
1.26.10.28
1.46
1.24
1,0910.52
0.38
Pubmed
GAT
12.32.10.80
1.19.10.66
1.00-0.32
1.20.10.32
0.98110.31
1.03.10.32
GCN
3.00+0.80 2.65+0.57
2.70=0.63
2.58=0.70
1.72++0.53
2.23=+0.99
Computers
GAT
1.88
1.63:00.66
1.67
1.64
2.03
1.39 0.39
GCN
2.24.11.03 1.68
1.75
1.68
1.99-10556
1.51:10.52
Photo
GAT
2.02
1.61
1.63=0.69
1.67
2.10
1.48
GCN
1.65++0.92
0.98
0.96=0.30
0.94
2.27+11007
0.88
CS
GAT
1.40:
0.93+-0.34
0.87=0.35
0.88+
1.52
0.81 0.330
GCN
0.52:10.29
0,51
0.48=0.16
0.52:10.19
0,94110551
0.46110116
Physics
GAT
0.45:10.21
0.50:1021
0.52=0.20
0,50=0..1
1.17:0.42
0.42:10.14
GCN
6.50
5.54+-0.43
5.76-0.42
5.38+0..9
5.86++2.52
3.76+0074
CoraFull
GAT
1.73
4.00++0.50
4.17=0.43
3.89
6.55+
3.54++0.3
We also observe that the results tend to have high variations, since GNN backbones tend to predict highly varying results when trained with different initial weights and random splits [26]. We ensure the reliability of the results by averaging over a total of 75 runs with various initial weights and random splits for each case.
6.2 Data-efficiency and expressivity of GATS
Furthermore, we analyze the data-efficiency and the expressivity of GATS for GNN calibration. For this we reuse the GNN models trained on the CoraFull dataset, and consider the influence calibration sample size on the GNN calibration performance. For comparison we also report the corresponding results using ensemble temperature scaling and CaGCN, Figure visualizes the re- sults with GCN backbone. The results for GAT backbone are summarized in Appendix D
ETS
CaGON
GATS
Uncal.
ECE 1963
Overall. we see that GATS is both data-efficient and expressive. requires few calibration sam- ples to get decent calibration performance. This in contrast CaGCN which needs more than 5% of nodes for calibration to get acceptable re- sults. Compared to ETS, GATS is more expressive and has considerably lower calibration emror for CoraFull, which a large graph dataset.
1.0 15
20
25
30
Calibration Dataset Size (36)
Figure ECEs (in percentage) on CoraFull for ETS, CaGCN, and GATS using various amounts calibration data. We see that GATS is data- efficient and expressive for GNN calibration.
6.3 Ablation study
To empirically analyze the effect of various GATS design choices, we conduct series of ablation study experiments in this section. Overall, we notice that all designs are advantageous and removing any of them leads to general decrease in performance.

---

Table Ablation study results in terms of ECE (in percentage) for various GATS designs. Overall, all designs are beneficial and removing any of them leads to worse results in general.
Dataset
Model
w/o To
w/o 7/
w/o 66;
w/o attention
w/o sorting
GATS
GCN
3.72:11.20
3.80:11.51
3.71111.18
3.6311
4.35
3,64-1
Cora
GAT
3.25++1.00
3.46+1.00
0.24
3.69++0.9
4.18+1.70
3.18=0..0
GCN
1.50
4.73
4.49:1
1.95
5.87.1
4.43-1
Citeseer
GAT
3.56.11..73
1.39
3.87
4.81
4.99112.34
3.86
GCN
2.20=
1.60+:0,64
1.54
1,59+0.67
1.68+0.61
0.51
Photo
GAT
2.37=1001
1.53+0.63
1.47:0662
1.62+0.66
1.77+0.70
1.48=0.61
Effect of global bias Ta- We consider the effect of global bias To by comparing GATS variant without it (i.e., setting To = 0 in Eq. ( Its results are collected in column "w/o To" of Table 2 Overall we that learnable bias To is beneficial in most cases.
Effect of nodewise scaling factor %/- The nodewise scaling factors % provide custom adjustment for training nodes and their neighbors. Removing its influence be done by fixing all py one in Eqs. (8) and (TO). The results of the variant without "/ are recorded in column "w/o " of Table They worse general, suggesting that nodewise scaling factors 7/4 indeed helpful.
Effect of relative confidence level dé;, To evaluate the impact of introducing nodewise relative confidence level 6ē; create GATS variant where in Eq. (8) is fixed to zero, which effectively removes the influence of the relative confidence level 8ć, In column "w/o 52," of Table we have the results corresponding to this variant, which is in general slightly worse than the standard GATS which includes 86;-
Effect of attention-based aggregation. To understand the role played by the attention mechanism, we create a GATS variant which completely removes the attention related term (Ejent)) Ci,j N TJ') from Eq. (8. Its performance is shown in column "w/o attention" of Table Again, we observe that removing the attention-based component tends worsen the calibration performance.
Effect of logit sorting. GATS uses normalized and sorted logits & as input to generate nodewise temperature contributions 7/. And we find that this sorting transform is essential for learning good representations of T/¹ In column "w/o sorting" of Table 2 we have the results for GATS variants which uses the logits without sorting to compute T.h And we observe that this deteriorates the calibration performance.
Table 3: Calibration results (ECE in percentage) GATS models with different numbers of attention heads. 8 attention heads are sufficient for optimal GNN calibration results.
Number of Heads 4
Dataset
Model
8
16
GCN
1.45
3.75+1.42
3.79+1.445 1.45
3.66+ 1.33
3.50
Cora
GAT
3.28
3.49=1.18
3.43+11.15
3.20++0.00
3.31+:0.74
GCN
4.74.11.43
1.77
1.70
1.43
1.42-11.04
Citeseer
GAT
1.63
4.57
1.20
3.86.11.56 1.56
1.20
GCN
1.51+:0.4 1.54
1.52:0.58
1.51
1.50+0.51
Photo
GAT
1.52-40.61
1.52.10.65
1.53-10.60
1.48-10.61
1.50+L0.60
Effect of attention head count H. Finally, we analyze the influence of multi-head count H on the
GNN calibration results. For this we run a series of experiments using GATS models with 1, 2, 4, and 16 attention heads. The results are collected in Table 3 We see that for GCN backbones, GATS models with more attention heads tend to get better results. However, for GAT backbones, using 16 heads results in worse performance compared to 8 heads. Accounting also for the fact that doubling

---

the attention head count effectively doubles the computational requirements, 8 attention heads is decent general setting for GATS.
Conclusion
In this work, tackle the GNN calibration problem. We conduct systematic study to analyze the calibration properties of GNNs predictions. Our study reveals five influential factors and manifests the unique challenges raised by GNN calibration. Based the insights from our studies, we propose novel calibrator, GATS, which accounts for the identified factors and is tailored for calibrating GNNs. GATS is accuracy-preserving demonstrate that GATS achieves state-of-the-ar performance for GNN calibration on various graph datasets and for different GNN backbones.
data-efficient, and expressive the same time. Our experiments
Our work focuses on the node classification tasks for assortative graphs, where neighbors tend to agree with each other. It thus important to realize that the validity of the conclusions from Section 4 limited to the assortative case, and might no longer hold for disassortative graphs [39 22 It can be an interesting future work to conduct similar studies for GNN calibration in the heterophilous case, especially when more established GNN architectures are available. More generally, devising calibration methodologies for other graph learning tasks such link prediction (36 and graph classification [3] could also be an interesting direction for future research.
Acknowledgments and Disclosure of Funding
This work was supported by the Munich Center for Machine Learning (MCML) and by the ERC Advanced Grant SIMULACRON, The authors would like thank Nikita Araslanov for proofreading and helpful discussions, as well as the anonymous reviewers for their constructive feedback.
References
[1] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In ICML, 2015.
[2] A. Bojehevski and S. Günneeaann. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. In ICLR, 2018.
[3] F. Errica, M. Podda, D. Bacciu, and A. Micheli. A fair comparison of graph neural networks for graph classification. In ICLR, 2020.
[4] Y. Gal and Z. Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. ICML, 2016.
[5] C. Guo, G. Pleiss. Y. Sun, and K. Weinberger. On calibration of modern neural networks. In ICML, 2017.
[6] Gupta, Rahimi, T. Ajanthan, Mensink, C. Sminchisescu, and R. Hartley, Calibration of neural networks using splines. In ICLR, 2021.
[7] Hamilton, Ying, and J. Leskovec. Inductive representation learning on large graphs. In NeurlPS. 2017.
[8] A. Hasanzadeh, E. Hajiramezanali, S. Boluki, M. Zhou, N. Duffield, K. Narayanan, and X. Qian. Bayesian graph neural networks with adaptive connection sampling. In ICMI., 2020.
[9] K. He, X. Zhang. S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV, 2016.
[10] J. M. Hemandezz-Lobto and R. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In ICML, 2015.
[11] T. N. Kipf and and 2017.
M. Welling, Semi-supervised classification with graph convolutional networks. In ICLR,
[12] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
10

---

[13] V. Kuleshov. N. Fenner, and S. Ermon. Accurate uncertainties for deep learning using calibrated regression. In ICML, 2018.
[14] M. Kull, T. M. S. Filho, and P. Flach. Beyond sigmoids: How to obtain well well-calibrated probabilities from binary classifiers with beta calibration. Electronic Journal of Statistics, 11(2):5052 5080, 2017.
[15] M. Kull, M. Perello Nieto, M. Kängsepp. T. Silva Filho, H. Song, and P. Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilitiess with dirichlet calibration. In Newr/PS, 2019.
[16] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable prodictive uncertainty estimation using deep ensembles. In NIPS, 2017.
[17] W. J. Maddox, P. Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson. A simple baseline for bayesian uncertainty in deep learning. In NewrlPS, 2019.
[18] G. Namata, B. London, L. Getoor, and B. Huang. Query-driven active surveying for collective classification. In ICML Workshop, 2012.
[19] A. Niculescu-Mizit and R. Caruana. Predicting good prohabilities with supervised learning. In ICML,, 2005.
[20] S. Nowozin and C. H. Lampert. Structured learning and prediction in computer vision. Foundations and Trends@ in Computer Graphics and Vision, ((3-4):185-366 365,3655225 2011.
[21] M. Pakdaman Naeini, G. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI, 2015.
[22] H. Pei. B. Wei, K. C.-C. Chang, Y. Lei, and B. Yang. Geom-GCN: Geometric Graph Convolutional Networks. In ICLR, 2020.
[23] A. Rahimi, A. Shaban, C. Cheng, R. Hartley, and B. Boots. Intra order-preserving functions for calibration of multi-class neural networks. In NeurlPS, 2020.
[24] P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Gallagher, and T. Eliassi-Rad. Collective classification in network data. Al Mag., 29(3):93-106, 2008
[25] M. Sensoy, M. Kaplan, and M. Kandemir. Evidential deep learning to quantify classification uncertainty. In NeurlPS, 2018.
[26] O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann, Pitfalls graph neural network evaluation. arXiv preprint 181.881811.05888, 2018.
[27] Stadler, Charpentier, Geisler, Zügner, and Günnemann. Graph posterior network: Bayesian predictive uncertainty for node classification. In NeurIPS, 2021.
[28] L. Teixeira, Jalaian, and B. Ribeiro. Are graph neural networks miscalibrated?? JCML Workshop. 2019.
[29] C. Tomani and F. Buettner. Towards trustworthy predictions from deep neural networks with fast adversarial calibration. In AAAI, 2021.
[30] C. Tomani, S. Gruber, M. F. Erdem, D. Cremers, and F. Buettner. Post-hoc uncertainty calibration for domain drift scenarios. In CVPR, 2021.
[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. In NIPS, 2017.
[32] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. In ICLR, 2018
[33] X. Wang. H. Liu. C. Shi, and C. Yang. Be confident! towards trustworthy graph neural networks via confidence calibration. NeurIPS, 2021.
[34] Y. Wen, P. Vicol, J. Ba, D. Tran, and R. Grosse. Flipout: Efficient beedd-independent weight perturbations on mini-batches. arXiv preprint arXiv:1803.04388 2018
[35] J. Zhang, B. Kailkhura, and T. Y. Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In /CML, 2020.
[36] M. Zhang and Y. Chen. Link prediction based graph neural networks. In NeurIPS, 2018
11

---

[37] Y. Zhang, S. Pal. M. Coates, and D. Üstebay. Bayesian graph convolutional neural networks for semi- supervised classification. In AAAI, 2019.
[38] X. Zhao, F. Chen, S. Hu, and J. Cho. Uncertainty aware semi-supervised learning graph data, In NewrIPS, 2020.
[39] J. Zhu, Y. Yan, L. Zhao, M. Heimann, L. Akoglu, and D. Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In NeurlPS, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] c.f. Section 7
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "ql7mN6PKTs5", "reviewer": "Reviewer_hr2j", "summary": "The paper proposed a new method to solve the GNN calibration problem. It identifies several factors that make the calibration different on GNNs from general multi-class classification tasks. Based on the factors a new calibration method is developed which produces nodewise temperature scaling using an attention-based architecture. Experiments demonstrate that the proposed GATS is accurate, data-efficient and expressive.", "questions": "see above weakness questions.", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\n* A thorough study of the factors which impact the calibration error of GNNs\n* A new temperature scaling model which considers the factors and mitigates the calibration problem in GNNs\n\nWeakness:\n\n* The writing is generally clear, but as a reader who is not familiar with calibration, it is not easy to understand how the calibration error is calculated and why the GCN model has a different pattern from general multi-class classification. I think it is better to explain it a little bit in the beginning of the method.\n\n* Section 3 and Section 4 should be aligned better. It seems all factors in Sec 3 are used later, but not all of them are directly corresponding a parameter in Eq. (5) (e.g. h in Eq. (2)). Since there are too many new brough-in parameters in Eq(5), it should be explained more. Some notations can be reclaimed (\\eta_{c_i}), some notations should be described more clearly (e.g. what does the h mean in \\tau_i^h ).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bQsthlKF5v", "reviewer": "Reviewer_mbVJ", "summary": "Calibration of neural networks is an important area of research. This paper provides empirical insights on the miscalibration of GNNs for node classification, and then proposes an approach that takes all those factors into account to calibrate the GNN outputs.", "questions": "Q1: The \"distance to training nodes\" seems to lead to miscalibration mostly on Cora and Citeseer, and not on the the other graphs. This is also true about the \"general under-confident tendency\". Is there a reason for this? Is there a correlation between the two factors? Are Cora and Citeseer different in any specific way that makes them more miscalibrated for nodes far away from the training set? \nMore evidence may be necessary to back up these two factors.\n\nQ2: From what I understand, \"diversity of node distributions\" is a direct effect of \"general under-confident tendency\". That is, if the model is under confident in its probability for the correct class, it means other classes have higher probabilities, and so the entropy is high. Is there anything novel about 3.2 then?\n\nQ3: Among the five factors, I was mostly surprised/impressed by \"relative confidence level\". It seems like a good finding and the analysis in Fig 3 shows the importance of it. However, I was disappointed when I saw the ablation results; none of the degradations due to removing \\delta c_i seem to be statistically significant. Is there a reason for the mismatch between what figure 3 shows regarding the importance of this factor, and what Table 2 shows regarding the unimportance of it? Is it because the proposed calibration algorithm doesn't utilize this factor in the best way?\n\nQ4: Figs 1 and 3 seem to imply that the GCN outputs are well-calibrated for the physics dataset. However, Fig 4 implies that for a subset of the nodes (those with low relative confidence), the outputs are quite miscalibrated. Can you explain how these results can all hold simultaneously? Are there only very few nodes with low relative confidence for Physics (and that's why there is an extremely large std)?\n\nQ5: This is more of an observation than a question, but \"distance to training nodes\" and \"neighborhood similarity\" are both factors that affect how \"easy\" the prediction is for a node, in the sense that, e.g., nodes with many neighbors of the same class are highly probable to belong to the same class (under homophily -- which the studied datasets all exhibit highly). Therefore, it is not quite surprising that these two factors play a role in miscalibration. Nevertheless, seeing empirical backup for the two is definitely nice.", "limitations": "1- Among the five factors studied in the paper, the first one has been previously studied, the second one seems to be a direct effect of the first one, the third-one and the fifth one are not quite surprising, and the fourth one is quite interesting but doesn't seem to have a significant effect according to the ablation studies.\n\n2- According to the ablation study, many of the sub-components of the proposed algorithm seem to have negligible effects on the results for many of the datasets (although it is partly understandable because some components may make up for the others).\n\nNevertheless, I believe a dedicated work for studying multiple factors about the miscalibration of GNNs and providing a calibration approach based on those factors could be beneficial to the community.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ Well written\n+ Good, in-depth analysis of different factors\n+ Tested on a suite of 7/8 datasets\nWeaknesses:\n- Some of the findings from the analyses in section 3 do not seem to match with the ablation study results.\n- Some of the identified factors are somewhat expected, and not surprising", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8B_BcQmNQ2w", "reviewer": "Reviewer_rsXN", "summary": "This paper considers confidence calibration for GNN node classification. It conducted a very throughout and careful empirical study of the factors in graph structure that may effect the calibration. It discovered 5 factors that contributing to miscalibration: general under-confidence, diversity of distribution, distance to training, relative confidence level, and neighborhood similarity. After demonstrating each factor independently, the authors proposed a temperature scaling method where each node is assigned with a different temperature, based on a formula that is inspired by the empirical analysis. The proposed GATS method is parameteric efficient but also captures the effects that influences confidence calibration. In particular, GATS out performs other calibration methods for most datasets.", "questions": "N/A", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n  - the GNN calibration problem is important but underexplored\n  - the empirical study is conducted very carefully, the results are very inspiring\n  - the proposed method is very simple, interpretable and effective\n  - the ablation study is very informative\n  - the logic flow and writing is very clear\n\nWeakness\n  - the authors should explain how the GATS parameters are learned\n  - the author could compare GATS & GaGCN in terms of the number of parameters", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7Qtk2rUvI9", "reviewer": "Reviewer_hxXe", "summary": "This work studies prediction calibration for GNNs on node prediction. It performs empirical analysis to identify five factors that are correlated with the GNN prediction confidence. Then an attention-based temperature scaling method is proposed to calibrate GNNs. Experiments show that the proposed method achieves lower ECE compared to baselines.\n", "questions": "In figure 1, several plots show that the low confidence predictions achieve super high accuracy close to 1. Why is this the case?\n", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nStudying GNN calibration by analyzing its specific challenges that are different from CNNs is a good contribution.\nThe result in terms of ECE seems promising.\nThe paper presentation is mostly clear.\n\nWeaknesses:\n\nI have some concerns about the five factors proposed by the paper.\n\nThe definition and analysis for some of the five factors are a bit vague. For example, Section 3.2 is regarding the \"diversity of node distributions\" factor, which is analyzed through the \"entropy distribution\" plots in Figure 2. If I understand them correctly, in Figure 2, the entropy of each GNN output is computed, and then a density estimation based on all such entropy values is plotted. In this case, the name \"node distribution\" can be confusing because the factor is actually the distribution of GNN predictions on nodes. Node distribution should refer to the actual node features/labels, and it is a property of the data. Also, \"diverse entropy implies uniform temperature scaling is insufficient\" doesn't seem quite right. We can have a very large uniform temperature to make all predictions very uniform or a very small uniform temperature to make all predictions close to deterministic. Both cases will make the entropy distribution very concentrated. The latter can generate a plot close to the GNN case. Furthermore, the plots in the second row seem quite similar to me, which questions whether this factor is always necessary because it is dataset-dependent.\n\nThe analysis of these five factors is not thorough given some of these factors may be correlated. For example, the relative confidence level and the neighborhood similarity should be correlated. Intuitively, we are unlikely to have neighbors with the same prediction but very different confidence, or different predictions and both very high confidence. Then doing a more detailed analysis can help to better understand their correlations and importance and maybe disentangle the factors. In the current version, although the ablation study in Table 2 suggests all of the factors help with some ECE reduction, their significants are actually questionable. Like for $\\delta c_i$, removing it only increases the ECE by less than 0.1% whereas the std is around 1%.\n\nAs a core contribution of the paper, what factors truly influence the GNN prediction and confidence is the key contribution. Ideally, we would want fewer factors and more disentangled factors. I feel the factor definition and analysis can be further improved.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept