GETS: Ensemble Temperature Scaling for Calibration in Graph Neural Networks
Authors: Dingyi Zhuang
Chonghe Jiang
Yunhan Zheng
Shenhao Wang
Jinhua Zhao
Venue: ICLR 2025 Spotlight (2025)
Keywords: Uncertainty Quantification; Graph Neural Networks; Ensemble Learning; Mixture of Experts
Primary area: learning on graphs and other geometries & topologies
TL;DR: GETS is a novel framework for calibrating GNNs, combining input and model ensemble strategies to improve uncertainty calibration across 10 benchmark datasets, offering significant performance gains while remaining efficient and scalable.

Abstract: Graph Neural Networks (GNNs) deliver strong classification results but often suffer from poor calibration performance, leading to overconfidence or underconfidence. This is particularly problematic in high-stakes applications where accurate uncertainty estimates are essential. Existing post-hoc methods, such as temperature scaling, fail to effectively utilize graph structures, while current GNN calibration methods often overlook the potential of leveraging diverse input information and model ensembles jointly. In the paper, we propose Graph Ensemble Temperature Scaling (GETS), a novel calibration framework that combines input and model ensemble strategies within a Graph Mixture-of-Experts (MoE) architecture. GETS integrates diverse inputs, including logits, node features, and degree embeddings, and adaptively selects the most relevant experts for each node’s calibration procedure. Our method outperforms state-of-the-art calibration techniques, reducing expected calibration error (ECE) by $\geq$ 25% across 10 GNN benchmark datasets. Additionally, GETS is computationally efficient, scalable, and capable of selecting effective input combinations for improved calibration performance. The implementation is available at https://github.com/ZhuangDingyi/GETS/.

Published as a conference paper at ICLR 2025
GETS: ENSEMBLE TEMPERATURE SCALING FOR CALIBRATION IN GRAPH NEURAL NETWORKS
Dingyi Zhuang
Chonghe Jiang'
Massachusetts Institute of Technology di dingyi @mit.edu
The Chinese University of Hong Kong ch c jingegg 1 ink, cuhk edu. hk
Yunhan Zheng
Singapore- MIIT Alliance for Research and Technology (SMART) yunhan. zheng@ smart mit edu
Shenhao Wang
Jinhua Zhao
University of Florida
Massachusetts Institute of Technology jinhua@mit.cs
shenhaowang@ufl.edd edu
ABSTRACT
Graph Neural Networks (GNNs) deliver strong classification results but often suffer from poor calibration performance, leading to overconfidence under- confidence. This is particularly problematic in high-stakes applications where accurate uncertainty estimates essential. Existing post-hoc methods, such as temperature scaling, fail to effectively utilize graph structures, while cur- rent GNN calibration methods often overlook the potential of leveraging di- verse imput information and model ensembles jointly. In this paper, we propose Graph Ensemble Temperature Scaling (GETS), novel calibration framework that combines input and model ensemble strategies within Graph Mixture-of- Experts (MoE) architecture. GETS integrates diverse inputs, including logits, node features, and degree embeddings, and adaptively selects the most relevant experts for each node's calibration procedure. Our method outperforms state- of-the-art calibration techniques, reducing expected calibration error (ECE) by 2 25% across 10 GNN benchmark datasets. Additionally, GETS compu- tationally efficient, scalable, and capable of selecting effective input combina- tions for improved calibration performance. The implementation is available at https::/ wwg.tht...c.o.m com http://cc////haggggggggog htgpg//TS/nn//////////m
INTRODUCTION
Graph Neural Networks (GNNs) have emerged as powerful tools for learning representations in numerous real-world applications, including social networks, recommendation systems, biological networks, and traffic systems, achieving state-of-the--a performance in tasks like node classifica- tion, link prediction, and graph classification (Kipf & Welling 2016 Veličković al.) (017). Their
ability to capture complex relationships and dependencies makes them invaluable for modeling and
predicting interconnected systems (Fan al. 2019 Wu et al. 2019 Scarselli al. 2008 Wu al.
2022
2021
Jiang* et al. 2024 Luo et al. 2024).
However, ensuring the reliability and trustworthiness of GNN prediction remains critical challenge, especially high-stakes applications where decisions can have significant consequences (e.g. hu- man safety). Consider GNN classification as an example, one key aspect of refining model reliability uncertainty calibration, which aims to align the predicted probabilities with the true likelihood of outcomes (Guo et al.) (017). In short, well-calibrated models provide confidence estimates that reflect wolld probabilities, which essential for risk assessment informed decision-making
Zhuang et al. 2023)
"Conresponding author.
1 Equal contribution.

---

Published as a conference paper at ICLR 2025
Puhnool
Cau--all
Oyla Maxi
Ploto
0.00
0.07
D06
0.00
0.00
610
8000
Dearce
Degroe Rarge
Figure 1: Expected calibration error (ECE), see Equation 3 for a well-trained CaGCN model (Wang al. 2021). ECE measured by grouping nodes based on degree ranges rather than predicted confidence. Datasets are sorted by average degree TVT as a measure of connectivity (see Table from low to high. In lower- coneecconnettivity datasets like Pubmed and Cora-full, high-degree nodes tend show larger calibration errors, whereas in more connected datasets like Photo and Ogbn-arxiv, low-degree nodes exhibit higher calibration errors.
21E
Calibration methods are typically applied post-hoc and can be broadly classified into categories: nonparametric and parametric. Nonparametric methods, such as histogram binning and isotonic re- gression, adjust confidence estimates based on observed data distributions without assuming spe- cific functional form (Zadrozny & Elkan, 2001 Naeini et al., (013). While these methods allow flexibility in modeling, they often require large calibration datasets and can struggle with high- dimensional or unevenly distributed graph data, making them less practical for graphs with vast numbers of nodes. In contrast, parametric methods, including temperature scaling (TS) and Platt scaling, assume functional relationship to adjust model outputs (Guo et al., 2017 Platt et al., 1999). Among these, TS has gained popularity due to its simplicity and effectiveness in calibrating models by smoothing predicted probabilities. This method is scalable and easy to implement, espe- cially for lcrge-scale graphs. Recent work has moved beyond the uniform adjustments applied to all data in classic TS, focusing on learning node-specific calibration parameters to capture individual characteristics that affect graph calibration performance (Tang et al. 2024 Hsu et al. 2022).
Motivated by the methodologies discussed above, when we dive into the GNN (classification) cali- bration task, we find that the subtleties lie in two main challenges: (i) the difficulty of representing the calibration eгror score for the individual sample points; (ii) the disability to identify and utilize multiple information related to the GNN calibration. Existing approaches for graph temperature scaling primarily rely
additional GNN model to output node-wise wide temperatures. However, systematic methods for determining the inputs to the calibration network and for integrating various input information have not been thoroughly explored.
Given these challenges, we aim to address the following issues: (i) What are the key factors in- fluencing the calibration task for GNN classification? (ii) How can we effectively integrate these factors, which may exist across different spaces and scales, perform GNN calibration?
For issue (i), we observe that, in addition to the logits and features discussed in prior work WWagg al., 2021 Huang al. 2022), Figure demonstrates that node degree influences the calibration error, consistent with the message-passing mechanism in GNNs. In some cases, low node degree can result in worse calibration performance due to insufficient information propagation. Conversely, excessive (and potentially mislcading) information may flood single node, leading to overconfi- dence in its predictions (Wang et al. 2022). Regarding issue (ii), it remains unclear how to develop an effective algorithm that incorporates multiple factors for calibration within the graph structure. To address these challenges, we propose a novel calibration framework called Graph Ensemble Temperature Scaling (GETS). Our approach integrates both input ensemble and model ensem- ble strategies within a Graph Mixture-of -ofeeets (MoE) architecture, thereby overcoming the lim- itations of existing methods. By incorporating multiple calibration experts -each specializing in different aspects of the influential factors affecting calibration -EET can adaptively select and combine relevant information for each node.
2

---

Published as a conference paper at ICLR 2025
Experimental results 10 GNN benchmark datasets demonstrate that GETS consistently achieves superior calibration performance, significantly reducing the ECE across all datasets. This highlights its effectiveness in addressing the calibration problem in the GNN classification task. Additionally, GETS is scalable and capable of handling large-scale datasets with computational efficiency.
Our key contributions are summarized as:
We introduce novel framework, GETS, which combines input and model ensemble strate- gies within a Graph MoE architecture. This enables robust node-wise calibration by inte- grating diverse inputs, such as logits, features, and degree embeddings.
GETS significantly reduces Expected Calibration Error (ECE) across 10 GNN benchmark datasets, outperforming state-of-the-ar methods like CaGCN and GATS, and demonstrat- ing effectiveness across diverse graph structures.
GETS employs sparse gating mechanism select the top k experts for each node, en- abling scalable calibration on large graphs while maintaining computational efficiency.
2 RELATED WORK
2.1 UNCERTAINTY CALIBRATION
Uncertainty calibration adjusts the predicted probabilities of the model to more accurately reflect the true likelihood of outcomes. Calibration methods are generally classified into in-training and post-hoc approaches.
In-training calibration adjusts uncertainty during model training. Bayesian methods, such as Bayesian Neural Networks (BNNs) and variational inference, achieve this by placing distributions over model parameters (Gal & Ghahramani 2016 Sensoy et al. 2018 Detommaso al. 20222) Fre-
quentist approaches, including conformal prediction (Zargarbashi et al. [023] Huang et al.) 2024] Deshpande et al. 2024) and quantile regression (Chung al. 2021), are also employed ensure well-calibrated outputs.
Despite these efforts, model misspecification and inaccurate inference often lead to miscalibration (Kulehhov et al., 2018). To address this, post-hoc methods recalibrate models after training. These can be divided into nonparametric and parametric approaches. Nonparametric methods, such as histogram binning (Zadrozny& & Elkan 200T), isotonic regression (Zadrozny & Elkan 2002), and Dirichlet calibration (Kull et al. 2019). adjust predictions flexibly based on observed data (Naeini al. 2014). Parametric methods assume specific functional form and learn parameters that di- rectly modify predictions. Examples include temperature scaling (Guo et al. 2017), Platt scaling (Platt al. 1999), beta calibration (Kull et al. 2017), and
2.2 GNN CALIBRATION
Calibration of GNNs remains under-explored area (Zhuang et al. 2023). Take the node clas- sification task as an example, most existing methods adopt post-hoc approaches, often relying on temperature scaling where the temperature is learned via certain network.
Teixeira et al. (2019) noted that ignoring structural information leads to miscalibration in GNNs. GNNs also tend be underconfident, unlike most multi-class classifiers, which are typically over- confident. To address this, CaGCN introduces GCN layer on top of the base GNN for improved uncertainty estimation using graph structure (Wang et al., (021). [ssu al.] (2022) identified key factors affecting GNN calibration and proposed GATS, which employs attention layers to account for these factors. TTagg et al. (2024) further explored node similarity within classes and proposed ensembled temperature scaling jointly optimize accuracy and calibration.
PROBLEM DESCRIPTION
3.1 UNCERTAINTY CALIBRATION GNN
This paper addresses the calibration of common semi-supervised node classification tasks. Let G (V, E) represent the graph, where V is the set of nodes and E is the set of edges. The adjacency matrix is denoted by A N where N |V] represents the number of nodes. Let X' be the input space and y be the label space, with y; E y {1..... ; K), where K 2 is the number of classes. Let X X and Y denote the input features and labels, respectively. A subset of
3

---

Published as a conference paper at ICLR 2025
nodes with ground-trutt truth labels, C V, is selected for training. The goal of semi-supervisee node classification is to infer the labels the unlabeled nodes, V.C. The node-wise input feature matrix and the label vector denoted by X Xi....... xv] and Y, respectively.
A GNN model fo is trained to address this problem by considering the node-wise features {x; J and the adjacency matrix A, where 0 represents the learnable parameters. For given node the logit output the GNN is represented as 2, = fo(x;, A) = [ =i, where is the total number of classes. The predicted label for node is given by D1 arg max 2i selecting the class with the highest logit value. The confidence or predicted probability for node i then defined as P. max softmax (2,,)) where softmax(2,,) transforms the logits into probability scores for each class k. We define fo to be perfectly calibrated as (Guo et al. 2017
Chung et 2021
P(u: yo 1 - =) p. vp 10.1.
The calibration error (CE) for graph neural network fo can be defined the discrepancy between the predicted probability P; = max, softmax(2i.x) and the true probability. For node i, the calibra- tion error is expressed
CE(ʃs) = E [| - P(Y = Ds P+]]--
(2)
CB(ff) O, then fo is perfectly calibrated. However, direct computation of the calibration error from the model outputs is challenging in practice. Therefore, the ECE is proposed to approximate CE by discretization (binning) and summation, where the model outputs are partitioned into B intervals (bins) with binning strategy B = (41, 12....... I A common binning strategy 8 divides bins by sorting confidence scores P. in ascending order, ensuring each bin I contains approximately the same number of nodes, i.e., 1191 N
π.
The ECE is defined as the weighted average of the absolute difference between the accuracy and confidence in each bin, denoted by Acc( [2) and Cons(Iz))
B Σ
11 N
B E|y 0s IA E E
14 N
ECE(fo, B)
AcC(IT) - Conf(I,)) =
Σ
SEL;
Importantly, (CE(Je, B) < CE(ff) for any binning strategy, and this approximation is widely used for estimating the overall calibration error (Kumar et al. (019)). By default, we evaluate models with ECB(Jo, B.).
3.2 NOSE-WISE PARAMETRIC CALIBRATION
TS is the classical parametric calibration approach, which adjusts model confidence by scaling logits with a temperature T > before softmax, smoothing outputs without affecting accuracy. The temperature T controls the smoothness of predicted probabilities: T = 1 uses logits directly, while increasing spreads probabilities, improving calibration in overconfident models.
Recent studies extend this by learning node-wise temperatures for each node i in graph classification (Wang et al., 2021] Hsu al., 2022). Commonly, node-wise temperature scaling for calibration is conducted by applying extra GNN layer to learn the node-wise optimal smoothing parameter Ti and do the following operation for the logits:
We should mention that in training the smoothing temperature T, the loss function is always the negative log- likellhoooo function, which categorized to the proper score (Gruber Buettner [022) in the classification task. The famous ECE error can not be utilized as training objective function for their discontinuity. Moreover, these post-hoc calibration methods are usually trained on the validation set. We follow these training settings in our main context.
4 GETS
4.1
ENSEMBLE STRATEGY MEETS NODE-WIST CALIBRATION
Ensemble strategies are widely recognized for improving model robustness and calibration by ag- gregating predictions from multiple models. For example, in the context of ensemble temperature
4

---

Published as a conference paper at ICLR 2025
scaling (ETS) (Zhang et al. 2020), each model learns its own temperature parameter Tm calibrate its predictions. The ensemble strategy combines the calibrated outputs from multiple calibration models, averaging their predictions produce more reliable and better-calibrated result. The ensemble weighting is formulated as a weighted sum over temperature-scaled logits (Zhang et al,) 2020 T(=) E 2m=1 Wm tm (2), where tmm are non-negative coefficients summing to 1, and mm represents the value of temperature of the m-th model TS model trained on logit 2, with M being the number of models.
The ensemble inherits the prreeerii properties from its individual components, with the weights um controlling the contribution of each calibrated prediction. While ETS leverages model diversity to mitigate overconfidence and improve calibration quality, falls short in handling the intricate structural problems inherent in graph data. Specifically, ETS considers only the tempera- tures fm (z), logit 2, and bin size B to optimize the weights, without incorporating graph structural information such as node degrees. Moreover, the learned temperature is single value uniformly applied across all nodes, which cannot adapt to the node-wise calibration required in graphs.
We propose to learn node-wise ensemble strategy for the GNN calibration task. To incorporate the different sources of information, we ensemble based on not only the calibration models but also diverse inputs, including logits 2,, node feature x; and node degrees d,, as well their combination as the calibration model inputs. We use 9m to represent the m-th GNN calibration model that outputs the calibrated logits. Due to the monooone--pesserving property of TS, the weighted summation of the logits still keeps the invariant accuracy output. For each node i, the calibrated logits 2 are computed as a weighted sum of the outputs from multiple calibration models:
Wi,m 9m X1, di 6m),
(5)
where Wi,m 2 O are the node-specific ensemble weights for node i and model m, satisfying Em=1 Wm.m 1. We further point out that in our expert setting, the imput of each expert only contains one element in x,, d;, 2,. This gives reasonable learning tasks for different experts and rules out the problem arising in concatenating vectors of different input spaces. Each calibration model Im can focus on different aspects of the calibration process by utilizing various combinations of inputs. For example, one model may calibrate based solely on the logits, another may incorporate node features, and another may consider the node degrees. Specifically, Wi,m can therefore be deter- mined by a gating function G(.) that outputs a probability distribution over the calibration models for each node:
Wh,1, Wi,2.-.., W. = G((a(= x,, d;; dm:mm))MMMM W Wn),
(6)
where W, and W, represent the parameters of the gating function G(-), which will be introduced later on. We ensemble the feature inputs by concatenating different combinations of inputs into Im (-). The key component that remains unknown is how we should do the information aggregation and obtain the gating function to select models.
We include logits 2, node features x, and degree (embeddings) capture key factors influencing calibration. Including logits is crucial because they contain the raw prediction information before applying softmax, reflecting the model's confidence. Incorporating node features allows the model to address individual calibration needs, as beature-based methods binning methods have successfully captured unique sample point characteristics calibration task (Huang et al., 2022). Adding degree embeddings tackles structural disparities since our experiments show that degree imbalance leads to varying calibration errors among nodes. Note that degrees are the foundation for graph structure information but also explore other structural features inputs in Appendix A..9 By combining these inputs, our ensemble leverages multiple information sources, resulting in a more robust and effective calibration strategy.
4.2 ENSEMBLE CALIBRATION BY GRAPH MoE
The node-wise ensemble idea mentioned in the last section coincides with the so-called MoE frame- work (Jordan& & Jacobs| 1994 Jacobs et al. 1991 Jacobs 1995 Wang et al. 2024). MoE allows for specialization by assigning different experts to handle distinct inputs, enabling each expert to focus on specific factors. This specialization is particularly beneficial when inputs are diverse, as it allows experts to learn the unique properties of each factor more effectively than one single GAT model
5

---

Published as a conference paper at ICLR 2025
Legtz
(1(2:8))
Q1
92(x: 0,)
Q2
Dogan (enbended)
Q3
G()
Σ
(a(d:
Gating function
Calibrated logit
Legite
QM
GOm.. 9m mm)
[w
Expeits (GNN callbrators)
Input Ensemble
Gating scores
Model Ensemble
Figure 2: Illustration of input and model ensemble calibration. The input ensemble explores differ- ent combinations of imput types, while the model ensemble employs MoE framework to select the most effective experts for calibration. The final calibrated outputs are weighted averages determined by the gating mechanism. The notation (-) indicates different input types for the function.
(Hsu et al. 2022) for many inputs at the same time. Moreover, MoE employs sparse activation
through a gating network, which selectively activates only relevant experts for each input, reducing
computational overhead compared to the dense computation in GAT's (Shazeer et al.) 2017 Lepikhin
ctt al. 2020). The overall framework design shown in Figure 2 (Zhang et al. 2020)). To adapt MoE for graph-based calibration tasks, we propose the GETS framework that integrates both the input ensemble and the model ensembles. In GNNs like GCNs, nodes update their representations by aggregating information from their neighbors. The standard GCN propagation rule is (Wang al. 2024):
= ( Σ JEN: Vddelddyl
h W
where hy is the updated feature vector for node after the GCN layer, de is the set of neighbor- ing nodes of node We train different GCNs based on various ensemble of inputs as different experts, d, is the feature vector of the neighboring node j, and W is a learnable matrix for the GCN network.
/dd.lldyl is the normalization term where |d]| and [dy| are the degrees (number of 1
neighbors) of nodes and j. respectively. In many GNN papers, the normalization term is also de- noted by D 1/2 --12 ÄD 1/2 with A A is the adjacency matrix with added self-loops. D is the corresponding degree matrix with D x Σ3 Aij and is non-linear activation function.
While GCNs aggregate neighbor information effectively, they apply the same transformation to all nodes, which may not capture diverse patterns or complex dependencies in the graph. Following our aforementioned notations, the Graph MoE framework introduces multiple experts by learning different GNN models with gating mechanism for aggregation. The propagation rule in Graph MoE training is:
M
h =
Σ
G(h.)m Im (by: 0m))
(8)
where, for consistency with Equation 5 (.) represents the m-th GNN expert function that pro- specific ensemble of inputs-such logits 2;, node features x;, and node degrees. The parameter 8m. associated with expert m, is analogous to the weight matrix W., from Equation 7 (we omit the message-passing aggregation for clarity). The gating mechanism determines the contribution of each expert for node i is
G(h.) Softmax (Qop)(Q(h:))

---

Published as a conference paper at ICLR 2025
with the function, TopK to select the best i experts with gating scores Q(h,). G(h))m is the gating function outputted weights for m-th expert, with h; usually being the last layer output of m-th expert. The gating score Q(h,) is computed as:
Q(h,) h, W c Softplus(h, h, W.),
(10)
where W, and W,, are learnable weight matrices that control clean and noisy scores, respectively. Noisy gating ~ (0, 1) introduces Gaussian noise for exploration. In the context of calibration, we adapt GETS combine multiple calibration experts, each specialized in handling specific cal- ibration factors. For those unselected models the gating function G(-) sets them to zeroes to be consistent with Equation | for example, we select the best experts, the rest M outputs, the gating function G(-) assigns them to O. Finally, for each node i, the calibrated logits z/ are computed as, in a similar form as Equation
M
G(h,)m (m bm))
(11)
Each calibration expert Im can implement different node-wise calibration strategies, where the model usually includes node-wise temperatures. The model trained by minimizing a calibration loss over all nodes: = N Σ feal (z',, y:). where feal is differentiable calibration loss function (e.g., cross-entropy loss) mentioned before and N is the number of nodes. By leveraging GETS, enable the calibration model to specialize in handling diverse calibration factors via different experts and adaptively select the most relevant experts for each node through the gating mechanism using node features and topology.
EXPERIMENTS
5.1 EXPERIMENTAL. SETUP
We include the 10 commonly used graph classification networks for thorough evaluation, The data summary is given in Table I. refer to Appendix [A.2] for their sources. The train-val- val-tst split is 20-10-70 (Hsu al. 2022 Tang al. 22024). note that uncertainty calibration models are trained on the validation set, which is also referred to as the calibration set. We randomly generate 10 different splits of training, validation, and testing inputs and run the models 10 times on different splits.
For the base classification GNN model, i... the uncalibrated model fo. we choose the vanilla GCN (Kipf & Welling 2016), GAT (Veličković et al., (017], and GIN (Xu et al.) 2018). We tune the vanilla models to get optimal classification performance, see Appendix 1. for more details about the parameters. After training, we evaluate models by ECE with 10 equally sized bins.
measure the connectivity of Table 1: Summary of datasets. The average degree is defined as network.
28
Citeseer
Computers
Cora
Cora-full
CS
Ogbn-arxiv
Pholo
Physics
Puluned
Reddit
#Nodes
3,327
13,381
2,708
18,800
18,333
169,343
7,487
34,493
19,717
232,965
#Edges
12,431
491,556
13,264
144,170
163,788
2,501,829
238.087
495,924
108,365
114,848.857
Avg. Degree
7.4
73.4
9.7
15.3
17.8
29.5
63.6
28.7
10.9
98.5
#Features
3,703
767
1,433
8.710
6,805
128
745
8,415
500
602
#Classes
10
7
70
15
40
5
41
All our experiments implemented on a machine with Ubuntu 22.04, with AMD EPYC 9754 128-Core Processors, 1TB RAM, and 10 NVIDIA L40S 48GB GPUs.
5.2 CONFIDENCE CALIBRATION EVALUATION
Baselines. For comparison, we evaluate our approach against several baseline models, including both classic parametric post-hoc calibration methods and graph calibration techniques. The classic methods include TS (Guo al.) 2017 Kull et al.. 2019), Vector Scaling (VS) (Guo et al. 2017), and ETS (Zhang et al. 2020). These baselines are primarily designed for standard i.i.d. clati-class classification.
7

---

Published as a conference paper at ICLR 2025
Table Calibration performance across datasets evaluated by ECE. Results are reported as mean + standard deviation over 10 runs. 'Uncal.' denotes uncalibrated results, and 'OOM' indicates out-of- memory issues where the model could be run. The best results for each dataset are marked bold.
Dataset
Classifier
Uncal.
TS
VS
ETS
CaGCN
GATS
GETS
GCN
20.51 + 2.94
2.82
0.56
3.06 = 0.33
2.79 = 0.37
3,36 = 0.68
2,50 = 1.42
Citeseer
GAT
16.06 + 0.62
2.18 108-0.22
2.89
2.38 + 0.25
2.49 0.47
2.50 = 0.36
1.98 = 0.30
GIN
4.12
2.39 0.32
0.34
2.58 0.34
2,96 # 1.49
2.03 0.23
1.86 0.22
GCN
5.63 1.06
3.29 0.48
2.80+-0.29
3.45 + 0.56
1,90 0.42
3.28 0.52
2.03 4: 0.35
Computers
GAT
1.57
1.83 0.27
2.19++0.15
1.89 + 0.24
1.88 0.36
1.89 0.34
1.77 0.27
GIN
4.00 2.84
3,92 2.59
2.57
2.91 + 2.07
2.37 1.87
3.34 2.08
3.14 3,70
GCN
2.41
3.22 0.81
Cora
16.70
0.39
1.71
2.85 0.66
0.65
0.33
2.20
2.79 0.26
GCN
27.73 = 0.22
4.43 1.43=0.m
3.35
4.34 = 0.09
4.08 0.25
4,46 = 0.17
3.32 = 1.24
Cora-full
GAT
37.51 = 0.22
2:27 = 0.33
0.13
1.37 0.25
3,96 1.15
2.47 0.35
1.52 2.27
GIN
10.69
3.55
3.33 0.22
2.43 0.199
2.18 1: 0.16
4.74 : 0.66
3.98
0.35
1.95 1: 0.37
GCN
0.10
1.49 0.07
1.39 0.100
1.44 1 0.08
2.00 1 0.98
1.51
0.10
1.34 0.10
CS
GAT
0.44
2,98 0.15
3.02 0.18
2,98 0.15
2.57 0.38
2.56
1 0.15
1.05 0.27
GIN
4.364:0.31
4.31 0.31
4.28 0.28
3.04 1.19
1.81 1.09
1.24
0.45
1.15 :: 0.32
GCN
9.77
9.17 1.10
9.29 2299200.88
9.60 0.92
2.32 0.33
2.97 = 0.34
1.85 0.22
Ogbn-arxiv
GAT
6.90
3,56 = 0.10
4,39 0.08
3.55 - 0.10
3.52 0.13
3.90 = 0.77
2.34 0.12
GIN
11.56 0.27
11.35 1 0.27
11.26 1: 0.43
11,38 1: 0.40
6.01 1 0.28
6.41 = 0.29
3.33 1: 0.36
GCN
4,05 0.42
2.14 0.46
2.11 + 0.23
0.34
2,42 0.76
2.24 = 0.30
2.07 = 0.31
Photo
GAT
0.75
2.06 0.57
1.71 + 0.23
2.57 = 0.59
1.69 0.14
2.05 = 0.45
1.10 = 0.25
GIN
2.40
3.12 1.85
3.77 = 1.10
3,34 1.70
2.42 1.90
3.37 = 1.85
1.43 = 0.71
GCN
0.99 0.10
0,96 0.05
0.96 0.05
0.87 + 0.05
1.34 0.45
0.91
0.04
0,87 0.09
Physics
GAT
0.22
0.37 0.10
0.44 0.05
0.47 + 0.22
0.69 0.11
0.48 0.23
0.29 + 0.05
GIN
0.11
2.08 0.14
0.14
1.64
2.36 0.52
0.90 - 0.66
0.44
GCN
1..26
2,45 0.30
2.34
1.93
2.19
Pubmed
0.8.
0.80 0.07
0.95
0.08
0.15
1.01 0.06
1.04
0.47
1.00
0.06
GCN
0.10
1.72 0.07
2.02 - 0.07
1.72 = 0.07
1.50 = 0.08
OOM
1.49 = 0.07
Reddit
GAT
0.15
3,26 0.08
3.43 1
3.52 - 0.10
0.81 0.09
0OM
0.62 = 0.08
GIN
0.09
3.17 0.13
0.10
3.25 1 0.14
1.63 0.21
0OM
1.57 1: 0.12
In addition, we compare against graph-based calibration models that utilize a two-layer GNN to learn temperature values for each node. These include Graph convolution network as a calibration function (CaGCN) (Wang et al. 2021) and Graph Attention Temperature Scaling (GATS) (Hsu et al.) 2022), which use GCN GAT layers to process logits and incorporate graph-specific information for calibration. All models are tuned to their optimal performance, with detailed parameter settings provided in Appendix A.1.3
GETS settings. For the parameters of GETS, we constructed input ensembles for each node i as {Z,, X4, di, [Zi, x;], [x; di]. [z, di), [z,, x,, d,]] We trained M = 7 experts and selected the top experts based on their gating scores, as described in Equation | For the degree input d,, we mapped the integer degree of each node into a dense vector of fixed size 16 using torch. Embedding, before training the experts with this representation.
After parameter tuning, we train the GNN expert 9m (.).. 1,..., M for 1000 epochs with patience of 50 across all datasets. The calibration learning rate is set to 0.1, except for the Reddit dataset, which uses rate of 0.01. Weight decay is set to O for most datasets, except for Citeseer (0.01) and Reddit (0.001). By default, we use a two-layer GCN to train an expert and apply noisy gating with NO0, 1). We also report the ECE for uncalibrated predictions reference.
For all experiments, the pre-trained GNN classifiers are frozen, and the predicted logits = from the validation set are fed into our calibration model as inputs. Further comparison settings and hyperparameters are detailed in the Appendix A.1.2
Results. Table presents the calibration results evaluated by ECE, showing that our proposed method, GETS. consistently achieves superior performance across various classifiers and datasets. On average, GETS improves ECE by 28.60% % over CaGCN, 26.62% over GATS, and 28.09 over ETS across all datasets. While GETS demonstrates significant improvements most datasets,
8

---

Published as a conference paper at ICLR 2025
Possayy rapat
PBT cascx ++0 6A2S
Seumaday eigea
Blarnl laz- IEUUO
Rodii
Hureay ripnt
Soonaleg rager
Ryda
Compents
Company
(Xo)
Duadd Suscelyy Ang. Deped)
(b) Input ensemble selected by sparse activation of experts.
(a) Elapsed time for 10 runs.
Figure 3: Illustration of computational efficiency and expert selection properties. (a): Elapsed time for training each model for 10 runs; (b): Primary and secondary expert selections across datasets for various input ensembles. The top bar plot shows the frequency of expert selection, highlighting the significance of combining logits and features in calibration across datasets.
some exceptions are noted Computers, Cora, and Cora-full, where the gains are less pronounced or slightly negative.
Interestingly, the best results often come from models that incorporate ensemble strategies, such ETS, highlighting the importance of combining these techniques for effective calibration. This underscores the value of our GETS framework, which leverages both ensemble learning and GNN structures to achieve node-specific calibration.
Moreover, GETS proves to be more scalable than GATS, which suffers from out-of-memory (OOM) issues in larger networks like Reddit. These results affirm the robustness and scalability of GETS, showcasing its ability to handle both small and large-scale datasets while significantly reducing cal- ibration error. Overall, GETS demonstrates its effectiveness in addressing node-wise calibration by incorporating input and model ensemble, offering clear improvements over other GNN calibration methods like CaGCN and GATS.
5.3 TIME COMPLEXITY
CaGCN uses two-layer GCN with total time complexity of F [V[F2)), (M({))) feature dimension (Blakely et al. (02T). GATS adds attention mechanisms, leading to complexity of ((2((E)F F))"), with I/ the number of attention heads (Veličković al. 2017). GETS, fundamentally Graph MoE, introduces additional complexity by selecting the top k experts per node, yielding ( (V)MF, where < M. This reduces computational costs compared to using all experts, making GETS scalable while maintaining competitive performance. Training multiple experts might make GETS less time-efficient than its counterpart. However, GETS scales linearly with the number of nodes, and selecting lightweight models like GCN experts helps manage the time complexity effectively. In practice, the GETS model computationally efficient, as shown in Figure Bal even though multiple experts are trained, GETS remains stable elapsed time comparably efficient with the other two models that are trained with single GNN model.
where F is the
5.4 EXPERT SELECTION
One advantage of GETS is its ability to visualize which experts, trained on specific input types, are most important for each node. We illustrate this by coloring each node according to the selected expert for it. As sparse activation in the gating function highlights the significance of different input ensembles, present the top two expert selections (primary and secondary) for each dataset in Fig- ure 3b Gating results are averaged over 10 runs to select the primary and secondary experts.
The results show diverse range of expert choices, with the combination of logits and feature [z, frequently chosen as the primary experts, particularly for complex datasets like Ogbn-arxiv and Red- dir. In contrast, segree-based experts (d) often selected secondary. For smaller datasets such as Cora and Pubmed, individual expert selections are more prevalent. Note that only using logits the inputs is not preferred by the gating function, which indicates the importance of ensembling

---

Published as a conference paper at ICLR 2025
different inputs. These findings indicate that integrating multiple imput types- especially logits and features features- immroves calibration, with the optimal expert combination depending on the complexity of the dataset.
5.5 ABLATION STUDIES
In this section, we evaluate the impact of the backbone calibration model in proposed GETS framework. We investigate whether the choice of GNN backbone (e.g.. GCN, GAT, GIN) signifi- cantly affects calibration performance, offering insights into the robustness of GETS across different architectures. We reuse the parameters tuned for default GETS. For the ablation discussion that fol- lows, we use GCN the default uncalibrated model.
Table 3: Ablation studies on different expert models, measured by ECE.
Expert
Citeseer
Computerss
Corn
Cora-full
CS
Ogbn-arsiv
Photo
Physies
Pubased
Reddit
GRTS-GAT
0.71
0.70
+
4.91
1.82
2.57 2.23
1:66
0OM
GETS-GIN
4.34 # 1,36
4.56 + 3.35
5.53 = 0.59
2,83 # 0.46
2.29 = 0.82
2.48 0.30
4.06 + 2.96
1.16 a: 0.34
2.30 a 0.58
4.64 =: 1,03
Generally, choosing GAT and GIN as the models to train experts does not provide significant ad- vantages using GCN, as shown in Table 3 GETS-GAR yields comparable performance to GATS, and also encounters the same OOM issue in the Reddit dataset, demonstrating its limitations in handling large-scale datasets. Furthermore, while GETS-GIN shows improved results datasets like Cora-full, it underperforms on several other datasets, such as Citeseer, Computers, and Red- dit, compared GCN-based calibration. Notably, in simpler datasets or those with smaller graph structures, GCN tends to achieve better calibration results without overfitting. This suggests that simpler GNN-base calibrator, like GCN, is preferable for training experts it offers a more bal- anced crade-off between model complexity and calibration performance, while avoiding potential issues like overfitting, which may arise when more complex GNN architectures are used.
CONCLUSION
We propose the GETS framework that combines input and model ensemble strategies within a Graph MoE architecture. It effectively incorporates diverse inputs logits, node features, and degree embeddings, well as using a sparse gating mechanism to adaptively select the most relevant experts for each node. This mechanism enhances node-wise calibration performance in general GNN classification tasks.
Our extensive experiments on benchmark GNN datasets demonstrated that GETS consistently outperforms state-of-the-art calibration methods, reducing the expected calibration error signifi- cantly across different datasets and GNN architectures. GETS also proved to be computationally efficient and scalable, effectively handling large-scale graphs without significant overhead. By inte- grating multiple influential factors and leveraging ensemble strategies, GETS enhances the reliability and trustworthiness of GNN prediction results.
ACKNOWLEDGMENTS
Dingyi Zhuang acknowledges the support by the U.S. Department of Energy's Office of Energy Effi- ciency and Renewable Energy (EERE) under the Vehicle Technology Program Award Number DE- EE0009211. Yunhan Zheng acknowledges the support by the National Research Foundation (NRF), Prime Minister's 's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. The Mens, Manus, and Machina (M3S) is an interdisciplinary research group (IRG) of the Singapore MIT Alliance for Research and Technology (SMART) cen- tre. Shenhao Wang acknowledges the support of Research Opportunity Seed Fund (ROSF2023) from the University of Florida.
REFERENCES
Derrick Blakely, Jack Lanchantin, and Yanjun Qi. Time and space complexity of graph convolutional networks. Accessed on: Dec, 31:2021, 2021.
Youngseog Chung. Willie Neiswanger, Ian Char, and Jeff Schneider. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. Advances in Neural Information Processing Systems, 34:10971-1099.. 2021.
10

---

Published as a conference paper at ICLR 2025
Shachi Deshpande, Charles Marx, and Volodymyr Kuleshov. Online calibrated and conformal pre- diction improves bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pp. 1450-1458. PMLR. 2024.
Gianluca Detommaso, Alberto Gasparin, Andrew Wilson, and Cedric Archambeau. Uncertainty cal- ibration in bayesian neural networks via distance-aware priors. arXiv preprini arXiv: 08200,
2022.
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The world wide web conference, pp. 417-426, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning, In international conference on machine learming, pp. 1050-1059. PMLR, 2016.
C Lee Giles, Kurt D Bollacker, and Steve Lawrence. Citeseer: An automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries, pp. 89-98, 1998.
Sebastian Gruber and Florian Buettner. Better uncertainty calibration via proper scores for classifi- cation and beyond, Advances Neural Information Processing Systems, 35:8618-8633 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Will Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs.
Advances in neural information processing systems, 30, 2017.
Hans Hao-Hsun Hsu, Yuesong Shen, Christian Tomani, and Daniel Cremers. What makes graph neural networks miscalibrated? Advances in Neural Information Processing Systems, 35:13775- 13786, 2022.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118 22-133,,, 2020.
Kexin Huang, Ying Jin, Emmanuel Candes, and Jure Leskovec. Uncertainty quantification graph with conformalized graph neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Siguang Huang, Yunli Wang, Lili Mou, Huayue Zhang, Han Zhu, Chuan Yu, and Bo Zheng. Mbct: Tree-based feature-aware binning for individual uncertainty calibration. In Proceedings of the ACM Web Conference 2022, pp.
2022.
Robert A Jacobs. Methods for combining experts' probability assessments. Neural computation, (5):867-888, 1995.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Xinke Jiang* Rihong Qiu*, Yongxin Xu*, Wentao Zhang, Yichen Zhu, Ruizhe Zhang, Yuchen Fang, Xu Chu, Junfeng Zhao, and Yasha Wang. Ragraph: A general retrieval-augmente graph learning framework. NeurIPS, 2024.
Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
Jian Kang, Yan Zhu, Yinglong Xia, Jiebo Luo. and Hanghang Tong. Rawlsgen: Towards rawlsian difference principle on graph convolutional network. Proceedings of the ACM Web Conference 2022, pp. 1214-1225, 2022.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: wet....090009.7 2016.

---

Published as a conference paper at ICLR 2025
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning. pp. 2804. PMLR, 2018.
Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: well-founded and easily im- plemented improvement on logistic calibration for binary classifiers. In Artificial intelligence and statistics, pp. 623-631. PMLR, 2017.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32, 2019.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Jiayuan Luo, Wentao Zhang, Yuchen Fang, Xiaowei Gao, Dingyi Zhuang, Hao Chen, and Xinke Jiang. Timeseries suppliers allocation risk optimization via deep black litterman model. AAAI 2025, 2024.
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 3:127-163, 2000.
Mahdi Pakdaman Naeini, Gregory F Cooper, and Milos Hauskrecht. Binary classifier calibration: Non-parametric approach. arXiv preprint arXiv: 1XXX::1401.3390 2014.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence 2015.
John Platt al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
John Rawls. Atheory of justice. Cambridge (Mass.), 1971.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification network data. Al magazine, 29(3):93-93, 2008
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifica- tion uncertainty. Advances in neural information processing systems, 31, 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expper layer. arXiv preprint orXii:/701.066 2017.
Oleksandr Shchur, Maximilian Mumme, Alcksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. Relational Representation Learning Workshop, NeurIPS 2018, 2018
Boshi Tang, Zhiyong Wu, Xixin Wu, Qiaochu Huang. Jun Chen, Shun Lei, and Helen Meng. Sim- calib: Graph neural network calibration based on similarity between nodes. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 15267-15275, 2024.
Leonardo Teixeira, Brian Jalaian, and Bruno Ribeiro. Are graph neural networks miscalibrated? arXiv preprint arXiv:1905.02296 2019.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: /7/0. 10903, 2017.
12

---

Published as a conference paper at ICLR 2025
Haotao Wang. Ziyu Jiang, Yuning You, Yan Han, Gaowen Liu, Jayanth Srinivasa, Ramana Kom- pella, Zhangyang Wang, et al. Graph mixture of experts: Learning on large-scale graphs with explicit diversity modeling. Advances in Neural Information Processing Systems, 36, 2024.
Ruijia Wang, Xiao Wang, Chuan Shi, and Le Song. Uncovering the structural fairness in graph contrastive learning, Advances in neural information processing systems, 35:32465-32473, 2022.
Xiao Wang, Hongrui Liu, Chuan Shi, and Cheng Yang. Be confident! towards trustworthy graph neural networks via confidence calibration. Advances Neural Information Processing Systems, 14-23768-237799 2021.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: a survey. ACM Computing Surveys, 55(5):1-37, 2022.
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-basee rec- ommendation with graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, pp. 346-353, 2019.
Yuankai Wu, Dingyi Zhuang, Aurelie I Labbe, and Lijun Sun. Inductive graph neural networks for spatiotemporal kriging. Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pp. 4478 4488,444552222 2021.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful graph neural 2018.
networks? arXiv preprint
Jaewon Yang and Jure Leskovec. Defining and evaluating network communities based on ground- truth. In Proceedings of the ACM SIGKDD workshop on mining data semanties, pp. 1-8, 2012.
Bianca Zadrozny and Charles Elkan. Learning and making decisions when costs and probabilities are both unknown. In Proceedings of the seventh ACM SIGKDD international conference Knowledge discovery and data mining, pp. 204-213, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates. In Proceedings of the eighth ACM SIGKDD international conference Knowl- edge discovery and data mining, pp. 694-699, 2002.
Soroush H Zargarbashi, Simone Antonelli, and Aleksandar Bojchevski. Conformal prediction sets for graph neural networks. In International Conference on Machine Learning, pp. 12292-12318. PMLR, 2023.
Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration deep learning. In International conference on machine learning, pp. 11117-11128, PMLR, 2020.
Dingyi Zhuang, Yuheng Bu, Guang Wang, Shenhao Wang, and Jinhua Zhao. Sauc: Sparsity-aware uncertainty calibration for spatiotemporal prediction with graph neural networks. In Temporal Graph Learning Workshop@ NewrIPS 2023, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1
MODEL DETAILS
A.1.1 BASE GNN CLASSIFIER SETTINGS
For the base GNN classification model (i.e., the uncalibrated model), we follow the architecture and parameter setup outlined by Kipf & Welling (2 Veličković al. (2017); Xu al. (2018), with modifications to achieve optimal performance. Specifically, we use two- two-layer GCN, GAT, or GIN model and tune the hidden dimension from the set {16, 32, 64). We experiment with dropout rates ranging from 0.5 to I, and we do not apply any additional normalization.
During training,
learning of 1 10-2 We tune the weight decay parameter to prevent overfitting and consider adding early stopping with patience of 50 epochs. The model is trained for maximum of 200 epochs ensure convergence. The specifics summarized in Table
Table 4: Summary of GNN and Training Parameters
Dataset
Num Layers
Hidden Dimension
Dropout
Training Epochs
Learning Rate
Weight Decay
Citeseer
16
0.5
200
Te-2
5e-4
Computers
64
0.8
200
lc-2
le-3
Cora-Full
64
0.8
200
le-2
le-3
Cora
16
le-2
CS
64
0.8
200
le-2
le-3
Oghn-arxiv
256
0.5
200
le-2
Photo
64
0.8
200
le-2
lo-3
Physics
64
0.8
200
lc.2
le-3
Pubmed
1c-2
Reddit
16
0.5
200
1c-2
5c-4
A.1.2 GETS PARAMETER SETTINGS
The backbone GNN models used for calibration are identical to the base classification models, ensur- ing consistent feature extraction and processing. We tune the hidden dimension for the calibrators from the set (16, 32, 64), depending on the dataset's complexity, while dropout rates are chosen between 0.2 and 0.8 balance regularization and model performance.
GETS includes an expert selection mechanism that incorporates logits, features, and node degree information, either individually in combination, depending the configuration that best fits the dataset. For all experiments, we maintain a fixed bin size of 10 for uncertainty estimation. The calibrators are trained with learning rate of either 1 10 or 1 10 -3 depending on the model's complexity, and we tune weight decay ranging from 0 to 0.1 to regularize the training process further. The calibrators are trained for up 1000 epochs, with early stopping applied based on patience of 50 epochs.
This comprehensive setup ensures robust calibration of the GNN models across diverse datasets.
The full details of the calibration parameters are summarized in Table
Table 5: Summary of GETS Parameters Across Datasets
Dataset
Hidden Dim
Dropout
Num Layers
Learning Rate
Weight Decay
Citeseer
16
0.2
2
0.1
0.01
Computers
16
0.8
2
0.1
Cora-Full
0.8
2
0.1
Cora
0.5
2
0.001
CS
0.8
0.1
Ogbn-arxiv
16
0.2
2
0.1
0.01
Photo
32
0.8
2
0.1
0.001
Physics
16
0.2
2
0.1
0.001
Pubmed
16
0.8
2
0.1
0
Reddit
64
0.8
2
0.01
0.001
14

---

Published as a conference paper at ICLR 2025
A.1.3 BASELINE MODELS SETTINGS
For all the baseline calibration models, we ensure consistent training and calibration settings. The calibration process is set to run for maximum of 1000 epochs with early stopping applied after 50 epochs of no improvement. The calibration learning rate set to 0.01, with no weight decay (0) to avoid regularization effects during calibration. We use 10 bins for the calibration process, and the default calibration method is chosen as Vector Scaling (VS). For the GATS model, we apply two attention heads and include bias terms. Additionally, apply dropout rate of 0.5 during the calibration to prevent overfitting, ensuring robustness across different setups. Full implementations GitHub.
be also found
TS [uoo et al., 2017) adjusts the logits output by model using single scalar temperature parameter 0. The calibrated logits z/ for sample are computed as:
21
(12)
The temperature 7 is learned by minimizing the negative log-likelihood (NLL) loss on the validation (calibration) set:
min (T) - Σ N log softmax 750
7
2
(13)
where Ui (1....,KK) K) is the true label for sample i, N is the number of validation samples.
Vector Scaling (VS) (Guo et al. 2017) extends TS by introducing class-specific temperature scaling and bias parameters. The calibrated logits z/ are computed as:
Ot+bb
(14)
where: E RK is the vector of temperature scaling parameters for each class, b RK vector of bias parameters for each class, and @ denotes element-win wise multiplication. Therefore, the parameters and b are learned by minimizing the NL. loss on the validation set:
K is the
N N
min (t t,b
log Softmma (2t O t b)ys)
(15)
ETS (Zhang al. 2020) combines multiple calibration methods by weighting their outputs im- prove calibration performance. Specifically, ETS combines the predictions from temperature-scaled logits, uncalibrated logits, and uniform distribution. The calibrated probability vector p; for sample i is computed as:
w1 soft
+ w2 softmax(2;) + wz
(16)
T
where: E R is the vector of logits for sample i. T > is the temperature parameter learned from TS, Wi, W2, W3 2 0 are the ensemble weights, satisfying wi + w2 + w3 = 1, and = + R the uniform probability distribution over K classes.
The ensemble weights w = [Wi, W2, wa] are optimized by minimizing the NLL loss:
min LNLL(W) (w)
log (pi.).
(17)
subject Σ Wy 1, Wi for j 1,2,3
15

---

Published as a conference paper at ICLR 2025
CaGCN (Wang et al. 2021) incorporates graph structure into the calibration process by applying GCN to the logits to compute node-specific temperatures. The temperature T; for node is computed as:
T; = softplus (feeN A);
(18)
where R is the matrix of logits for all nodes, A is the adjacency matrix of the graph, faCN (-) is the GCN function mapping from logits to temperature logits, softplus(-) ensures the temperatures are positive, and T, scalar for node i. The calibrated logits follows 12 Note that CaGCN scales the logits multiplicatively with T, where T; > 0, to ensure its accuracy preserving.
The GCN parameters are learned by minimizing the NLL loss on the validation set:
min LILL (0)
Σ N
log softmax (21)
(19)
GATS (Hsu et al. 2022) employs GAT to compute node-specific temperatures, taking into account both the graph structure and node features. The notations are nearly the same as CaGCN, with a difference in the model architecture, from faeN to SGAT. The number of attention heads is set to
A.2 DATASET SOURCE DESCRIPTIONS
We evaluated our method on several widely used benchmark datasets, all accessible via the Deep Graph Library (DGL These datasets encompass variety of graph types and complexities, allow- ing us to assess the robustness and generalizability of our calibration approach.
Citation Networks (Cora, Citeseer, Pubmed, Cora-Full): In these datasets (Sen et al. 2008 Mc-
Callum et al., 2000: Giles al., 1998), nodes represent academic papers, and edges denote citation links between them. Node features are typically bag-of-words representations of the documents, capturing the presence of specific words. Labels correspond the research topics fields of the papers. The Cora-Full dataset is an extended version of Cora, featuring more nodes and larger number of classes, which introduces additional classification challenges.
Coauthor Networks (Coauthor CS, Coauthor Physics): These datasets (Shchur et al. 2018) rep- resent co-authorship graphs where nodes are authors, and edges indicate collaboration between thors. Node features are derived from the authors' published papers, reflecting their research inter- ests. Labels represent the most active field of study for each author. These datasets larger and have higher average degrees compared to the citation networks, testing the models' ability to handle more densely connected graphs.
Amazon Co-Purchase Networks (Computers, Photo): these graphs (Yang & Leskovec 2012), nodes represent products on Amazon, and edges connect products that are frequently co- co-purchased. Node features are extracted from product reviews and descriptions, providing rich textual informa- tion. Labels correspond to product categories. These datasets exhibit strong community structures and higher connectivity, offering a different perspective from academic networks.
Part of the Open Graph Benchmark (Hu et al., 2020), this dataset is large-scale citation network of ArXiv papers. Nodes represent papers, edges denote citation relationships, node features are obtained from paper abstracts, and labels are assigned based on subject areas. Its size and complexity make it suitable for evaluating scalability and performance on real-world, large graphs.
Ogbn-arxiv:
Reddit:
2017) models interactions Reddit. Nodes are posts, and This dataset (Hamilton et al.
edges connect posts if the same user comments on both, capturing user interaction patterns. Node features derived from post content and metadata, and labels correspond to the community (sub- reddit) of each post. Its large size and dense connectivity challenge models to scale efficiently while maintaining performance.
16

---

Published as a conference paper at ICLR 2025
Dataset
Uncalibrated (x10*)
GETS (x10
Improvement (%)
Citeseer
4.46 0.82
2.95 1.08
33.9%
Computers
8.44 1.11
5,86 1:0::000 3.02
30.6%
Cora-full
4.87
51.4%
Cora
2.07 0.17
1.69 0.28
18.3%
CS
+
2.05 0.18
0.0%
Ogbn-arxiv
1.81 0.29
+ +.72
Photo
5.72
2.62 + 0.30
48.8%
Physics
1.29 + 0.12
1,46 + 0.20
Pubmed
4.69 0.28
1.63 = 0.23
65.3%
Reddit
26.38 = 0.45
16.51 = 1.54
37.4%
Table 6: Comparison of VarECE between Uncalibrated and GETS models, with improvement in percentage. Positive values indicate improvement.
A.3 ALGORITHMIC FAIRNESS AMONG DEGREE GROUPS
While fairness is not the primary focus of this paper, model enhances algorithmic fairness by incorporating graph structural information into the calibration process, improving fairness across different degree groups. Although grouping nodes based degrees is nonparametric binning strat- egy egy-which may diverge from our main focus on parametric calibration we include this discussion address calibration
among these groups. This demonstrates that our method promotes fair- ness by achieving similar calibration results across nodes with varying degrees. The metrics and analyses presented can be extended algorithmic fairness studies to evaluate model trustworthi- ness.
P2..... DB} that groups nodes based on their degrees, each containing approximately nodes, where N is the total number of nodes. The average degree of group D, is: AvgDeg(D) (D,) = [D.] EVED, deg(v), where deg(v) the degree of node u. Groups are ordered such that (vg < AvgDeg(D,) for < other topological statistics.
We define binning strategy B* (Di, D2,
j.. Note that B can also be defined using
Kang et al. 2022). we aim to balance calibration errors across degree-basee groups by minimizing their variance. We define the Variation of ECE (VarECE) as fairnsss metric:
Following the Rawlsian Difference Principle Rawls, 1971
VarECE(fo, Var { [Dyl
B
|Ac((D)) - Conf(D,))
(20)
where AC)) and Con)(D,) are the accuracy and confidence on group Dj, respectively.
Results are given in Table 6 By minimizing VarECE, our model promotes fairness across different groups, ensuring that no group particularly those with lower degrees suffers from disproportion- ately higher calibration errors.
A.4 ABLATION OF ENSEMBLE STRATEGIES
If ablate the input ensemble, we still use M 7, but only use the logit for the input. In this case, we have trained same calibration models and do the average.
Table 7: Ablation studies on different input configurations, measured by ECE.
Enpot-AAlaiine
Citsser
Computers
Cara
Cars-flll
CS
Oghn-arsáv
Photo
Phosics
Pubemed
Reddit
GETS
6.70
= 1.79
0:43
5:42
1.78
0.1
2.0 0.34
0.49
If we ablate the model ensemble, the diverse input type is concatenated and is fed into one GNN calibrator, which is reduced to the idea GATS, which is demonstrated to perform less efficiently and effectively than GETS in our previous discussion of Table
https://gitthie http::///ggthhu..co///hhhhhhhhhh htwpa//..oo//ccccrggnnee-eeee
17

---

Published as a conference paper at ICLR 2025
Table 8: GPU reserved memory usage for each algorithm. The OOM case is marked with the attempted reserved memory from the algorithm.
Dataset
Classifier
TS
VS
ETS
CaGCN
GATS
GETS
Citeseer
GCN
164 MB
164 MB
118 MB
164 MB
168 MB
184 MB
Computers
GCN
210 MB
210 MB
190 MB
212 MB
348 MB
250 MB
Cora
GCN
98 MB
98 MB
98 MB
MB
102 MB
110 MB
Cora-full
GCN
1380 MB
1380 MB
1380 MB
1380 MB
1384 MB
2018 MB
CS
GCN
1546 MB
1546 MB
620 MB
1060 MB
1066 MB
1546 MB
Ogbn-arxiv
GCN
3382 MB
3382 MB
3382 MB
1328 MB
2858 MB
2418 MB
Photo
GCN
186 MB
186 MB
130 MB
154 MB
216 MB
186 MB
Physies
GCN
2368 MB
2368 MB
2368 MB
2368 MB
2390 MB
3504 MB
Pubmed
GCN
224 MB
224 MB
224 MB
166 MB
176 MB
224 MB
Reddit
GCN
7208 MB
7208 MB
7208 MB
7206 MB
17.54 GB
7208 MB
A.5 CALIBRATION RELIABILITY DIAGRAM RESULTS
Based on the classifier GCN, the reliability diagrams reflect how the confidence aligns with the accuracies in each bin. The reliability diagram of calibration results of different models is given in Figure and 5
The ECE calculated by degree-based binning is shown in Figure 6
A.6 GPU MEMORY USAGE
We summarize the GPU memory reserved by each of the calibration methods. By default, all the calibration methods the GCN classifier outputs. The memory usage results are shown in
Table g
A.7 SIMCALIB COMPARISON
further compare with work of SimCalib (Tang et al. 2024), as shown in Table
Table 9: BCE Comparison GETS and SimCalib results for various datasets and classifiers. Smaller values are highlighted in bold.
Dataset
Classifier
GETS
SimCalib
Cora
GCN
2.29 +: 0.52
3.32 + 0.99
GAT
2.00 + 0.48
2.90 + 0.87
Citeseer
GCN
2.50 + 1.42
3.94 + 1.12
GAT
1.98 1: 0.30
3.95 1 1.30
Pubmed
GCN
1.90 1 0.40
0.93 1 0.32
GAT
0.78 +: 0.15
0.95 + 0.35
Computers
GCN
2.03 1 0.35
1.37 1 0.33
GAT
1.77 + 0.27
1.08 1 0.33
Photo
GCN
2.07 + 0.31
1.36 + 0.59
GAT
1.10
1.29 1 0.55
CS
GCN
1.34
+ 0.30
GAT
1.05 + 0.27
0.83 + 0.32
Physics
GCN
0.87 + 0.09
0.39 + 0.14
GAT
0.29 + 0.05
0.40 + 0.13
CoraFull
GCN
3.32 + 1.24
3.22 + 0.74
GAT
1.52 + 2.27
3.40 + 0.91
A.8 GATING SCORES VISUALIZATION
We also visualize the stack plot of the gating score changes during training. This will reflect how different experts are preferred during different stages of training. By default, we look into the GETS- GCN case on the GCN classifier.
18

---

Published as a conference paper at ICLR 2025
A.9 ABLATION ON THE INPUT ENSEMBLE
A.9.1 DITTERENT STRUCTURAL INFORMATION
We conducted a test to replace degree embedding with betweenness centrality embedding, page rank centrality, clustering coefficient, harmonic centrality, Katz centrality, and Node2Vec embedding. We conduct experiments all datasets, but some of the datasets statistics. We left those datasets empty due to limited time rebuttal periods.
large to compute the network
The results shown in Table 10 For naming convenience,
GET'S-centallnn and GETS- Node2 to represent different structural information. The original GETS is also repeated here for comparison. By default, wa malkkkeenthh2222 num_wal ks=10, workers=4
the classifier GCN. For Node2Vec embedding. we by default
Table 10: Ablation studies on different structural information, measured by ECE (cleaned values).
Expert
Clesser
Cumpulers
Cura
Cora Full
CS
Oghu-arsá
Photo
Plysies
Pabmee
Reddit
0.10
GETS BETTS-Beteeennee
766 +: 1.35
5.01 3.49
5.20 0.35
3.63 0.68
2.21 0.90
1884 0.1.7
2.18 0.35
Clustering
3.54 4 2.30
0.77
0.05
200
1,15 0.36
0.30
JEUS-PaucRankk
7.22
0.11
1.73 0:122
GFVS-Nole2iee
4.5 1.28
0:99
3.06 8 0.54
3.88 106
1,82 0: 0.89
185 0:40
1.1 13..2.0.0
3.50 0.35
A.9.2 DIFFERENT INPUT TYPES
We further ablate on different input types. We ablate one of the three input types and create the input ensemble based on the other two. For example, we ablate z and then construct the input ensemble use GETS-DX, GETS-DZ and GETS-XZ to
as {x, d, [x, d)}. For the convenience of naming, represent the input ensemble without including 2, x, and d. The results can be found in Table 11 Generally, incorporating more information in the input types would make the results better.
Table 11: Ablation studies different input ensembles, measured by ECE (cleaned values).
Expert
Citeseer
Computers
Cora
Cora-full
CS
Ogbe-arsiv
Photo
Physics
Pubused
Reddit
GRTS
0.35
1.85
0:31
GETS-DX
4.00 0.94
2.76 +: 1140
3:27 0.62
3.60 1:008:0..4 0.54
1,86 + 0.21
2.15 + 0.46
2.12 0.42
1.66 + 0.08
2.0 0.35
1.55 4: 0.25
0.34
0.16
GETS-NZ
299 0:72
3 1.47
3.09 0.47
3.26 0.40
1,94 2 0.52
2.24 0.39
1.33 4:338:00 0.35
1,02 + 0.177
2.56 0.55
2.28 # 0.54
A. ABLATION ON THE EXPERTS
We extend the Table 3 by including Multi-layer Perceptron (MLP) as the backbone model with the same layer number for training the algorithm. MLP is structune-unaware, which serve as the baseline. The MLP results show that the selection of the backbone methods for the experts does not necessarily require graph-based models, which offer the opportunity include a broader range of models for the graph calibration task.
Table 12: Ablation studies on different expert models, measured by ECE.
Expori
Computees
Cara-full
Puhmed
Reddit
GETS
2.50.1 1.42
200 1 0.35
2.29 1991.00 0.52
333 1 1.24
1.34 14.11.0 0.10
1,85 0.22
2.07 1 0.31
087 0:09
1,00 1 0.40
1.49 1 0,07
GETS-GAT
1.94
3.93
188
0.59
GETS-GIN
4.34 1.36
4.56 + 3.35
5.53 0.59
233 0.46
2.29 + 0.2
2,48 + 0.30
4.06 296
1.16 0.14
2.30 +
4.64 1,03
GETS-MLP
1.02
0.10
1.38 0.46
0:99
0.45
19

---

Published as a conference paper at ICLR 2025
Clncea
Car
mcLibardd
(a) Reliability diagram of CaGCN.
1.0
Corp
Cowital
92
Coffence
(b) Reliability diagram of GATS.
10
Chooe
Comsuer
Coraul
Dow
3.
Confoonce
Conferics
ComMerts
Carfoare
Phuto
Flydic
Dovitee
Cerffrncs
Confoonce
Conferce
Car
(c) Reliability diagram of GETS.
1.0
1.0
Confornce
Conterte
Contbare
(d) Reliability diagram of TS.
Figure Reliability diagrams of GaGCN, GATS, GETS, and TS.
20

---

Published as a conference paper at ICLR 2025
LO
Chone
Coraus
CHPAI
Cow
Genter
Feurter
Physics
Confiancs
Confeence
Conferice
(a) Reliability diagram of VS.
1.0
3.5
Confoonce
Conferics
Conterts
Carteare
Phato
Fhydic
FLemed
Doiiae
(b) Reliability diagram of ETS.
Figure 5: Reliability diagrams VS and ETS.
21

---

Published as a conference paper at ICLR 2025
210
2
GA1S
(4) GATS
.ATS
GATS
045
22000 MOT
C0rpeae 3il
440 MA8
Mill
146
002
8800
840
Plch
Nocke
8.00
161
0.2
Opioraais
2lan 36.6
Foiural MGA
160
Figure 6: Visualization of the ECE performance of three deep learning models (CaGCN, GATS, and GETS) across various datasets. Confidence bins are sorted by degree. Note that GATS encountered an out-of-mon of-memory issue the Reddit dataset, resulting in a blank figure for that case.
22

---

Published as a conference paper at ICLR 2025
Data: dileeeer
Dalk cimputecs
10
Festan
Portbo
6W
Purtao
Boggs eroon
64
64
62
64
00
0.0
Training aptondd
Training
(a) Citescer
(b) Citescer
Data: coCK
Dasts com--illl
1.0
Destans
Degnns
[Fes
Purtkon
Purtoo
XX
[Loghs enomn
64
64
6/2
64
00
000
splunde
Training
(c) Cora
(d) Cora-full
Tais: egin-erziv
1.0
0X
Degrns
[Fes
Purtion
Purtan
Ronges croon
044
6.4
0.2
62
0.0
0.0
Training epliodde
Training
(c) CS
(f) Ogbn-arxiv
Data: photo
Date: phystrx
1.0
Destarss
Deginss
Pertinn
Purtan
6X
(Logghs Di greon)
[Laapin,
04
644
0/2
62
0.0
6.0
Training epiiodd
Training epliode
Photo
(h) Physics
Tivte: pobmed
Theb: modile
1.0
Degrass
Pwrtin
Purton
(ooghs De groan
[Loptts Deggnest
04
64
6.2
62
00
6.0
Training episode
Training
(i) Pubmed
(i) Reddit
Figure 7: GETS gating scores changes during training.
23


Reviews:
[{"review_id": "rQ92P4589K", "reviewer": "Reviewer_sXvB", "summary": "The paper proposes graph ensemble temperature scaling (GETS), a novel calibration technique for node-classification GNNs.\nGETS is a parametric calibration method which calibrates the predictions for each node in a graph by predicting a temperature for each node which is used as a factor that is applied to the respective class logits.\nGETS predicts node-wise temperatures using a set of GNN-based calibration models trained on different features (predicted logits, node degrees, node features or combinations thereof). The temperatures returned by the GNN calibrators are then combined via a learned gating function which selects a top-k subset of the calibrators for each vertex.\n\nTo summarize, GETS combines the previously proposed ensemble temperature scaling (ETS) method with the idea of using a GNN to predict structure-aware temperatures (e.g., CaGCN or GATS).", "questions": "Apart from the suggestions/questions described in the weaknesses, I only have two minor questions:\n1. You report that the GATS calibration approach suffers from OOM problems not present in the other methods. In Section 5.3 you note that the main difference between CaGCN and GATS, time-complexity-wise, lies in the addition of the attention heads $H$. If all computations are parallelized in memory, I would therefore assume the space complexity to be similar. However, in supplement A1.3, you state that you used $H=2$, which appears to be reasonably small, especially considering the fact that $H$ is only a linear factor in the complexity. Does GATS really require orders of magnitude more memory than the other approaches or were the OOM-issues caused by GATS requiring slightly more memory than available? If the former, do you have an explanation as to why this is the case?\n2. As I understand, GETS performs temperature scaling (TS) for each vertex. In your experiments, you also evaluated vector scaling (VS). Have you also considered a variant of GETS that uses VS instead of TS?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As reported in the experiments, the combination of structural information and a mixture of calibrators trained on different sets of features seems to clearly outperform previously proposed calibration techniques for node classifiers.\n\nThis combination of ideas is well motivated by the paper and the claims regarding the advantages of GETS are supported by the broad range of used datasets.\n\nPresentation-wise, the paper is well structured and the ideas are presented concisely and clearly.", "weaknesses": "As mentioned, the reported advantages of the GETS approach seem to stem from the combination of two ideas:\n1. Predicting node-wise temperatures based on class logits, node features and node degrees.\n2. Incorporating structural dependencies between vertices to predict temperatures.\n\nBoth ideas have (to some extent) been utilized independently in prior work. While the performed experiments show that the combination of both ideas works well in practice, it only gives limited insight into why exactly this is the case.\n\nMore specifically, I believe it would have been interesting to investigate the following questions:\n1. *What is the influence of the three types of features used by the experts ($z$, $x$ and $d$)?*\n   Figure 3b only partially answers this question by showing how frequently the different experts are selected by the gating function. It does not show to which extent the different features are able to substitute each other, i.e., it would be interesting to investigate ablations of GETS without different subsets of experts/features being part of the pool.\n2. *How well does a GETS-like MoE approach with a structure-unaware MLP instead of a GNN work?* While the evaluated ETS calibration approach also uses the idea of combining multiple calibrators, those calibrators do not use the combinations of $z$, $x$ and $d$ used by GETS. Isolating the effect of this choice of features on the calibration performance would be interesting. A related work in this direction by Liu et al. (2022) [1] could be considered here.\n3. *Are other structural features than the degree count $d$ also useful for calibration?* There is an overlap in the information provided by the node degree inputs and the graph convolutions performed by each expert. As an addition to the previous point, it would be interesting to investigate whether other structural features (e.g., other node centrality measures or node2vec-like embeddings) can serve as effective surrogates to the structural information that is incorporated via the graph convolutions in each calibrator.\n\nBy addressing (at least some) of these questions, I believe, that the paper could shed a more nuanced light on the importance of different features for calibration in the node classification setting.\n\n---\n[1] Liu, T. et al.: On Calibration of Graph Neural Networks for Node Classification. In: 2022 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (2022). [https://doi.org/10.1109/IJCNN55064.2022.9892866](https://doi.org/10.1109/IJCNN55064.2022.9892866).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HJ4dhngFfg", "reviewer": "Reviewer_XSU8", "summary": "This paper addresses a critical challenge in graph node classification: the issue of calibration. The authors propose a  method that integrates a combination of features, logits, and node degree within a framework of expert models, aiming to enhance the calibration performance in graph model. By innovatively focusing on the calibration aspect, the study offers new insights into improving the reliable of graph neural networks.", "questions": "My questions can be found in weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The authors propose a method that leverages node features, logits, and node degree, combined with a framework of  expert models, to enhance the reliability of node predictions. Through extensive experiments, the authors demonstrate the effectiveness of their method compared to existing baselines.", "weaknesses": "1. This paper lacks adequate theoretical backing for why using features, logits, and degree with experts can effectively address calibration issues. Additionally, it does not explain why other node factors are not considered. This needs to be clarified.\n\n2. The paper does not explain how the chosen experts can resolve calibration issues, nor does it discuss why a separate model is not designed for each input. This lack of innovation and theoretical justification may undermine the reliability of the research.\n\n3. The baseline models used in the experiments, particularly the graph calibration model baselines, are outdated. For instance, CAGCN was published in NeurIPS 2021 and GATS in 2022. It is recommended that the authors update their baseline models to enhance the comparability and validity of their approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vzToLrKj1e", "reviewer": "Reviewer_uyzv", "summary": "This work leverages multiple GNNs with different inputs within a Graph Mixture-ofExperts (MoE) architecture for confidence calibration of GNNs.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Paper is written well and easy to read.\n\n- The proposed framework is intuitional and the design choices are justified. \n\n- Experimental results are thorough and enough empirical analyses are provided.\n\n- Proposed method seems to perform well compared to the selected baselines.", "weaknesses": "- More recent baselines can be included to further strengthen experimental results.\n\n- The novelties of the proposed framework seem to be rather limited based on the current presentation. It would be better to clearly emphasize the novelties over the existing solutions in the Related Work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MwuZNtQ2bi", "reviewer": "Reviewer_aQtF", "summary": "This paper introduces Graph Ensemble Temperature Scaling (GETS), a calibration framework for Graph Neural Network that combines input and model ensemble strategies with a Graph Mixture-of-Experts architecture. GETS integrates multiple types of inputs and adaptively selects relevant experts for calibrating such node's predictions. The framework achieves improvements in calibration performance across 10 benchmark datasets.", "questions": "1. Could the authors provide quantitative analysis of expert specialization during the training? This could be potentially done by \n (1) Visualization of expert activation patterns (investigation of gating function values) across training epochs, or\n (2) Per-expert contribution to the total calibration loss over training epochs, or\n (3) Show how the preference for certain experts emerges and stabilizes (again through analyzing the gating function G or gating scores Q)\nWhile Figure 3(b) demonstrates the final expert selection patterns through sparse activation, a dynamic analysis of expert contribution to performance would be valuable.\n\n2. How does the architecture of individual experts impact calibration performance? Given that experts process different input types, different architecture may be more suitable for each input modality. Currently, GCN with 2 layers is used for all types except the integer degree. Does the same GCN architecture uniformly reflect the effects across the experts of different types?\n\n3. How does GETS perform on heterogeneous graphs where nodes have different feature types or varying degrees of connectivity?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The comprehensive experimental results in Table 2 show consistent improvements across different architectures and datasets.\n* The sparse activation mechanism described in Section 4.2 enables efficient scaling to larger graphs. \n* The well-presented analysis of expert selection patterns provides insights into which input combinations are most effective for calibration for each dataset.", "weaknesses": "* The hyperparameter sensitivity analysis is somewhat limited, say regarding the number of experts, or the expert embedding dimension. For example, \n (1) How do calibration performance and computation costs scale with the number of experts (M) ?\n (2) What is the impact of expert network capacity (e.g. the number of neurons, the number of layers) on calibration quality?\n (3) How does the choice of top-k in sparse gating affect the trade-off between performance and efficiency? An empirical experiment on synthetic dataset would provide valuable implementation guide. \n\n* While computational efficiency is emphasize, the need to store multiple expert models could be problematic for resource-constrained settings. This practical limitation deserves more discussion. For instance, computational memory requirements for different types of experts handling various input might be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)