GETS: Ensemble Temperature Scaling for Calibration in Graph Neural Networks

## Abstract

Abstract Graph Neural Networks (GNNs) deliver strong classification results but often suffer from poor calibration performance, leading to overconfidence or underconfidence.
This is particularly problematic in high-stakes applications where accurate uncertainty estimates are essential.
Existing post-hoc methods, such as temperature scaling, fail to effectively utilize graph structures, while current GNN calibration methods often overlook the potential of leveraging diverse input information and model ensembles jointly.
In the paper, we propose Graph Ensemble Temperature Scaling (GETS), a novel calibration framework that combines input and model ensemble strategies within a Graph Mixture-of-Experts (MoE) architecture.
GETS integrates diverse inputs, including logits, node features, and degree embeddings, and adaptively selects the most relevant experts for each node’s calibration procedure.
Our method outperforms state-of-the-art calibration techniques, reducing expected calibration error (ECE) by ≥ 25 % absent percent 25 \geq 25\% ≥ 25 % across 10 GNN benchmark datasets.
Additionally, GETS is computationally efficient, scalable, and capable of selecting effective input combinations for improved calibration performance.
Codes are available when published.

## 1 Introduction

Graph Neural Networks (GNNs) have emerged as powerful tools for learning representations in numerous real-world applications, including social networks, recommendation systems, biological networks, and traffic systems, achieving state-of-the-art performance in tasks like node classification, link prediction, and graph classification (Kipf & Welling, ; Veličković et al., ).
Their ability to capture complex relationships and dependencies makes them invaluable for modeling and predicting interconnected systems (Fan et al., ; Wu et al., ; Scarselli et al., ; Wu et al., ; ).

However, ensuring the reliability and trustworthiness of GNN prediction remains a critical challenge, especially in high-stakes applications where decisions can have significant consequences (e.g. human safety).
Take GNN classification as an example, one key aspect of model reliability refinement is uncertainty calibration, which aims to align the predicted probabilities with the true likelihood of outcomes (Guo et al., ).
In short, well-calibrated models provide confidence estimates that reflect real-world probabilities, which is essential for risk assessment and informed decision-making (Zhuang et al., ).

Calibration methods are typically applied post-hoc and fall into two categories: nonparametric and parametric.
Nonparametric methods, like histogram binning and isotonic regression, adjust confidence estimates based on observed data distributions without assuming a specific functional form (Zadrozny & Elkan, ; Naeini et al., ).
While allowing for the freedom in modeling, these methods often require large calibration datasets and struggle with high-dimensional or unevenly distributed graph data, making them less practical for graphs with vast numbers of nodes.
In contrast, parametric methods such as temperature scaling (TS) and Platt scaling assume a functional relationship to adjust model outputs (Guo et al., ; Platt et al., ). Among them, TS has gained popularity for its simplicity and effectiveness in calibrating models by smoothing predicted probabilities.
This method is scalable, and easy to implement, especially for large-scale graphs.
Moving beyond the uniform adjustments applied to all data in classic TS, recent work focuses on learning node-specific calibration parameters to capture individual characteristics that impact graph calibration performance (Tang et al., ; Hsu et al., ).

Motivated by the methodologies discussed above, when we dive into the GNN (classification) calibration task, we find that the subtleties lie in two main challenges: (i) the difficulty of representing the calibration error score for the individual sample points; (ii) the disability to identify and utilize multiple information related to the GNN calibration.
Existing approaches for graph temperature scaling primarily rely on an additional GNN model to output node-wise temperatures. However, systematic methods for determining the inputs to the calibration network and for integrating various input information have not been thoroughly explored.

Given the challenges we have, we aim to resolve the following issues:
(i) What are the influential factors in the calibration task for GNN classification? (ii) How can we wisely integrate these factors, which may exist in different spaces and scales, to effectively perform GNN calibration?

Figure: Figure 1: Expected calibration error (ECE), see Equation 3, for a well-trained CaGCN model (Wang et al., ). ECE is measured grouping nodes based on degree ranges rather than predicted confidence. Datasets are sorted by average degree $\frac{2|\mathcal{E}|}{|\mathcal{V}|}$ as a measure of connectivity (see Table 1), from low to high. In lower-connectivity datasets like Pubmed and Cora-full, high-degree nodes tend to show larger calibration errors, whereas in more connected datasets like Photo and Ogbn-arxiv, low-degree nodes exhibit higher calibration errors.
Refer to caption: x1.png

For issue (i), we note that besides the logits and features mentioned in the prior work (Wang et al., ; Huang et al., ), Figure 1 demonstrates that node degree also affects the calibration error, aligning with the message-passing mechanism in GNNs.
In some cases, a low degree can lead to worse calibration performance due to insufficient information passing.
Conversely, overconfidence may occur when excessive (potentially misleading) information floods into a single node
(Wang et al., ).
Regarding issue (ii), it remains unclear how to develop an appropriate algorithm that incorporates multiple factors for calibration within the graph structure.
To address these issues, we propose a novel calibration framework named Graph Ensemble Temperature Scaling (GETS).
Our approach leverages both input ensemble and model ensemble strategies within a Graph Mixture-of-Experts (MoE) architecture to address the limitation of existing methods.
By incorporating multiple calibration experts—each specializing in different aspects of the influential factors affecting calibration—GETS can adaptively select and combine relevant information for each node.

Experimental results on 10 GNN benchmark datasets demonstrate that GETS consistently achieves superior calibration performance, significantly reducing the ECE across all datasets.
This highlights its effectiveness in addressing the calibration problem in the GNN classification task.
Additionally, GETS is scalable and capable of handling large-scale datasets with computational efficiency.

Our key contributions are summarized as:

- •
We introduce a novel framework GETS that combines input and model ensemble strategies within a Graph MoE architecture. This enables robust node-wise calibration by integrating diverse inputs like logits, features, and degree embeddings.
- •
GETS significantly reduces ECE across 10 GNN benchmark datasets, outperforming state-of-the-art methods like CaGCN and GATS, and demonstrating effectiveness across diverse graph structures.
- •
GETS employs the sparse gating mechanism to select the top $k$ experts per node. This enables scalable calibration on large graphs with computational efficiency.

## 2 Related Work

### 2.1 Uncertainty Calibration

Uncertainty calibration adjusts a model’s predicted probabilities to better reflect the true likelihood of outcomes.
Methods for uncertainty calibration are generally classified into in-training and post-hoc approaches.

In-training calibration adjusts uncertainty during model training.
Bayesian methods, such as Bayesian Neural Networks (BNNs) and variational inference, adjust uncertainty by placing distributions over model parameters (Gal & Ghahramani, ; Sensoy et al., ; Detommaso et al., ).
Frequentist approaches, including conformal prediction (Zargarbashi et al., ; Huang et al., ; Deshpande et al., ) and quantile regression (Chung et al., ), are also employed to ensure calibrated outputs.

Despite these efforts, model misspecification and inaccurate inference often lead to miscalibration (Kuleshov et al., ).
To address this, post-hoc methods recalibrate models after training.
These can be divided into nonparametric and parametric approaches. Nonparametric methods, such as histogram binning (Zadrozny & Elkan, ), isotonic regression (Zadrozny & Elkan, ), and Dirichlet calibration (Kull et al., ), adjust predictions flexibly based on observed data (Naeini et al., ).
Parametric methods assume a specific functional form and learn parameters that directly modify predictions.
Examples include temperature scaling (Guo et al., ), Platt scaling (Platt et al., ), beta calibration (Kull et al., ) and so on.

### 2.2 GNN Calibration

Calibration of GNNs remains an under-explored area (Zhuang et al., ).
Take the node classification task as an example, most existing methods adopt post-hoc approaches, often relying on temperature scaling where the temperature is learned via a certain network.

Teixeira et al. () noted that ignoring structural information leads to miscalibration in GNNs.
GNNs also tend to be underconfident, unlike most multi-class classifiers, which are typically overconfident.
To address this, CaGCN introduces a GCN layer on top of the base GNN for improved uncertainty estimation using graph structure (Wang et al., ).
Hsu et al. () identified key factors affecting GNN calibration and proposed GATS, which employs attention layers to account for these factors.
Tang et al. () further explored node similarity within classes and proposed ensembled temperature scaling to jointly optimize accuracy and calibration.

## 3 Problem Description

### 3.1 Uncertainty Calibration on GNN

This paper addresses the calibration of common semi-supervised node classification tasks.
Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$ represent the graph, where $\mathcal{V}$ is the set of nodes, and $\mathcal{E}$ is the set of edges.
The adjacency matrix is denoted by $\mathbf{A}\in\mathbb{R}^{N\times N}$, with $N=|\mathcal{V}|$ being the number of nodes.
Let $\mathcal{X}$ be the input space and $\mathcal{Y}$ be the label space, with $y_{i}\in\mathcal{Y}=\{1,\dots,K\}$, where $K\geq 2$ is the number of classes.
Let $X\in\mathcal{X}$ and $Y\in\mathcal{Y}$ be the input feature and label.
A subset of nodes with ground-truth labels $\mathcal{L}\subset\mathcal{V}$ is selected for training.
The goal of semi-supervised node classification is to infer the labels of the unlabeled nodes, $\mathcal{U}=\mathcal{V}\backslash\mathcal{L}$.
The node-wise input feature matrix and the label vector are denoted as $\mathbf{X}=[\mathbf{x}_{1},\dots,\mathbf{x}_{N}]^{\top}$ and $\mathbf{Y}$, respectively.

A GNN model $f_{\theta}$ is trained to address this problem by considering the node-wise features $\{\mathbf{x}_{i}\}_{i:v_{i}\in\mathcal{L}}$ and $\mathbf{A}$, where $\theta$ represents the learnable parameters.
For a given node $i$, the logit output of the GNN is represented as $\mathbf{z}_{i}=f_{\theta}(\mathbf{x}_{i},\mathbf{A})=[z_{i,1},\dots,z_{i,K}]^{\top}$, where $K$ is the total number of classes.
The predicted label for node $i$ is given by $\hat{y}_{i}=\arg\max_{k\in[1,K]}z_{i,k}$, selecting the class with the highest logit value.
The confidence or predicted probability for node $i$ is then defined as $\hat{p}_{i}=\max_{k\in[1,K]}\operatorname{softmax}(z_{i,k})$, where $\operatorname{softmax}(z_{i,k})$ transforms the logits into probability scores for each class $k$.
We define $f_{\theta}$ to be perfectly calibrated as (Guo et al., ; Chung et al., ):

$$ $\mathbb{P}(\hat{y}_{i}=y_{i}\mid\hat{p}_{i}=p)=p,\quad\forall p\in[0,1].$ (1) $$

The calibration error (CE) for a graph neural network $f_{\theta}$ can be defined as the discrepancy between the predicted probability $\hat{p}_{i}=\max_{k}\operatorname{softmax}(z_{i,k})$ and the true probability. For node $i$, the calibration error is expressed as:

$$ $\text{CE}(f_{\theta})=\mathbb{E}\left[\left|\hat{p}_{i}-\mathbb{P}(Y=\hat{y}_{ i}\mid\hat{p}_{i})\right|\right].$ (2) $$

If $\text{CE}(f_{\theta})=0$, then $f_{\theta}$ is perfectly calibrated.
However, direct computation of the calibration error from the model outputs is challenging in practice.
Therefore, the expected calibration error (ECE) is proposed to approximate CE by discretization (binning) and summation, where the model outputs are partitioned into $B$ intervals (bins) with binning strategy $\mathcal{B}=\{I_{1},I_{2},\dots,I_{B}\}$.
A common binning strategy $\mathcal{B}^{0}$ divides bins by sorting confidence scores $\hat{p}_{i}$ in ascending order, ensuring each bin $I^{0}_{j}$ contains approximately the same number of nodes, i.e., $|I^{0}_{j}|\approx\frac{N}{B}$.

The ECE is defined as the weighted average of the absolute difference between the accuracy and confidence in each bin, denoted as $Acc(I_{j})$ and $Conf(I_{j})$:

$$ $\text{ECE}(f_{\theta},\mathcal{B})=\sum_{j=1}^{B}\frac{|I_{j}|}{N}\left|Acc(I_ {j})-Conf(I_{j})\right|=\sum_{j=1}^{B}\frac{|I_{j}|}{N}\left|\mathbb{E}[Y=\hat {y}_{i}\mid\hat{p}_{i}\in I_{j}]-\frac{1}{|I_{j}|}\sum_{i\in I_{j}}\hat{p}_{i} \right|.$ (3) $$

Importantly, $ECE(f_{\theta},\mathcal{B})\leq CE(f_{\theta})$ for any binning strategy, and this approximation is widely used for estimating the overall calibration error (Kumar et al., ).
By default, we evaluate models with $\text{ECE}(f_{\theta},\mathcal{B}^{0})$.

### 3.2 Node-wise Parametric Calibration

TS is the classical parametric calibration approach, adjusts model confidence by scaling logits with a temperature $T>0$ before softmax, smoothing outputs without affecting accuracy.
The temperature $T$ controls the smoothness of predicted probabilities: $T=1$ uses logits directly, while increasing $T$ spreads probabilities, improving calibration in overconfident models.

Recent studies extend this by learning node-wise temperatures for each node $i$
in graph classification (Wang et al., ; Hsu et al., ).
Commonly, node-wise temperature scaling for calibration is conducted by applying an extra GNN layer to learn the node-wise optimal smoothing parameter $T_{i}$ and do the following operation for the logits:

$$ $z_{i}^{\prime}=z_{i}/T_{i}.$ (4) $$

We should mention that in training the smoothing temperature $T$, the loss function is always the negative log-likelihood function, which is categorized to the proper score (Gruber & Buettner, ) in the classification task. The famous ECE error can not be utilized as a training objective function for their discontinuity.
Moreover, these post-hoc calibration methods are usually trained on the validation set.
We follow these training settings in our main context.

## 4 GETS

### 4.1 Ensemble Strategy Meets Node-wise Calibration

Ensemble strategy are widely recognized for improving model robustness and calibration by aggregating predictions from multiple models.
For example, in the context of ensemble temperature scaling (ETS) (Zhang et al., ), each model learns its own temperature parameter $T_{m}$ to calibrate its predictions.
The ensemble strategy combines the calibrated outputs from multiple calibration models, averaging their predictions to produce a more reliable and better-calibrated result.
The ensemble weighting is formulated as a weighted sum over temperature-scaled logits (Zhang et al., ):
$T(z)=\sum_{m=1}^{M}w_{m}\cdot t_{m}(z)$,
where $w_{m}$ are non-negative coefficients summing to 1, and $t_{m}(z)$ represents the value of temperature of the $m$-th model TS model trained on logit $z$, with $M$ being the number of models.

The ensemble inherits the accuracy-preserving properties from its individual components, with the weights $w_{m}$ controlling the contribution of each calibrated prediction.
While ETS leverages model diversity to mitigate overconfidence and improve calibration quality, it falls short in handling the intricate structural problems inherent in graph data.
Specifically, ETS considers only the temperatures $t_{m}(z)$, logit $z$, and bin size $B$ to optimize the weights, without incorporating graph structural information such as node degrees.
Moreover, the learned temperature is a single value uniformly applied across all nodes, which cannot adapt to the node-wise calibration required in graphs.

We propose to learn node-wise ensemble strategy for the GNN calibration task.
To incorporate the different sources of information, we ensemble based on not only the calibration models but also diverse inputs, including logits $z_{i}$, node feature $\mathbf{x}_{i}$ and node degrees $d_{i}$, as well as their combination as the calibration model inputs.
We use $g_{m}$ to represent the $m$-th GNN calibration model that outputs the calibrated logits.
Due to the monotone-preserving property of TS, the weighted summation of the logits still keeps the invariant accuracy output.
For each node $i$, the calibrated logits $z_{i}^{\prime}$ are computed as a weighted sum of the outputs from multiple calibration models:

$$ $z_{i}^{\prime}=\sum_{m=1}^{M}w_{i,m}\cdot g_{m}(z_{i},\mathbf{x}_{i},d_{i}; \theta_{m}),$ (5) $$

with $w_{i,m}\geq 0$ are the node-specific ensemble weights for node $i$ and model $m$, satisfying $\sum_{m=1}^{M}w_{i,m}=1$. We further point out that in our expert setting, the input of each expert only contains one element in $\mathbf{x}_{i},d_{i},z_{i}$. This gives reasonable learning tasks for different experts and rules out the problem arising in concatenating vectors of different input spaces.
Each calibration model $g_{m}$ can focus on different aspects of the calibration process by utilizing various combinations of inputs.
For example, one model may calibrate based solely on the logits, another may incorporate node features, and another may consider the node degrees.
Specifically, $w_{i,m}$ can therefore be determined by a gating function $G(\cdot)$ that outputs a probability distribution over the calibration models for each node:

$$ $[w_{i,1},w_{i,2},\dots,w_{i,M}]=G(\{g(z_{i},\mathbf{x}_{i},d_{i};\theta_{m})\} _{m=1}^{M};\mathbf{W}_{g},\mathbf{W}_{n}),$ (6) $$

where $\mathbf{W}_{g}$ and $\mathbf{W}_{n}$ represent the parameters of the gating function $G(\cdot)$, which will be introduced later on.
We ensemble the feature inputs by concatenating different combinations of inputs into $g_{m}(\cdot)$.
The key component remains unknown is how we should do the information aggregation and obtain the gating function to select models.

We include logits $z$, node features $\mathbf{x}$, and degree (embeddings) $d$ to capture key factors influencing calibration.
Including logits is crucial because they contain the raw prediction information before applying softmax, reflecting the model’s confidence.
Incorporating node features allows the model to address individual calibration needs, as feature-based methods binning methods have successfully captured unique sample point characteristics in calibration task (Huang et al., ).
Adding degree embeddings tackles structural disparities since our experiments show that degree imbalance leads to varying calibration errors among nodes.
By combining these inputs, our ensemble leverages multiple information sources, resulting in a more robust and effective calibration strategy.

### 4.2 Ensemble Calibration by Graph MoE

Figure: Figure 2: Illustration of input and model ensemble calibration. The input ensemble explores different combinations of input types, while the model ensemble employs a MoE framework to select the most effective experts for calibration. The final calibrated outputs are weighted averages determined by the gating mechanism. The notation $(\cdot)$ indicates different input types for the function.
Refer to caption: x2.png

The node-wise ensemble idea mentioned in the last section coincides with the so called MoE framework (Jordan & Jacobs, ; Jacobs et al., ; Jacobs, ; Wang et al., ).
MoE allows for specialization by assigning different experts to handle distinct inputs, enabling each expert to focus on specific factors.
This specialization is particularly beneficial when inputs are diverse, as it allows experts to learn the unique properties of each factor more effectively than one single GAT model (Hsu et al., ) for many inputs at the same time.
Moreover, MoE employs sparse activation through a gating network, which selectively activates only relevant experts for each input, reducing computational overhead compared to the dense computation in GATs (Shazeer et al., ; Lepikhin et al., ).
The overall framework design is shown in Figure 2 (Zhang et al., ).

To adapt MoE for graph-based calibration tasks, we propose the GETS framework that integrates both the input ensemble and the model ensembles.
In GNNs like GCNs, nodes update their representations by aggregating information from their neighbors. The standard GCN propagation rule is (Wang et al., ):

$$ $\mathbf{h}_{i}^{\prime}=\sigma\left(\sum_{j\in\mathcal{N}_{i}}\frac{1}{\sqrt{| d_{i}||d_{j}|}}\mathbf{h}_{j}\mathbf{W}\right),$ (7) $$

where $\mathbf{h}_{i}^{\prime}$ is the updated feature vector for node $i$ after the GCN layer, $\mathcal{N}_{i}$ is the set of neighboring nodes of node $i$.
We train different GCNs based on various ensemble of inputs as different experts, $\mathbf{h}_{j}$ is the feature vector of the neighboring node $j$ and $\mathbf{W}$ is a learnable matrix for the GCN network.
$\frac{1}{\sqrt{|N_{i}||N_{j}|}}$ is the normalization term where $|N_{i}|$ and $|N_{j}|$ are the degrees (number of neighbors) of nodes $i$ and $j$, respectively.
In many GNN papers, the normalization term is also denoted as $\tilde{\mathbf{D}}^{-1/2}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-1/2}$ with $\tilde{\mathbf{A}}=\mathbf{A}+\mathbf{I}$ is the adjacency matrix with added self-loops.
$\tilde{\mathbf{D}}$ is the corresponding degree matrix with $\tilde{\mathbf{D}}_{ii}=\sum_{j}\tilde{\mathbf{A}}_{ij}$ and $\sigma$ is a non-linear activation function.

While GCNs aggregate neighbor information effectively, they apply the same transformation to all nodes, which may not capture diverse patterns or complex dependencies in the graph.
Following our aforementioned notations, the Graph MoE framework introduces multiple experts by learning different GNN models with a gating mechanism for aggregation.
The propagation rule in Graph MoE training is:

$$ $\mathbf{h}_{i}^{\prime}=\sigma\left(\sum_{m=1}^{M}G(\mathbf{h}_{i})_{m}\cdot g _{m}\left(\mathbf{h}_{j};\theta_{m}\right)\right),$ (8) $$

where, for consistency with Equation 5, $g_{m}(\cdot)$ represents the $m$-th GNN expert function that processes a specific ensemble of inputs—such as logits $z_{i}$, node features $\mathbf{x}_{i}$, and node degrees.
The parameter $\theta_{m}$, associated with expert $m$, is analogous to the weight matrix $\mathbf{W}_{m}$ from Equation 7 (we omit the message-passing aggregation for clarity).
The gating mechanism determines the contribution of each expert for node $i$ is

$$ $G(\mathbf{h}_{i})=\operatorname{Softmax}\left(\text{TopK}(Q(\mathbf{h}_{i}),k) \right),$ (9) $$

with the function, TopK to select the best $k$ experts with gating scores
$Q(\mathbf{h}_{i})$.
$G(\mathbf{h}_{i})_{m}$ is the gating function outputted weights for $m$-th expert, with $\mathbf{h}_{i}$ usually being the last layer output of $m$-th expert.
The gating score $Q(\mathbf{h}_{i})$ is computed as:

$$ $Q(\mathbf{h}_{i})=\mathbf{h}_{i}\mathbf{W}_{g}+\epsilon\cdot\operatorname{ Softplus}(\mathbf{h}_{i}\mathbf{W}_{n}),$ (10) $$

where $\mathbf{W}_{g}$ and $\mathbf{W}_{n}$ are learnable weight matrices control clean and noisy scores, respectively.
Noisy gating $\epsilon\sim\mathcal{N}(0,1)$ introduces Gaussian noise for exploration.
In the context of calibration, we adapt GETS to combine multiple calibration experts, each specialized in handling specific calibration factors.
For those unselected models the gating function $G(\cdot)$ sets them to zeroes to be consistent with Equation 9, for example, if we select the best 2 experts, the rest $M-2$ outputs, the gating function $G(\cdot)$ assigns them to 0.
Finally, for each node $i$, the calibrated logits $z_{i}^{\prime}$ are computed as, in a similar form as Equation 5:

$$ $z_{i}^{\prime}=\sum_{m=1}^{M}G(\mathbf{h}_{i})_{m}\cdot g_{m}(z_{i};\theta_{m}),$ (11) $$

Each calibration expert $g_{m}$ can implement different node-wise calibration strategies, where node-wise temperatures are usually included in the model.
The model is trained by minimizing a calibration loss over all nodes: $L=\frac{1}{N}\sum_{i=1}^{N}\ell_{\text{cal}}\left(z_{i}^{\prime},y_{i}\right)$, where $\ell_{\text{cal}}$ is a differentiable calibration loss function (e.g., cross-entropy loss) mentioned before and $N$ is the number of nodes.
By leveraging GETS, we enable the calibration model to specialize in handling diverse calibration factors via different experts and adaptively select the most relevant experts for each node through the gating mechanism using node features and topology.

## 5 Experiments

### 5.1 Experimental Setup

We include the 10 commonly used graph classification networks for thorough evaluation, The data summary is given in Table 1, refer to Appendix A.2 for their sources.
The train-val-test split is 20-10-70 (Hsu et al., ; Tang et al., ), note that uncertainty calibration models are trained on validation set, which is also referred as the calibration set.
We randomly generate 10 different splits of training, validation, and testing inputs and run the models 10 times on different splits.

For the base classification GNN model, i.e., the uncalibrated model $f_{\theta}$, we choose the vanilla GCN (Kipf & Welling, ), GAT (Veličković et al., ), and GIN (Xu et al., ).
We tune the vanilla models to get optimal classification performance, see Appendix A.1.1 for more details about the parameters.
After training, we evaluate models by ECE with $B=10$ equally sized bins.

**Table 1: Summary of datasets. The average degree is defined as $\frac{|2\mathcal{E}|}{|\mathcal{V}|}$ to measure the connectivity of the network.**
|  | Citeseer | Computers | Cora | Cora-full | Cs | Ogbn-arxiv | Photo | Physics | Pubmed | Reddit |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| #Nodes | 3,327 | 13,381 | 2,708 | 18,800 | 18,333 | 169,343 | 7,487 | 34,493 | 19,717 | 232,965 |
| #Edges | 12,431 | 491,556 | 13,264 | 144,170 | 163,788 | 2,501,829 | 238,087 | 495,924 | 108,365 | 114,848,857 |
| Avg. Degree | 7.4 | 73.4 | 9.7 | 15.3 | 17.8 | 29.5 | 63.6 | 28.7 | 10.9 | 98.5 |
| #Features | 3,703 | 767 | 1,433 | 8,710 | 6,805 | 128 | 745 | 8,415 | 500 | 602 |
| #Classes | 6 | 10 | 7 | 70 | 15 | 40 | 8 | 5 | 3 | 41 |

All our experiments are implemented on a machine with Ubuntu 22.04, with 2 AMD EPYC 9754 128-Core Processors, 1TB RAM, and 10 NVIDIA L40S 48GB GPUs.

### 5.2 Confidence Calibration Evaluation

Baselines.
For comparison, we evaluate our approach against several baseline models, including both classic parametric post-hoc calibration methods and graph calibration techniques.
The classic methods include TS (Guo et al., ; Kull et al., ), Vector Scaling (VS) (Guo et al., ), and ETS (Zhang et al., ).
These baselines are primarily designed for standard i.i.d. multi-class classification.

In addition, we compare against graph-based calibration models that utilize a two-layer GNN to learn temperature values for each node.
These include Graph convolution network as a calibration function (CaGCN) (Wang et al., ) and Graph Attention Temperature Scaling (GATS) (Hsu et al., ), which use GCN or GAT layers to process logits and incorporate graph-specific information for calibration.
All models are tuned to their optimal performance, with detailed parameter settings provided in Appendix A.1.3.

GETS settings.
For the parameters of GETS, we constructed input ensembles for each node $i$ as $\{z_{i},\mathbf{x}_{i},d_{i},[z_{i},\mathbf{x}_{i}],[\mathbf{x}_{i},d_{i}],[z_
{i},d_{i}],[z_{i},\mathbf{x}_{i},d_{i}]\}$.
We trained $M=7$ experts and selected the top 2 experts based on their gating scores, as described in Equation 9.
For the degree input $d_{i}$, we mapped the integer degree of each node into a dense vector of fixed size 16 using torch.nn.Embedding, before training the experts with this representation.

After parameter tuning, we train the GNN expert $g_{m}(\cdot),m=1,\dots,M$ for 1000 epochs with a patience of 50 across all datasets.
The calibration learning rate is set to 0.1, except for the Reddit dataset, which uses a rate of 0.01.
Weight decay is set to 0 for most datasets, except for Citeseer (0.01) and Reddit (0.001).
By default, we use a two-layer GCN to train an expert and apply noisy gating with $\epsilon\sim\mathcal{N}(0,1)$.
We also report the ECE for uncalibrated predictions as a reference.

For all experiments, the pre-trained GNN classifiers are frozen, and the predicted logits $z$ from the validation set are fed into our calibration model as inputs.
Further comparison settings and hyperparameters are detailed in the Appendix A.1.2.

**Table 2: Calibration performance across datasets evaluated by ECE. Results are reported as mean $\pm$ standard deviation over 10 runs. ‘Uncal.’ denotes uncalibrated results, and ‘OOM’ indicates out-of-memory issues where the model could not be run. The best results for each dataset are marked bold.**
| Dataset | Classifier | Uncal. | TS | VS | ETS | CaGCN | GATS | GETS |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Citeseer | GCN | 20.51 $\pm$ 2.94 | 3.10 $\pm$ 0.34 | 2.82 $\pm$ 0.56 | 3.06 $\pm$ 0.33 | 2.79 $\pm$ 0.37 | 3.36 $\pm$ 0.68 | 2.50 $\pm$ 1.42 |
| GAT | 16.06 $\pm$ 0.62 | 2.18 $\pm$ 0.20 | 2.89 $\pm$ 0.28 | 2.38 $\pm$ 0.25 | 2.49 $\pm$ 0.47 | 2.50 $\pm$ 0.36 | 1.98 $\pm$ 0.30 |  |
| GIN | 4.12 $\pm$ 2.83 | 2.39 $\pm$ 0.32 | 2.55 $\pm$ 0.34 | 2.58 $\pm$ 0.34 | 2.96 $\pm$ 1.49 | 2.03 $\pm$ 0.23 | 1.86 $\pm$ 0.22 |  |
| Computers | GCN | 5.63 $\pm$ 1.06 | 3.29 $\pm$ 0.48 | 2.80 $\pm$ 0.29 | 3.45 $\pm$ 0.56 | 1.90 $\pm$ 0.42 | 3.28 $\pm$ 0.52 | 2.03 $\pm$ 0.35 |
| GAT | 6.71 $\pm$ 1.57 | 1.83 $\pm$ 0.27 | 2.19 $\pm$ 0.15 | 1.89 $\pm$ 0.24 | 1.88 $\pm$ 0.36 | 1.89 $\pm$ 0.34 | 1.77 $\pm$ 0.27 |  |
| GIN | 4.00 $\pm$ 2.84 | 3.92 $\pm$ 2.59 | 2.57 $\pm$ 1.06 | 2.91 $\pm$ 2.07 | 2.37 $\pm$ 1.87 | 3.34 $\pm$ 2.08 | 3.14 $\pm$ 3.70 |  |
| Cora | GCN | 22.62 $\pm$ 0.84 | 2.31 $\pm$ 0.53 | 2.43 $\pm$ 0.41 | 2.56 $\pm$ 0.47 | 3.22 $\pm$ 0.81 | 2.60 $\pm$ 0.76 | 2.29 $\pm$ 0.52 |
| GAT | 16.70 $\pm$ 0.75 | 1.82 $\pm$ 0.39 | 1.71 $\pm$ 0.28 | 1.67 $\pm$ 0.33 | 2.85 $\pm$ 0.66 | 2.58 $\pm$ 0.71 | 2.00 $\pm$ 0.48 |  |
| GIN | 3.59 $\pm$ 0.65 | 2.51 $\pm$ 0.33 | 2.20 $\pm$ 0.38 | 2.29 $\pm$ 0.30 | 2.79 $\pm$ 0.26 | 2.64 $\pm$ 0.39 | 2.63 $\pm$ 0.87 |  |
| Cora-full | GCN | 27.73 $\pm$ 0.22 | 4.43 $\pm$ 0.10 | 3.35 $\pm$ 0.22 | 4.34 $\pm$ 0.09 | 4.08 $\pm$ 0.25 | 4.46 $\pm$ 0.17 | 3.32 $\pm$ 1.24 |
| GAT | 37.51 $\pm$ 0.22 | 2.27 $\pm$ 0.33 | 3.55 $\pm$ 0.13 | 1.37 $\pm$ 0.25 | 3.96 $\pm$ 1.15 | 2.47 $\pm$ 0.35 | 1.52 $\pm$ 2.27 |  |
| GIN | 10.69 $\pm$ 3.55 | 3.33 $\pm$ 0.22 | 2.43 $\pm$ 0.19 | 2.18 $\pm$ 0.16 | 4.74 $\pm$ 0.66 | 3.98 $\pm$ 0.35 | 1.95 $\pm$ 0.37 |  |
| CS | GCN | 1.50 $\pm$ 0.10 | 1.49 $\pm$ 0.07 | 1.39 $\pm$ 0.10 | 1.44 $\pm$ 0.08 | 2.00 $\pm$ 0.98 | 1.51 $\pm$ 0.10 | 1.34 $\pm$ 0.10 |
| GAT | 4.30 $\pm$ 0.42 | 2.98 $\pm$ 0.15 | 3.02 $\pm$ 0.18 | 2.98 $\pm$ 0.15 | 2.57 $\pm$ 0.38 | 2.56 $\pm$ 0.15 | 1.05 $\pm$ 0.27 |  |
| GIN | 4.36 $\pm$ 0.31 | 4.31 $\pm$ 0.31 | 4.28 $\pm$ 0.28 | 3.04 $\pm$ 1.19 | 1.81 $\pm$ 1.09 | 1.24 $\pm$ 0.45 | 1.15 $\pm$ 0.32 |  |
| Ogbn-arxiv | GCN | 9.90 $\pm$ 0.77 | 9.17 $\pm$ 1.10 | 9.29 $\pm$ 0.88 | 9.60 $\pm$ 0.92 | 2.32 $\pm$ 0.33 | 2.97 $\pm$ 0.34 | 1.85 $\pm$ 0.22 |
| GAT | 6.90 $\pm$ 0.47 | 3.56 $\pm$ 0.10 | 4.39 $\pm$ 0.08 | 3.55 $\pm$ 0.10 | 3.52 $\pm$ 0.13 | 3.90 $\pm$ 0.77 | 2.34 $\pm$ 0.12 |  |
| GIN | 11.56 $\pm$ 0.27 | 11.35 $\pm$ 0.27 | 11.26 $\pm$ 0.43 | 11.38 $\pm$ 0.40 | 6.01 $\pm$ 0.28 | 6.41 $\pm$ 0.29 | 3.33 $\pm$ 0.36 |  |
| Photo | GCN | 4.05 $\pm$ 0.42 | 2.14 $\pm$ 0.46 | 2.11 $\pm$ 0.23 | 2.21 $\pm$ 0.34 | 2.42 $\pm$ 0.76 | 2.24 $\pm$ 0.30 | 2.07 $\pm$ 0.31 |
| GAT | 4.97 $\pm$ 0.75 | 2.06 $\pm$ 0.57 | 1.71 $\pm$ 0.23 | 2.57 $\pm$ 0.59 | 1.69 $\pm$ 0.14 | 2.05 $\pm$ 0.45 | 1.10 $\pm$ 0.25 |  |
| GIN | 3.37 $\pm$ 2.40 | 3.12 $\pm$ 1.85 | 3.77 $\pm$ 1.10 | 3.34 $\pm$ 1.70 | 2.42 $\pm$ 1.90 | 3.37 $\pm$ 1.85 | 1.43 $\pm$ 0.71 |  |
| Physics | GCN | 0.99 $\pm$ 0.10 | 0.96 $\pm$ 0.05 | 0.96 $\pm$ 0.05 | 0.87 $\pm$ 0.05 | 1.34 $\pm$ 0.45 | 0.91 $\pm$ 0.04 | 0.87 $\pm$ 0.09 |
| GAT | 1.52 $\pm$ 0.29 | 0.37 $\pm$ 0.10 | 0.44 $\pm$ 0.05 | 0.47 $\pm$ 0.22 | 0.69 $\pm$ 0.11 | 0.48 $\pm$ 0.23 | 0.29 $\pm$ 0.05 |  |
| GIN | 2.11 $\pm$ 0.14 | 2.08 $\pm$ 0.14 | 2.08 $\pm$ 0.14 | 1.64 $\pm$ 0.09 | 2.36 $\pm$ 0.52 | 0.90 $\pm$ 0.66 | 0.44 $\pm$ 0.10 |  |
| Pubmed | GCN | 13.26 $\pm$ 1.20 | 2.45 $\pm$ 0.30 | 2.34 $\pm$ 0.27 | 2.05 $\pm$ 0.26 | 1.93 $\pm$ 0.36 | 2.19 $\pm$ 0.27 | 1.90 $\pm$ 0.40 |
| GAT | 9.84 $\pm$ 0.16 | 0.80 $\pm$ 0.07 | 0.95 $\pm$ 0.08 | 0.81 $\pm$ 0.09 | 1.04 $\pm$ 0.08 | 0.81 $\pm$ 0.07 | 0.78 $\pm$ 0.15 |  |
| GIN | 1.43 $\pm$ 0.15 | 1.01 $\pm$ 0.06 | 1.04 $\pm$ 0.06 | 1.00 $\pm$ 0.04 | 1.40 $\pm$ 0.47 | 1.00 $\pm$ 0.06 | 0.92 $\pm$ 0.13 |  |
| Reddit | GCN | 6.97 $\pm$ 0.10 | 1.72 $\pm$ 0.07 | 2.02 $\pm$ 0.07 | 1.72 $\pm$ 0.07 | 1.50 $\pm$ 0.08 | OOM | 1.49 $\pm$ 0.07 |
| GAT | 4.75 $\pm$ 0.15 | 3.26 $\pm$ 0.08 | 3.43 $\pm$ 0.10 | 3.52 $\pm$ 0.10 | 0.81 $\pm$ 0.09 | OOM | 0.62 $\pm$ 0.08 |  |
| GIN | 3.22 $\pm$ 0.09 | 3.17 $\pm$ 0.13 | 3.19 $\pm$ 0.10 | 3.25 $\pm$ 0.14 | 1.63 $\pm$ 0.21 | OOM | 1.57 $\pm$ 0.12 |  |

Results. Table 2 presents the calibration results evaluated by ECE, showing that our proposed method, GETS, consistently achieves superior performance across various classifiers and datasets. On average, GETS improves ECE by 28.60% over CaGCN, 26.62% over GATS, and 28.09% over ETS across all datasets. While GETS demonstrates significant improvements on most datasets, some exceptions are noted in Computers, Cora, and Cora-full, where the gains are less pronounced or slightly negative.

Interestingly, the best results often come from models that incorporate ensemble strategies, such as ETS, highlighting the importance of combining these techniques for effective calibration.
This underscores the value of our GETS framework, which leverages both ensemble learning and GNN structures to achieve node-specific calibration.

Moreover, GETS proves to be more scalable than GATS, which suffers from out-of-memory (OOM) issues in larger networks like Reddit.
These results affirm the robustness and scalability of GETS, showcasing its ability to handle both small and large-scale datasets while significantly reducing calibration error.
Overall, GETS demonstrates its effectiveness in addressing node-wise calibration by incorporating input and model ensemble, offering clear improvements over other GNN calibration methods like CaGCN and GATS.

### 5.3 Time Complexity

CaGCN uses a two-layer GCN with a total time complexity of $O(2(|\mathcal{E}|F+|\mathcal{V}|F^{2}))$, where $F$ is the feature dimension (Blakely et al., ).
GATS adds attention mechanisms, leading to a complexity of $O(2(|\mathcal{E}|FH+|\mathcal{V}|F^{2}))$, with $H$ as the number of attention heads (Veličković et al., ).
GETS, fundamentally Graph MoE, introduces additional complexity by selecting the top $k$ experts per node, yielding $O(k(|\mathcal{E}|F+|\mathcal{V}|F^{2})+|\mathcal{V}|MF)$, where $k\ll M$.
This reduces computational costs compared to using all experts, making GETS scalable while maintaining competitive performance.
Training multiple experts might make GETS less time-efficient than its counterpart.
However, GETS scales linearly with the number of nodes, and selecting lightweight models like GCN as experts helps manage the time complexity effectively.
In practice, the GETS model is computationally efficient, as shown in Figure 3(a), even though multiple experts are trained, GETS remains stable elapsed time comparably efficient with the other two models that are trained with a single GNN model.

### 5.4 Expert Selection

One advantage of GETS is its ability to visualize which experts, trained on specific input types, are most important for each node.

Figure: (a) Elapsed time for 10 runs.
Refer to caption: x3.png

We illustrate this by coloring each node according to the selected expert for it.
As sparse activation in the gating function highlights the significance of different input ensembles, we present the top two expert selections (primary and secondary) for each dataset in Figure 3(b).
Gating results are averaged over 10 runs to select the primary and secondary experts.

The results show a diverse range of expert choices, with the combination of logits and features $[z,x]$ frequently chosen as the primary experts, particularly for complex datasets like Ogbn-arxiv and Reddit.
In contrast, degree-based experts ($d$) are often selected as secondary. For smaller datasets such as Cora and Pubmed, individual expert selections are more prevalent.
Noted that only using logits $z$ as the inputs are not preferred by the gating function, which indicates the importance of ensembling different inputs.
These findings indicate that integrating multiple input types—especially logits and features—improves calibration, with the optimal expert combination depending on the complexity of the dataset.

### 5.5 Ablation Studies

In this section, we evaluate the impact of the backbone calibration model in our proposed GETS framework.
We investigate whether the choice of GNN backbone (e.g., GCN, GAT, GIN) significantly affects calibration performance, offering insights into the robustness of GETS across different architectures.
We reuse the parameters tuned for default GETS.
For the ablation discussion that follows, we use GCN as the default uncalibrated model.

**Table 3: Ablation studies on different expert models, measured by ECE.**
| Expert | Citeseer | Computers | Cora | Cora-full | CS | Ogbn-arxiv | Photo | Physics | Pubmed | Reddit |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GETS-GAT | 4.09 $\pm$ 0.71 | 3.64 $\pm$ 1.94 | 2.96 $\pm$ 0.70 | 14.04 $\pm$ 5.70 | 4.91 $\pm$ 3.93 | 1.61 $\pm$ 0.28 | 3.43 $\pm$ 1.82 | 2.57 $\pm$ 2.23 | 1.96 $\pm$ 0.59 | OOM |
| GETS-GIN | 4.34 $\pm$ 1.36 | 4.56 $\pm$ 3.33 | 5.53 $\pm$ 0.59 | 2.83 $\pm$ 0.46 | 2.29 $\pm$ 0.82 | 2.48 $\pm$ 0.30 | 4.06 $\pm$ 2.96 | 1.16 $\pm$ 0.14 | 2.30 $\pm$ 0.58 | 4.64 $\pm$ 1.03 |

Generally, choosing GAT and GIN as the models to train experts does not provide significant advantages over using GCN, as shown in Table 3.
GETS-GAT yields comparable performance to GATS, also encounters the same OOM issue in the Reddit dataset, demonstrating its limitations in handling large-scale datasets. Furthermore, while GETS-GIN shows improved results on datasets like Cora-full, it underperforms on several other datasets, such as Citeseer, Computers, and Reddit, compared to GCN-based calibration. Notably, in simpler datasets or those with smaller graph structures, GCN tends to achieve better calibration results without overfitting. This suggests that a simpler GNN-based calibrator, like GCN, is preferable for training experts as it offers a more balanced trade-off between model complexity and calibration performance, while avoiding potential issues like overfitting, which may arise when more complex GNN architectures are used.

## 6 Conclusion

We propose the GETS framework that combines input and model ensemble strategies within a Graph MoE architecture.
It effectively incorporates diverse inputs — logits, node features, and degree embeddings, as well as using a sparse gating mechanism to adaptively select the most relevant experts for each node.
This mechanism enhances node-wise calibration performance in general GNN classification tasks.

Our extensive experiments on 10 benchmark GNN datasets demonstrated that GETS consistently outperforms state-of-the-art calibration methods, reducing the expected calibration error significantly across different datasets and GNN architectures.
GETS also proved to be computationally efficient and scalable, effectively handling large-scale graphs without significant overhead.
By integrating multiple influential factors and leveraging ensemble strategies, GETS enhances the reliability and trustworthiness of GNN prediction results.

## Appendix A Appendix

### A.1 Model Details

#### A.1.1 Base GNN Classifier Settings

For the base GNN classification model (i.e., the uncalibrated model), we follow the architecture and parameter setup outlined by Kipf & Welling (); Veličković et al. (); Xu et al. (), with modifications to achieve optimal performance. Specifically, we use a two-layer GCN, GAT, or GIN model and tune the hidden dimension from the set $\{16,32,64\}$. We experiment with dropout rates ranging from 0.5 to 1, and we do not apply any additional normalization.

During training, we use a learning rate of $1\times 10^{-2}$. We tune the weight decay parameter to prevent overfitting and consider adding early stopping with a patience of 50 epochs. The model is trained for a maximum of 200 epochs to ensure convergence. The specifics are summarized in Table 4.

**Table 4: Summary of GNN and Training Parameters**
| Dataset | Num Layers | Hidden Dimension | Dropout | Training Epochs | Learning Rate | Weight Decay |
| --- | --- | --- | --- | --- | --- | --- |
| Citeseer | 2 | 16 | 0.5 | 200 | 1e-2 | 5e-4 |
| Computers | 2 | 64 | 0.8 | 200 | 1e-2 | 1e-3 |
| Cora-Full | 2 | 64 | 0.8 | 200 | 1e-2 | 1e-3 |
| Cora | 2 | 16 | 0.5 | 200 | 1e-2 | 5e-4 |
| CS | 2 | 64 | 0.8 | 200 | 1e-2 | 1e-3 |
| Ogbn-arxiv | 2 | 256 | 0.5 | 200 | 1e-2 | 0 |
| Photo | 2 | 64 | 0.8 | 200 | 1e-2 | 1e-3 |
| Physics | 2 | 64 | 0.8 | 200 | 1e-2 | 1e-3 |
| Pubmed | 2 | 16 | 0.5 | 200 | 1e-2 | 5e-4 |
| Reddit | 2 | 16 | 0.5 | 200 | 1e-2 | 5e-4 |

#### A.1.2 GETS Parameter Settings

The backbone GNN models used for calibration are identical to the base classification models, ensuring consistent feature extraction and processing. We tune the hidden dimension for the calibrators from the set $\{16,32,64\}$, depending on the dataset’s complexity, while dropout rates are chosen between 0.2 and 0.8 to balance regularization and model performance.

GETS includes an expert selection mechanism that incorporates logits, features, and node degree information, either individually or in combination, depending on the configuration that best fits the dataset. For all experiments, we maintain a fixed bin size of 10 for uncertainty estimation. The calibrators are trained with a learning rate of either $1\times 10^{-1}$ or $1\times 10^{-3}$, depending on the model’s complexity, and we tune weight decay ranging from 0 0 to $0.1$ to further regularize the training process. The calibrators are trained for up to 1000 epochs, with early stopping applied based on a patience of 50 epochs.

This comprehensive setup ensures robust calibration of the GNN models across diverse datasets. The full details of the calibration parameters are summarized in Table 5.

**Table 5: Summary of GETS Parameters Across Datasets**
| Dataset | Hidden Dim | Dropout | Num Layers | Learning Rate | Weight Decay |
| --- | --- | --- | --- | --- | --- |
| Citeseer | 16 | 0.2 | 2 | 0.1 | 0.01 |
| Computers | 16 | 0.8 | 2 | 0.1 | 0 |
| Cora-Full | 16 | 0.8 | 2 | 0.1 | 0 |
| Cora | 16 | 0.5 | 2 | 0.001 | 0 |
| CS | 16 | 0.8 | 2 | 0.1 | 0 |
| Ogbn-arxiv | 16 | 0.2 | 2 | 0.1 | 0.01 |
| Photo | 32 | 0.8 | 2 | 0.1 | 0.001 |
| Physics | 16 | 0.2 | 2 | 0.1 | 0.001 |
| Pubmed | 16 | 0.8 | 2 | 0.1 | 0 |
| Reddit | 64 | 0.8 | 2 | 0.01 | 0.001 |

#### A.1.3 Baseline Models Settings

For all the baseline calibration models, we ensure consistent training and calibration settings. The calibration process is set to run for a maximum of 1000 epochs with early stopping applied after 50 epochs of no improvement. The calibration learning rate is set to $0.01$, with no weight decay (0 0) to avoid regularization effects during calibration. We use 10 bins for the calibration process, and the default calibration method is chosen as Vector Scaling (VS). For the GATS model, we apply two attention heads and include bias terms. Additionally, we apply a dropout rate of 0.5 during the calibration to prevent overfitting, ensuring robustness across different setups. Full implementations can be also found on our GitHub.

TS (Guo et al., ) adjusts the logits output by a model using a single scalar temperature parameter $T>0$ . The calibrated logits $z^{\prime}_{i}$ for sample $i$ are computed as:

$$ $z^{\prime}_{i}=\frac{z_{i}}{T}$ (12) $$

The temperature $T$ is learned by minimizing the negative log-likelihood (NLL) loss on the validation (calibration) set:

$$ $\min_{T>0}\mathcal{L}_{\text{NLL}}(T)=-\frac{1}{N}\sum{i=1}^{N}\log\left( \operatorname{softmax}\left(\frac{z_{i}}{T}\right)_{y_{i}}\right)$ (13) $$

where $y_{i}\in\{1,\dots,K\}$ is the true label for sample $i$, $N$ is the number of validation samples.

Vector Scaling (VS) (Guo et al., ) extends TS by introducing class-specific temperature scaling and bias parameters. The calibrated logits $z^{\prime}_{i}$ are computed as:

$$ $z^{\prime}_{i}=z_{i}\odot\mathbf{t}+\mathbf{b}$ (14) $$

where: $\mathbf{t}\in\mathbb{R}^{K}$ is the vector of temperature scaling parameters for each class, $\mathbf{b}\in\mathbb{R}^{K}$ is the vector of bias parameters for each class, and $\odot$ denotes element-wise multiplication. Therefore, the parameters $\mathbf{t}$ and $\mathbf{b}$ are learned by minimizing the NLL loss on the validation set:

$$ $\min_{\mathbf{t},\mathbf{b}}\ \mathcal{L}_{\text{NLL}}(\mathbf{t},\mathbf{b})= -\frac{1}{N}\sum_{i=1}^{N}\log\left(\operatorname{softmax}\left(z_{i}\odot \mathbf{t}+\mathbf{b}\right)_{y_{i}}\right)$ (15) $$

ETS (Zhang et al., ) combines multiple calibration methods by weighting their outputs to improve calibration performance. Specifically, ETS combines the predictions from temperature-scaled logits, uncalibrated logits, and a uniform distribution. The calibrated probability vector $p_{i}$ for sample $i$ is computed as:

$$ $p_{i}=w_{1}\cdot\operatorname{softmax}\left(\frac{z_{i}}{T}\right)+w_{2}\cdot \operatorname{softmax}(z_{i})+w_{3}\cdot\mathbf{u}$ (16) $$

where: $z_{i}\in\mathbb{R}^{K}$ is the vector of logits for sample $i$.
$T>0$ is the temperature parameter learned from TS, $w_{1},w_{2},w_{3}\geq 0$ are the ensemble weights, satisfying $w_{1}+w_{2}+w_{3}=1$,
and $\mathbf{u}=\frac{1}{K}\mathbf{1}\in\mathbb{R}^{K}$ is the uniform probability distribution over $K$ classes.

The ensemble weights $\mathbf{w}=[w_{1},w_{2},w_{3}]$ are optimized by minimizing the NLL loss:

$$ $\displaystyle\min_{\mathbf{w}}$ $\displaystyle\mathcal{L}_{\text{NLL}}(\mathbf{w})=-\frac{1}{N}\sum_{i=1}^{N} \log\left(p_{i,y_{i}}\right)$ (17) subject to $\displaystyle\sum_{j=1}^{3}w_{j}=1,\quad w_{j}\geq 0\quad\text{for }j=1,2,3$ $$

CaGCN (Wang et al., ) incorporates graph structure into the calibration process by applying a GCN to the logits to compute node-specific temperatures. The temperature $T_{i}$ for node $i$ is computed as:

$$ $T_{i}=\operatorname{softplus}\left(f_{\text{GCN}}(z,A)_{i}\right)$ (18) $$

where $z\in\mathbb{R}^{N\times K}$ is the matrix of logits for all nodes, $A$ is the adjacency matrix of the graph, $f_{\text{GCN}}(\cdot)$ is the GCN function mapping from logits to temperature logits, $\operatorname{softplus}(\cdot)$ ensures the temperatures are positive, and $T_{i}$ is scalar for node $i$. The calibrated logits follows 12:
Note that CaGCN scales the logits multiplicatively with $T_{i}$, where $T_{i}>0$, to ensure its accuracy preserving.

The GCN parameters $\theta$ are learned by minimizing the NLL loss on the validation set:

$$ $\min_{\theta}\ \mathcal{L}_{\text{NLL}}(\theta)=-\frac{1}{N}\sum_{i=1}^{N}\log \left(\operatorname{softmax}\left(z^{\prime}_{i}\right)_{y_{i}}\right)$ (19) $$

GATS (Hsu et al., ) employs a GAT to compute node-specific temperatures, taking into account both the graph structure and node features.
The notations are nearly the same as CaGCN, with a difference in the model architecture, from $f_{\text{GCN}}$ to $f_{\text{GAT}}$. The number of attention heads is set to 2.

### A.2 Dataset Source Descriptions

We evaluated our method on several widely used benchmark datasets, all accessible via the Deep Graph Library (DGL)(^1^11[https://github.com/shchur/gnn-benchmark](https://github.com/shchur/gnn-benchmark)). These datasets encompass a variety of graph types and complexities, allowing us to assess the robustness and generalizability of our calibration approach.

Citation Networks (Cora, Citeseer, Pubmed, Cora-Full): In these datasets (Sen et al., ; McCallum et al., ; Giles et al., ), nodes represent academic papers, and edges denote citation links between them. Node features are typically bag-of-words representations of the documents, capturing the presence of specific words. Labels correspond to the research topics or fields of the papers. The Cora-Full dataset is an extended version of Cora, featuring more nodes and a larger number of classes, which introduces additional classification challenges.

Coauthor Networks (Coauthor CS, Coauthor Physics): These datasets (Shchur et al., ) represent co-authorship graphs where nodes are authors, and edges indicate collaboration between authors. Node features are derived from the authors’ published papers, reflecting their research interests. Labels represent the most active field of study for each author. These datasets are larger and have higher average degrees compared to the citation networks, testing the models’ ability to handle more densely connected graphs.

Amazon Co-Purchase Networks (Computers, Photo): In these graphs (Yang & Leskovec, ), nodes represent products on Amazon, and edges connect products that are frequently co-purchased. Node features are extracted from product reviews and descriptions, providing rich textual information. Labels correspond to product categories. These datasets exhibit strong community structures and higher connectivity, offering a different perspective from academic networks.

Ogbn-arxiv: Part of the Open Graph Benchmark (Hu et al., ), this dataset is a large-scale citation network of ArXiv papers. Nodes represent papers, edges denote citation relationships, node features are obtained from paper abstracts, and labels are assigned based on subject areas. Its size and complexity make it suitable for evaluating scalability and performance on real-world, large graphs.

Reddit: This dataset (Hamilton et al., ) models interactions on Reddit. Nodes are posts, and edges connect posts if the same user comments on both, capturing user interaction patterns. Node features are derived from post content and metadata, and labels correspond to the community (subreddit) of each post. Its large size and dense connectivity challenge models to scale efficiently while maintaining performance.

### A.3 Algorithmic fairness among degree groups

**Table 6: Comparison of VarECE between Uncalibrated and GETS models, with improvement in percentage. Positive values indicate improvement.**
| Dataset | Uncalibrated $(\times 10^{-3})$ | GETS $(\times 10^{-3})$ | Improvement (%) |
| --- | --- | --- | --- |
| Citeseer | 4.46 $\pm$ 0.82 | 2.95 $\pm$ 1.08 | 33.9% |
| Computers | 8.44 $\pm$ 1.11 | 5.86 $\pm$ 3.02 | 30.6% |
| Cora-full | 10.02 $\pm$ 0.15 | 4.87 $\pm$ 0.62 | 51.4% |
| Cora | 2.07 $\pm$ 0.17 | 1.69 $\pm$ 0.28 | 18.3% |
| CS | 2.05 $\pm$ 0.13 | 2.05 $\pm$ 0.18 | 0.0% |
| Ogbn-arxiv | 1.81 $\pm$ 0.29 | 4.72 $\pm$ 0.65 | - |
| Photo | 5.12 $\pm$ 0.78 | 2.62 $\pm$ 0.30 | 48.8% |
| Physics | 1.29 $\pm$ 0.12 | 1.46 $\pm$ 0.20 | - |
| Pubmed | 4.69 $\pm$ 0.28 | 1.63 $\pm$ 0.23 | 65.3% |
| Reddit | 26.38 $\pm$ 0.45 | 16.51 $\pm$ 1.54 | 37.4% |

While fairness is not the primary focus of this paper, our model enhances algorithmic fairness by incorporating graph structural information into the calibration process, improving fairness across different degree groups.
Although grouping nodes based on degrees is a nonparametric binning strategy—which may diverge from our main focus on parametric calibration—we include this discussion to address calibration errors among these groups.
This demonstrates that our method promotes fairness by achieving similar calibration results across nodes with varying degrees.
The metrics and analyses presented can be extended to algorithmic fairness studies to evaluate model trustworthiness.

We define a binning strategy $\mathcal{B}^{*}=\{\mathcal{D}_{1},\mathcal{D}_{2},\dots,\mathcal{D}_{B}\}$ that groups nodes based on their degrees, each containing approximately $\frac{N}{B}$ nodes, where $N$ is the total number of nodes. The average degree of group $\mathcal{D}_{i}$ is:
$\text{AvgDeg}(\mathcal{D}_{i})=\frac{1}{|\mathcal{D}_{i}|}\sum_{v\in\mathcal{D
}_{i}}\text{deg}(v),$
where $\text{deg}(v)$ is the degree of node $v$. Groups are ordered such that $\text{AvgDeg}(\mathcal{D}_{i})<\text{AvgDeg}(\mathcal{D}_{j})$ for $i<j$. Note that $\mathcal{B}^{*}$ can also be defined using other topological statistics.

Following the Rawlsian Difference Principle (Rawls, ; Kang et al., ), we aim to balance calibration errors across degree-based groups by minimizing their variance. We define the Variation of ECE (VarECE) as a fairness metric:

$$ $\text{VarECE}(f_{\theta},\mathcal{B}^{*})=\text{Var}\left(\left\{\frac{| \mathcal{D}_{j}|}{N}\left|\text{Acc}(\mathcal{D}_{j})-\text{Conf}(\mathcal{D}_ {j})\right|\right\}_{j=1}^{B}\right),$ (20) $$

where $\text{Acc}(\mathcal{D}_{j})$ and $\text{Conf}(\mathcal{D}_{j})$ are the accuracy and confidence on group $\mathcal{D}_{j}$, respectively.

Results are given in Table 6.
By minimizing VarECE, our model promotes fairness across different groups, ensuring that no group—particularly those with lower degrees—suffers from disproportionately higher calibration errors.

### A.4 Ablatiion of Ensemble Strategies

If we ablate the input ensemble, we still use $M=7$, but only use the logit $z$ for the input. In this case, we have trained $M$ same calibration models and do the average.

**Table 7: Ablation studies on different input configurations, measured by ECE.**
| Input-Ablation | Citeseer | Computers | Cora | Cora-full | CS | Ogbn-arxiv | Photo | Physics | Pubmed | Reddit |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GETS | 6.70 $\pm$ 1.60 | 4.36 $\pm$ 1.79 | 2.95 $\pm$ 0.43 | 3.42 $\pm$ 0.53 | 1.78 $\pm$ 0.10 | 2.36 $\pm$ 0.11 | 2.03 $\pm$ 0.34 | 1.08 $\pm$ 0.09 | 1.87 $\pm$ 0.31 | 2.86 $\pm$ 0.49 |

If we ablate the model ensemble, the diverse input type is concatenated and is fed into one GNN calibrator, which is reduced to the idea GATS, which is demonstrated to perform less efficiently and effectively than GETS in our previous discussion of Table 2.

### A.5 Calibration Reliability Diagram Results

Based on classifier GCN, the reliability diagrams reflect how the confidence aligns with the accuracies in each bin. Reliability diagram of calibration results of different models are given in Figure 4 and 5.
The ECE calculated by degree based binning is shown in Figure 6.

Figure: (a) Reliability diagram of CaGCN.
Refer to caption: x5.png

Figure: (a) Reliability diagram of VS.
Refer to caption: x9.png

Figure: Figure 6: Visualization of the ECE performance of three deep learning models (CaGCN, GATS, and GETS) across various datasets. Confidence bins are sorted by degree. Note that GATS encountered an out-of-memory issue on the Reddit dataset, resulting in a blank figure for that case.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Apart from the suggestions/questions described in the weaknesses, I only have two minor questions:
1. You report that the GATS calibration approach suffers from OOM problems not present in the other methods. In Section 5.3 you note that the main difference between CaGCN and GATS, time-complexity-wise, lies in the addition of the attention heads $H$. If all computations are parallelized in memory, I would therefore assume the space complexity to be similar. However, in supplement A1.3, you state that you used $H=2$, which appears to be reasonably small, especially considering the fact that $H$ is only a linear factor in the complexity. Does GATS really require orders of magnitude more memory than the other approaches or were the OOM-issues caused by GATS requiring slightly more memory than available? If the former, do you have an explanation as to why this is the case?
2. As I understand, GETS performs temperature scaling (TS) for each vertex. In your experiments, you also evaluated vector scaling (VS). Have you also considered a variant of GETS that uses VS instead of TS?
soundness: 3
strengths: As reported in the experiments, the combination of structural information and a mixture of calibrators trained on different sets of features seems to clearly outperform previously proposed calibration techniques for node classifiers.

This combination of ideas is well motivated by the paper and the claims regarding the advantages of GETS are supported by the broad range of used datasets.

Presentation-wise, the paper is well structured and the ideas are presented concisely and clearly.
summary: The paper proposes graph ensemble temperature scaling (GETS), a novel calibration technique for node-classification GNNs.
GETS is a parametric calibration method which calibrates the predictions for each node in a graph by predicting a temperature for each node which is used as a factor that is applied to the respective class logits.
GETS predicts node-wise temperatures using a set of GNN-based calibration models trained on different features (predicted logits, node degrees, node features or combinations thereof). The temperatures returned by the GNN calibrators are then combined via a learned gating function which selects a top-k subset of the calibrators for each vertex.

To summarize, GETS combines the previously proposed ensemble temperature scaling (ETS) method with the idea of using a GNN to predict structure-aware temperatures (e.g., CaGCN or GATS).
weaknesses: As mentioned, the reported advantages of the GETS approach seem to stem from the combination of two ideas:
1. Predicting node-wise temperatures based on class logits, node features and node degrees.
2. Incorporating structural dependencies between vertices to predict temperatures.

Both ideas have (to some extent) been utilized independently in prior work. While the performed experiments show that the combination of both ideas works well in practice, it only gives limited insight into why exactly this is the case.

More specifically, I believe it would have been interesting to investigate the following questions:
1. *What is the influence of the three types of features used by the experts ($z$, $x$ and $d$)?*
   Figure 3b only partially answers this question by showing how frequently the different experts are selected by the gating function. It does not show to which extent the different features are able to substitute each other, i.e., it would be interesting to investigate ablations of GETS without different subsets of experts/features being part of the pool.
2. *How well does a GETS-like MoE approach with a structure-unaware MLP instead of a GNN work?* While the evaluated ETS calibration approach also uses the idea of combining multiple calibrators, those calibrators do not use the combinations of $z$, $x$ and $d$ used by GETS. Isolating the effect of this choice of features on the calibration performance would be interesting. A related work in this direction by Liu et al. (2022) [1] could be considered here.
3. *Are other structural features than the degree count $d$ also useful for calibration?* There is an overlap in the information provided by the node degree inputs and the graph convolutions performed by each expert. As an addition to the previous point, it would be interesting to investigate whether other structural features (e.g., other node centrality measures or node2vec-like embeddings) can serve as effective surrogates to the structural information that is incorporated via the graph convolutions in each calibrator.

By addressing (at least some) of these questions, I believe, that the paper could shed a more nuanced light on the importance of different features for calibration in the node classification setting.

---
[1] Liu, T. et al.: On Calibration of Graph Neural Networks for Node Classification. In: 2022 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (2022). [https://doi.org/10.1109/IJCNN55064.2022.9892866](https://doi.org/10.1109/IJCNN55064.2022.9892866).

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: My questions can be found in weakness.
soundness: 2
strengths: The authors propose a method that leverages node features, logits, and node degree, combined with a framework of  expert models, to enhance the reliability of node predictions. Through extensive experiments, the authors demonstrate the effectiveness of their method compared to existing baselines.
summary: This paper addresses a critical challenge in graph node classification: the issue of calibration. The authors propose a  method that integrates a combination of features, logits, and node degree within a framework of expert models, aiming to enhance the calibration performance in graph model. By innovatively focusing on the calibration aspect, the study offers new insights into improving the reliable of graph neural networks.
weaknesses: 1. This paper lacks adequate theoretical backing for why using features, logits, and degree with experts can effectively address calibration issues. Additionally, it does not explain why other node factors are not considered. This needs to be clarified.

2. The paper does not explain how the chosen experts can resolve calibration issues, nor does it discuss why a separate model is not designed for each input. This lack of innovation and theoretical justification may undermine the reliability of the research.

3. The baseline models used in the experiments, particularly the graph calibration model baselines, are outdated. For instance, CAGCN was published in NeurIPS 2021 and GATS in 2022. It is recommended that the authors update their baseline models to enhance the comparability and validity of their approach.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 4
questions: Please see weaknesses.
soundness: 3
strengths: - Paper is written well and easy to read.

- The proposed framework is intuitional and the design choices are justified. 

- Experimental results are thorough and enough empirical analyses are provided.

- Proposed method seems to perform well compared to the selected baselines.
summary: This work leverages multiple GNNs with different inputs within a Graph Mixture-ofExperts (MoE) architecture for confidence calibration of GNNs.
weaknesses: - More recent baselines can be included to further strengthen experimental results.

- The novelties of the proposed framework seem to be rather limited based on the current presentation. It would be better to clearly emphasize the novelties over the existing solutions in the Related Work.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Could the authors provide quantitative analysis of expert specialization during the training? This could be potentially done by 
 (1) Visualization of expert activation patterns (investigation of gating function values) across training epochs, or
 (2) Per-expert contribution to the total calibration loss over training epochs, or
 (3) Show how the preference for certain experts emerges and stabilizes (again through analyzing the gating function G or gating scores Q)
While Figure 3(b) demonstrates the final expert selection patterns through sparse activation, a dynamic analysis of expert contribution to performance would be valuable.

2. How does the architecture of individual experts impact calibration performance? Given that experts process different input types, different architecture may be more suitable for each input modality. Currently, GCN with 2 layers is used for all types except the integer degree. Does the same GCN architecture uniformly reflect the effects across the experts of different types?

3. How does GETS perform on heterogeneous graphs where nodes have different feature types or varying degrees of connectivity?
soundness: 3
strengths: * The comprehensive experimental results in Table 2 show consistent improvements across different architectures and datasets.
* The sparse activation mechanism described in Section 4.2 enables efficient scaling to larger graphs. 
* The well-presented analysis of expert selection patterns provides insights into which input combinations are most effective for calibration for each dataset.
summary: This paper introduces Graph Ensemble Temperature Scaling (GETS), a calibration framework for Graph Neural Network that combines input and model ensemble strategies with a Graph Mixture-of-Experts architecture. GETS integrates multiple types of inputs and adaptively selects relevant experts for calibrating such node's predictions. The framework achieves improvements in calibration performance across 10 benchmark datasets.
weaknesses: * The hyperparameter sensitivity analysis is somewhat limited, say regarding the number of experts, or the expert embedding dimension. For example, 
 (1) How do calibration performance and computation costs scale with the number of experts (M) ?
 (2) What is the impact of expert network capacity (e.g. the number of neurons, the number of layers) on calibration quality?
 (3) How does the choice of top-k in sparse gating affect the trade-off between performance and efficiency? An empirical experiment on synthetic dataset would provide valuable implementation guide. 

* While computational efficiency is emphasize, the need to store multiple expert models could be problematic for resource-constrained settings. This practical limitation deserves more discussion. For instance, computational memory requirements for different types of experts handling various input might be helpful.

Meta-review: The paper presents a novel approach, Graph Ensemble Temperature Scaling (GETS), aimed at improving the calibration of Graph Neural Networks (GNNs). By leveraging various input sources and employing a Mixture-of-Experts architecture, GETS effectively reduces expected calibration error across multiple benchmark datasets. This is a significant contribution, as it not only addresses a critical gap in existing calibration methods but also highlights the importance of accurate uncertainty estimation in high-stakes applications.

Reviewers presented mixed opinions regarding the paper's contributions and execution. While some reviewers appreciated the empirical results and potential improvements in calibration performance, others expressed concerns about the theoretical justification and the choice of baseline models used for comparison. Nonetheless, these differences did not impede the overall assessment of the paper's quality.

As the rebuttal phase progressed, the authors addressed the reviewers' concerns satisfactorily, clarifying the innovation in their approach and enhancing their experimental results. All reviewers ultimately expressed their satisfaction with the improvements made, leading to a consensus on the paper's merit. In conclusion, the reviewers unanimously agree that the quality of the paper is high and recommend acceptance based on its significant contributions and the authors' responsiveness to feedback.

Decision: Accept (Spotlight)