Graph-based Confidence Calibration for Large Language Models

## Abstract

Abstract One important approach to improving the reliability of large language models (LLMs) is to provide accurate confidence estimations regarding the correctness of their answers. However, developing a well-calibrated confidence estimation model is challenging, as mistakes made by LLMs can be difficult to detect. We propose a novel method combining the LLM’s self-consistency with labeled data and training an auxiliary model to estimate the correctness of its responses to questions. This auxiliary model predicts the correctness of responses based solely on their consistent information. To set up the learning problem, we use a weighted graph to represent the consistency among the LLM’s multiple responses to a question. Correctness labels are assigned to these responses based on their similarity to the correct answer. We then train a graph neural network to estimate the probability of correct responses. Experiments demonstrate that the proposed approach substantially outperforms several of the most recent methods in confidence calibration across multiple widely adopted benchmark datasets. Furthermore, the proposed approach significantly improves the generalization capability of confidence calibration on out-of-domain (OOD) data.

## 1 Introduction

In recent years, large language models (LLMs) have demonstrated remarkable capabilities across various natural language processing tasks such as question answering (Wei et al., ; Shen et al., ; Zheng et al., ; Qin et al., ; Singhal et al., ), text summarization (Tang et al., ; Deroy et al., ; Tam et al., ; Roit et al., ), and even creative writing (Gómez-Rodríguez & Williams, ; Wang et al., ; Deng et al., ). Despite their impressive performance, LLMs often give wrong answers in question-answering tasks. There is an urgent need to check the correctness of LLMs’ responses. One particularly interesting question is to calibrate the confidence levels of the correctness of responses from LLMs (Kuhn et al., ; Ulmer et al., ; He & Jiang, ; Van Landeghem et al., ; Vazhentsev et al., ; Ulmer et al., ). Accurate confidence estimation is vital for deploying LLMs in the real world, as it allows users to gauge the reliability of the model’s predictions and make informed decisions based on these outputs. On the contrary, miscalibrated confidence may lead to over-trust in incorrect responses or doubts about the correct ones. For example, a misleading response may steer a patient in the wrong direction when making health decisions; it may also lead an investor to impulsive investment decisions.

In this work, we consider calibrating the confidence with the correctness of LLMs’ responses. This task is challenging in several aspects. First, due to LLMs’ superior ability to generate text, mistakes in an LLM’s response are usually at the semantic level, making it hard to detect even for humans. There are methods using an auxiliary Language Model (e.g., DeBERTa (He et al., ))
to check whether the LLM’s response answers the question Ulmer et al. (). Since the LLM is supposed to be much stronger than the LM, the LLM should be able to avoid most mistakes that can be detected by an LM; this type of method may omit a significant fraction of wrong answers. Second, it is hard to detect mistakes from the LLM’s internal working mechanism. Because the LLM uses many hidden layers to process the information, it is hard to discern the signal from a small number of hidden units. Even if this is possible, it is not easy to apply this type of method to black-box LLMs.

Recently, there has been some progress in quantifying the model’s own confidence in a response through consistency among the model’s responses (Chen & Mueller, ; Lin et al., ).
If the LLM always gives similar responses, then there is less uncertainty, and any one of these responses tends to have a high probability of being correct. In particular, the results show that the model’s own confidence in its response has a strong correlation with the correctness of the response. This new development leads to an important research question: whether we can calibrate the confidence of correctness from consistency among the LLM’s responses.

In this work, we develop a new method to calibrate the confidence of correctness from the consistency of an LLM’s responses. To achieve this goal, we train a separate calibration model to predict the correctness of the LLM’s responses. To get the input to the calibration model, we form a similarity graph over the LLM’s multiple responses to the same question. The similarity graph encodes information about the consistency between LLM’s responses. A response consistent with more answers tends to have a higher likelihood of being correct, so the consistency graph is predictive of the responses’ correctness. In our new approach, the calibration model only considers the consistency among responses without processing any actual language information. Thus, we can use a relatively simple and efficient model.

Our model achieves premium performance in the empirical study. Compared with previous calibration methods, our model has much better calibration performance because of the usage of consistency graphs. Compared to prior methods based on consistency inputs, our method improves not only the calibration performance but also the ranking performance. Furthermore, our method improves the generalizability of confidence calibration for out-of-domain settings, demonstrating the advantage of using a separate learning model. In summary, our main contributions are:

- •
Graph-based confidence calibration method: We propose a novel graph-based confidence calibration approach to improve the reliability of LLMs.
- •
Enhanced calibration performance: Our evaluations demonstrate that the proposed method substantially outperforms recent methods in confidence calibration across several widely used benchmark datasets.
- •
Improved OOD generalizability: Evaluations on OOD confidence calibration show that our graph-based approach significantly improves generalizability in OOD settings.

## 2 Related Work

Due to the urgent need to improve the reliability of LLMs, confidence estimation and calibration for these models have become active areas of research. Existing research in LLM uncertainty quantification can be summarized into two main categories: uncertainty quantification and confidence calibration (Geng et al., ). Confidence estimation for short responses (e.g., for multi-choice or yes-no questions) is generally less complicated than for long responses (Ye et al., ). For a brief response, the LLM’s output logits are informative about its confidence; the easy comparisons of responses to the true answer facilitate both calibration and evaluation. Confidence estimation for long responses cannot simply depend on LLM’s output logits (Duan et al., ; Bakman et al., ) because the logits indicate more about the probability of text and less about the semantics behind it. There are also methods using the internal state of an LLM (Ren et al., ), but it is not always available to have such information about the LLM interface.

Another approach is to check the LLM’s consistency in its responses. Kotelanski et al. () demonstrate that repeated sampling and consistency checks across multiple outputs can serve as reliable proxies for model confidence. Manakul et al. () generates multiple responses from the LLM and checks the consistency between responses using various methods, including querying the LLM.
Chen & Mueller () combines the consistency between responses and the LLM’s self-reflection certainty to quantify the uncertainty. Kuhn et al. () considers confidence from semantic equivalence and proposes a method based on clustering of responses. Lin et al. () organize responses in a graph with their pairwise semantic similarity and then extract graph statistics for confidence estimation. Zhang et al. () examines methods of comparing responses via entailment and contradiction relationships. These studies highlight the importance of semantic consistency in confidence estimation. These methods are evaluated by comparing their estimated confidence values against the correctness of responses. The correct labels are often obtained by reference matching (Huang et al., ), checking whether responses match true answers with a particular similarity measure.

To better calibrate the confidence estimation, some methods directly use correctness labels in their calibration procedures. Mielke et al. () trains a calibrator to predict the correctness of a response for a given question. With a similar idea, Ulmer et al. () trains a language model (e.g., DeBERTa) based on question-response pairs to predict the probability of responses’ correctness. Based on SelfCheckGPT (Manakul et al., ) and JAFC (Tian et al., ), Chen et al. () train supervised models to reduce grouping losses and improve the confidence estimation. The method by Liu et al. () uses an LLM’s latent representations to predict the correctness of responses. Detommaso et al. () uses the “multicalibration” technique to calibrate the probability of correctness. (Fadeeva et al., ) offers a detailed comparative study of various confidence estimation methods, providing empirical evidence on their effectiveness across different tasks. However, these studies have not sufficiently exploited response consistency to predict the probabilities of the responses being correct.

## 3 Method

Our ultimate goal is to quantify the probability of the correctness of a response from an LLM. Since the LLM can give a correct answer with different phrases, we need to consider the probability that the response is semantically correct.

Background:  The formulation of semantic equivalence (Kuhn et al., ) provides a framework for our analysis. Let $\mathcal{R}$ be the space of all possible responses. Given a question $q$, the space $\mathcal{R}$ is divided into a set $\mathcal{C}_{q}$ of semantic classes: $\mathcal{R}=\cup_{C\in\mathcal{C}_{q}}C$ and $C^{\prime}\cap C=\emptyset$ for any two different semantic classes $C,C^{\prime}\in\mathcal{C}_{q}$. For two responses $r_{1},r_{2}\in C$ in the same equivalent class, they are considered as the same semantic response: if one is the correct answer, the other is correct as well, and vice versa. Then, we can consider the quality of the LLM’s responses at the semantic level. In particular, a semantic response $C$ has probability

$$ $\displaystyle p(C|q)=\sum_{r\in C}p(r|q).$ (1) $$

Here $p(r|q)$ is the probability of a single response from the LLM.

However, it is non-trivial to define the equivalent class, and we will discuss the approximation later. To estimate $p(C|q)$, one approach is through semantic similarities between response samples of an LLM for the same question $q$. Let $(r_{1},\ldots,r_{n})$ be $n$ responses from the same question $q$, and they form $k$ clusters $\tilde{\mathcal{C}}_{q}=\{\tilde{C}_{1},\ldots,\tilde{C}_{k}\}$ by their semantic similarity. We can use Natural language inference (NLI) systems to predict the relationships (e.g., entailment and contradiction) between responses and derive their similarity.

We assume that each cluster cluster $\tilde{C}$ is from a different semantic class $C$, then $p(C|q)$ can be approximated by

$$ $\displaystyle p(C|q)\approx\frac{|\tilde{C}|}{n}.$ (2) $$

From the cluster probabilities, the uncertainty of the LLM on the question $q$ is estimated as the entropy of the empirical distribution over clusters (Kuhn et al., ), and the confidence of a response $r_{i}\in C$ is estimated as ${|\tilde{C}|}/{n}$ (assuming similarity values are binary) (Lin et al., ).

Now, we depart from the setup of semantic classes and consider the correctness of responses. Let $C^{*}$ be the correct semantic answer to question $q$. Without knowing which responses are correct answers, a common assumption is that the model’s confidence reflects the correctness, that is, the model’s confidence in a semantic response is approximately the probability of correctness, then

$$ $\displaystyle p(\tilde{C}_{k^{\prime}}\subseteq C^{*})\approx\frac{|\tilde{C}_ {k^{\prime}}|}{n}.$ (3) $$

It says that the more certain the model is about a semantic response, the more likely the response is correct. Conversely, a wide variation in the LLM’s responses indicates low confidence in all responses $r_{i}$ and low accuracy. This pattern is also found in previous studies (Kuhn et al., ; Lin et al., ).

While we agree that a positive correlation exists between the LLM’s confidence and the probability of correctness, we do not believe that they are equal, as shown in equation 3. Therefore, we need further calibration to reflect the probability of correctness.

Figure: Figure 1: The overall framework. Given an input question, our approach first samples multiple responses from the LLM and then constructs a similarity graph based on the samples, which serves as the input for the GNN model to calibrate the confidence of LLM responses. The left part is an example of question and responses from LLM (“R” denotes “response,” “C” denotes “confidence”).
Refer to caption: x1.png

### 3.1 Confidence calibration as graph learning problem

Now, we set a supervised learning problem and train a model to calibrate the confidence of the correctness of responses. We first consider the correctness labels of the LLM’s responses. In the supervised setting, we have a correct answer $r^{*}$ to the question $q$. Then $r^{*}$ to assign correctness labels to sampled responses $\{r_{1},r_{2},...,r_{n}\}$ for the same question $q$. In our work, we use the ROUGE similarity. Specifically, we compute the ROUGE similarity $\mathrm{sim}_{R}(r_{i},r^{*})$ between a sampled response and the correct answer to decide the correctness label.

$$ $\displaystyle y_{i}=\mathbbm{1}[\mathrm{sim}_{R}(a,r_{i})\geq\tau],~{}i=1, \ldots,n.$ (4) $$

Here $\mathbbm{1}[\cdot]$ is one if the condition is true or 0 otherwise. The ROUGE metric is reasonably accurate in measuring semantic similarity between short sentences (Lin & Och, ).

We then consider the input to the calibration model. We form a similar graph $G$ over responses to encode information about their consistency. The graph contains the clustering structure of responses and likely further useful information to predict the correctness of responses. The graph $G=(V,E,\mathbf{w})$ is a fully connected graph, with the node set $V$ consisting of $n$ responses and the edge weight $w_{ij}$ being the similarity between the pair of responses $(r_{i},r_{j})$. We compute the similarity from the two responses’ embeddings. In particular, we first use the Sentence-BERT model (Reimers & Gurevych, ) to compute the two responses’ vector representations and then compute the cosine similarity

$$ $\displaystyle w_{ij}=\mathrm{sim_{cos}}(\mathrm{emb}(r_{i}),\mathrm{emb}(r_{j} )),~{}~{}i,j=1,\ldots,n.$ (5) $$

Here, $\mathrm{emb}(\cdot)$ represents the embedding function.

Then, we treat the problem as a node classification problem (Xiao et al., ). In particular, we run a Graph Neural Network (GNN) $\mathrm{gnn}(\cdot)$ to predict the probability of each response being correct

$$ $\displaystyle\tilde{\mathbf{p}}=\mathrm{gnn}_{\theta}(G).$ (6) $$

Here $\tilde{\mathbf{p}}\in[0,1]^{n}$ contains the probabilities for $n$ responses being correct.

To provide clustering information to the GNN, we first run the K-means clustering algorithm on the responses’ embeddings and assign cluster IDs from 0 to $K-1$ based on the order from largest to smallest (ties are randomly broken). Then, we feed each response’s cluster membership as a one-hot feature input to the GNN. Therefore, the GNN’s predictions are purely based on the relationships between responses in semantic space. We choose NOT to feed in the embedding vectors of responses to avoid the GNN’s dependency on textual information. This helps the GNN generalize questions from different domains. The overall framework is shown in Fig 1.

The main purpose of the learning model is to calibrate $\tilde{\mathbf{p}}$. One approach is to minimize the cross-entropy loss of $\tilde{\mathbf{p}}$ against correctness labels. The loss computed from the question $q$ is

$$ $\ell_{q}=-\sum_{i=1}^{n}y_{i}\tilde{p}_{i}+(1-y_{i})\log(1-\tilde{p}_{i})$ (7) $$

Note that the loss is consistent marginally since the loss is minimized when $\tilde{p}_{i}=p(y_{i}|G)$. An alternative approach is to minimize the squared error $(y_{i}-\tilde{p}_{i})$, from which we get similar performances, so we choose the cross-entropy loss. A further consideration is to explicitly consider the similarity between $\tilde{p}_{i}$ and $\tilde{p}_{j}$ given the response similarity $w_{ij}$. We leave such exploration to the future.

### 3.2 Improve the estimation through multiple prompts

It is well known that the syntactic form of a question influences responses and introduces additional variance. To reduce this variance and evaluate the LLM’s semantic consistency, we analyze the LLM’s responses to multiple prompts derived from the same question. These responses are treated as answers to the same semantic question. We then apply the same method as before to predict the correctness of each response.

In particular, we rephrase the original question $q$ into $k$ different forms $\{q_{1},...,q_{k}\}$ while maintaining the original sentence’s semantic meaning. We employ a multiple rephrased questions strategy for answer sampling. Specifically, we prompt the GPT-4 to give $k$ different but with the same meaning rephrased questions for the given question $q$. Then, we sample $n/k$ responses from the LLM for each rephrased question and still get a total of $n$ responses, from which the confidence calibration is the same as we have described above.
The model is more likely to produce diverse responses for questions about which the LLM is less certain. Confidence calibration is more accurate in this scenario because the model’s uncertainty becomes more apparent.

## 4 Experiments

The goal of this section is to compare our proposed framework with baseline methods in terms of confidence calibration. All experiments are conducted on NVIDIA A100 GPUs with 80GB of memory.
The supplementary materials and Appendix provide the code for our model, more experiment details in Appendix A.1, and prompting strategy and Appendix A.6.

### 4.1 Dataset and Experiment setup

Dataset: We conduct experiments on two public benchmark datasets: (1) CoQA (Reddy et al., ), an open-book conversational question answering (QA) task; (2) TriviaQA (Joshi et al., ), a commonsense QA task.
We use the first 4k as training data and 1582 for validation and testing, both of equal size. We provide more details about training in the Appendix A.1.

Baselines: We compare our methods with the following baselines.

Length-normalized sequence likelihoods (Seq. likelihood) (Malinin & Gales, ; Kuhn et al., ) is a standard measure for confidence. This method calculates the likelihood of each sequence and normalizes it by the length of the sequence to provide a fair comparison between different lengths of sequences. Platt scaling (Platt, ), a variant of the sequence likelihood baseline, applies Platt scaling to the raw likelihoods. GraphSpectral (Chen et al., ) uses the graph theory to estimate the confidence. Then we also include post-hoc uncertainty calibration,GraphSpectral+Iso and
GraphSpectral+Platt into the baseline methods.
Self-check GPT(Manakul et al., ) checked the consistency between responses querying the LLM.
Verbalized Uncertainty (Lin et al., ; Tian et al., ; Xiong et al., ) generates verbal statements about the model’s confidence in its predictions. Verbalized Qual maps the confidence percent (Verbalized %) into numerical values. APRICOT (Ulmer et al., ), a supervised method, fine-tunes the Deberta language model to predict confidence scores for LLM outputs. Furthermore, we also include the baseline of applying two post-hoc uncertainty calibration methods, APRICOT+Iso and APRICOT+Platt, to adjust the confidence scores obtained by Apricot. We performed all the baseline experiments utilizing the open-source codebase.

Graph construction:
For each question, we prompt the LLM to generate 30 answers. Each generated answer is then processed using the SentenceBert model Reimers & Gurevych () to obtain the answer’s high-dimensional embeddings. To quantify the semantic similarity between the answers, we compute the cosine similarity between every pair of answer embeddings. These similarity scores are then utilized as edge weights in our similarity graph, where each node represents an individual answer, and the edges signify the degree of semantic relation between them.

Model hyper-parameters:
To ensure our model can capture complex and abstract features at each layer, our model comprises three Graph Convolutional Network (GCN) layers, with embedding dimensions of 256, 512, and 1024 for the first, second, and third layers, respectively. The initial learning rate was set to $10^{-4}$. If the validation loss did not show improvement over ten consecutive epochs, the learning rate was reduced by a factor of 0.9. The optimization was performed using the Adam optimizer, configured with hyperparameters $\beta_{1}=0.9$ and $\beta_{2}=0.98$. The batch size was 16.

LLMs: We assess our confidence calibration method on two LLMs with excellent performance: Llama3-8B (Llama3)(Meta, ), and Vicuna-7b-v1.5 (Vicuna) (Zheng et al., ).

Labeling the data: To obtain the correctness label for CoQA and TriviaQA datasets, we followed previous work (Kuhn et al., ) and used the Rougel-L metric for labeling.

Evaluation metrics:
The evaluation metrics include Expectation Calibration Error (ECE), Brier Score, and AUROC. Specifically, (1) ECE quantifies the consistency between the prediction error and the uncertainty of the prediction. An ideal calibration curve should exhibit a lower ECE. It measures the consistency between the prediction error and the confidence of the prediction.
Specifically, the confidence interval is grouped into fixed bins, and the average of the difference between the confidence and error in each bin is compared. Formally, ECE is calculated as

$$ $ECE=\sum_{b=1}^{B}\frac{n_{b}}{N}|acc(b)-conf(b)|$ (8) $$

where $n_{b}$ is the number of predictions in bin $b$, $N$ is
the total number of data points and $acc(b)$ and $conf(b)$ are the accuracy and confidence of bin b, respectively.
(2) Brier Score (Brier, ), which is the mean squared difference between predicted probabilities and the actual binary results. Lower Brier Scores indicate better performance. (3) AUROC to indicate the models’ discriminatory ability.

**Table 1: Comparison of confidence calibration performance on TriviaQA and CoQA dataset for Llama3**
| Method | TriviaQA | CoQA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Brier$\downarrow$ | AUROC$\uparrow$ | ECE$\downarrow$ | Brier$\downarrow$ | AUROC$\uparrow$ | ECE$\downarrow$ |  |
| GraphSpectral (GS) | 0.22 | 0.80 | 0.056 | 0.19 | 0.76 | 0.091 |
| GS + Iso | 0.16 | 0.80 | 0.072 | 0.16 | 0.76 | 0.054 |
| GS + Platt | 0.16 | 0.79 | 0.048 | 0.16 | 0.74 | 0.041 |
| Self-check GPT | 0.33 | 0.65 | 0.187 | 0.21 | 0.63 | 0.178 |
| Seq. likelihood | 0.54 | 0.53 | 0.22 | 0.38 | 0.47 | 0.17 |
| Platt | 0.28 | 0.59 | 0.05 | 0.26 | 0.47 | 0.09 |
| Verbalized Qual | 0.32 | 0.62 | 0.14 | 0.30 | 0.68 | 0.16 |
| Verbalized % | 0.25 | 0.67 | 0.033 | 0.42 | 0.66 | 0.21 |
| APRICOT | 0.14 | 0.76 | 0.074 | 0.17 | 0.78 | 0.132 |
| APRICOT + Iso | 0.18 | 0.76 | 0.073 | 0.17 | 0.78 | 0.097 |
| APRICOT + Platt | 0.17 | 0.76 | 0.039 | 0.17 | 0.78 | 0.069 |
| Ours | 0.14 | 0.84 | 0.025 | 0.12 | 0.77 | 0.016 |
| Ours (Multi prompts) | 0.14 | 0.84 | 0.025 | 0.12 | 0.78 | 0.015 |

### 4.2 Experiment Results

For the Llama3 model, the confidence calibration performance on TriviaQA and CoQA are shown in Table 1. For the TriviaQA dataset, it can be observed that the likelihood-based method performs poorly on the calibration error (ECE and Brier Score) and AUROC due to unreliable model prediction probability (Zhang et al., ). Platt scaling improves the ECE post-calibration and enhances the model’s discriminative ability, resulting in higher AUROC results. However, this method cannot capture the semantic equivalence among answers, leading to sub-optimal performance. The Verbalized and Verbalized Qual prompts LLM to output confidence for their answers, improving AUROC by $3-5\%$ compared with the likelihood baseline. However, it faces the overconfidence issue; thus, the calibration errors are still high. The graphSpectral method can produce good confidence estimations, but its calibration performance is poor. Even with the addition of techniques such as Isotonic Calibration or Platt Scaling, this issue can only be partially mitigated. The auxiliary DeBERTa method combines the LLM outputs, Chain-of-Thoughts (CoT) outputs, and verbalized confidence to fine-tune the DeBERTa model for predicting confidence. Our method captures the prediction confidence based on the graph structure of LLM’s responses in semantic space and achieves better ECE and smECE results. The ECE is reduced from $0.07$ to $0.02$ and improves the AUROC from $0.76$ to $0.84$ compared with the baseline calibration methods. The experiment results on CoQA for the Llama3 model are shown in Table 1. These results show a similar trend, with our model achieving superior performance in confidence calibration compared to the baseline methods.

**Table 2: Comparison of confidence calibration performance on TriviaQA and CoQA dataset for Vicuna**
| Method | TriviaQA | CoQA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Brier$\downarrow$ | AUROC$\uparrow$ | ECE$\downarrow$ | Brier$\downarrow$ | AUROC$\uparrow$ | ECE$\downarrow$ |  |
| GraphSpectral (GS) | 0.19 | 0.79 | 0.112 | 0.27 | 0.69 | 0.202 |
| GS + Iso | 0.19 | 0.79 | 0.059 | 0.24 | 069 | 0.037 |
| GS + Platt | 0.17 | 0.79 | 0.067 | 0.22 | 0.69 | 0.055 |
| Self-check GPT | 0.35 | 0.63 | 0.18 | 0.22 | 0.64 | 0.192 |
| Verbalized Qual | 0.38 | 0.62 | 0.02 | 0.45 | 0.48 | 0.00 |
| Verbalized % | 0.39 | 0.52 | 0.38 | 0.49 | 0.53 | 0.32 |
| Seq. likelihood | 0.47 | 0.58 | 0.42 | 0.30 | 0.68 | 0.16 |
| Platt | 0.34 | 0.58 | 0.25 | 0.30 | 0.68 | 0.16 |
| APRICOT | 0.18 | 0.78 | 0.068 | 0.19 | 0.74 | 0.073 |
| APRICOT + Iso | 0.17 | 0.78 | 0.049 | 0.19 | 0.74 | 0.06 |
| APRICOT + Platt | 0.17 | 0.78 | 0.052 | 0.19 | 0.74 | 0.049 |
| Ours | 0.17 | 0.81 | 0.020 | 0.18 | 0.75 | 0.02 |
| Ours (Multi prompts) | 0.16 | 0.81 | 0.018 | 0.16 | 0.75 | 0.016 |

Furthermore, we also compare the confidence calibration performance for the Vicuna model on the TriviaQA and CoQA datasets. The results are summarized in Table 2. Our model consistently improves the calibration error compared to the baseline methods.

To understand the improvements of our model, we present the reliability diagrams for the baseline methods applied to the Vicuna model on TriviaQA. The reliability diagram is created by discretizing the confidence value into $10$ bins and then computing the average accuracy for samples in each bin. The ideal calibration curve should align with the diagonal line, indicating that the confidence value can match the probability of correctness. The reliability diagram is shown in Fig. 2. (We also show other reliability diagrams for the different methods for Llamas on TriviaQA and CoQA in the Appendix  A.5). The figure presents the reliability diagrams for different methods, each utilizing 10 bins. In these diagrams, both the color intensity and the percentage numbers within each bar represent the proportion of total responses that fall into each respective bin. Specifically, larger proportions are depicted with colors closer to purple, while the height of each bar indicates the ratio of correct predictions within that bin. An ideal reliability diagram should exhibit a wide distribution of responses across multiple bins, demonstrating the model’s strong ability to differentiate between varying confidence levels in its predictions.
Additionally, the heights of the bars should align closely with the diagonal line, which represents perfect calibration—where the predicted confidence matches the empirical accuracy. It can be observed that the likelihood-based confidence methods exhibit significant overconfidence, with curves below the diagonal, indicating many samples have high confidence but low accuracy. This results in poor ECE performance. Although the Platting scaling calibration method enhances the ECE performance, it still has poor AUROC. The Auxiliary DeBERTa (APRICOT) method, which integrates LLM outputs, Chain-of-Thought (CoT) outputs, and verbalized confidence to train an auxiliary DeBERTa model, enhances the AUROC. However, it still experiences some overconfidence issues, potentially caused by the inherent overconfidence in the input verbalized confidence scores. The baseline method’s reliability diagrams revealed that this method frequently assigned high confidence scores to incorrect predictions, deviating markedly from the ideal calibration represented by the diagonal line. For example, the verbalized method’s predictions in the highest confidence bins (80-90%) were significantly below the corresponding empirical accuracy, indicating a tendency to overestimate the certainty of its outputs. In contrast, our framework achieves a broad spread of responses across the bins, showing good differentiation capabilities; at the same time, the bar heights closely follow the diagonal line, indicating better calibration.

Figure: (a)
Refer to caption: x2.png

### 4.3 Out of Domain evaluation

Domain shift poses significant challenges for deploying machine learning models in real-world scenarios where data variability is expected. To comprehensively assess the robustness and generalization capabilities of our proposed model compared to baseline methods, we conducted a series of out-of-domain (OOD) evaluations.

Experiment setup: We evaluate the confidence calibration of different approaches under out-of-domain settings. We have two experiment configurations: out-of-domain dataset OODD, and out-of-domain LLMs (OODL). For OODD, we train the confidence calibration model on TriviaQA from Llama3 responses and test it on CoQA Llama3 answers. For OODL, we use the same training data from Llama3 but test the Vicuna model’s responses on the TriviaQA and CoQA datasets. We compare our model with the Apricot and GraphSpectral (with Platt scaling) methods.

Results and Analysis:
The OOD experiments results revealed that our model maintained a high level of performance across tested domains Table. 3. Specifically, the model demonstrated consistent calibration, as evidenced by low ECE values and strong discriminative ability, reflected in high AUROC scores on in-domain and OOD datasets. For example, while the model achieved an ECE of 0.025 and an AUROC of 0.84 on TriviaQA (in-domain), it maintained an ECE of 0.077 and an AUROC of 0.77 on CoQA. Furthermore, the Brier scores across domains remained within acceptable ranges, demonstrating reliable probabilistic predictions even when faced with unfamiliar data distributions. The relatively small increase in ECE and a slight decrease in AUROC for OOD datasets suggest that while there is some degradation in performance, the model retains substantial robustness and accuracy. This is primarily because similarity graph patterns are highly invariant to the data distribution. Specifically, our model employs the consistency graph and the clustering feature that does not alter with data distribution shifts, enabling it to maintain stable performance across different datasets.
In contrast, Apricot typically relies on specific dataset features, which leads to poor performance in OOD scenarios. Furthermore, calibration methods like the Platt scaling can improve the confidence calibration in-domain, but their calibration effectiveness remains limited under domain shift scenarios. This is because this calibration technique mainly adjusts the output probabilities but does not fundamentally address the biases introduced by feature representation changes across distributions. Table. 3 shows the OOD performance of the baseline methods.

**Table 3: OOD evaluation of the graph spectral(w Platt scaling) trained on the TriviaQA from Llama3 responses and tested out-of-domain datasets**
| Dataset | Method | Brier | AUROC | ECE |
| --- | --- | --- | --- | --- |
| llama3 CoQA | GraphSpectral(w Platt scaling) | 0.17 | 0.72 | 0.095 |
| Apricot | 0.24 | 0.59 | 0.154 |  |
| Ours | 0.13 | 0.77 | 0.077 |  |
| Vicuna TriviaQA | GraphSpectral(w Platt scaling) | 0.24 | 0.53 | 0.07 |
| Apricot | 0.19 | 0.76 | 0.13 |  |
| Ours | 0.17 | 0.81 | 0.07 |  |
| Vicuna CoQA | GraphSpectral(w Platt scaling) | 0.35 | 0.55 | 0.26 |
| Apricot | 0.24 | 0.59 | 0.08 |  |
| Ours | 0.22 | 0.73 | 0.10 |  |

### 4.4 Sensitivity Analysis

In this subsection, we conducted several sensitivity analyses of our model.

Number of training samples
We conducted experiments to examine the relationship between performance and the amount of training data. Specifically, we tested our model performance on the Llama3 TriviaQA dataset and varied the training size from 100 to 4000.
The results are displayed in Table  4. We observed that the model’s performance does not drop significantly with the reduced training data.
These experimental results indicate that the model performs well with limited data availability, demonstrating its applicability in real-world scenarios where only smaller datasets are available. We also tested the baseline performance, the results are shown in Appendix A.5.

**Table 4: Performance under varying Training Sample Sizes**
| # of Training Samples | ECE | AUROC | Brier |
| --- | --- | --- | --- |
| 100 | 0.095 | 0.770 | 0.201 |
| 300 | 0.062 | 0.786 | 0.187 |
| 500 | 0.049 | 0.792 | 0.181 |
| 1000 | 0.037 | 0.799 | 0.177 |
| 4000 | 0.020 | 0.820 | 0.172 |

Hyperparameter sensitivity
We conduct the sensitivity analysis of our model’s calibration error performance concerning two key configurations: the number of sampled answers used to construct the graph and the number of Graph Convolutional Network (GCN) layers in the GNN model. The results are displayed in Fig. 3. The experiments are conducted using the Llama3 model on the TriviaQA dataset. For Fig. 3 (a) experiments, we varied the number of sampled answers from 10 to 50 while keeping other configurations and hyperparameters fixed, as described in the experimental setup. We observe that increasing the number of sampled answers slightly improves performance, which then stabilizes. In Fig. 3(b), the sensitivity to the number of GCN layers indicates that our model remains stable with 1 to 4 layers, with the best performance observed at three layers.

Figure: (a)
Refer to caption: x10.png

## 5 Conclusion and Future Work

In summary, in this work, we proposed one effective strategy of confidence calibration by combining the LLM’s self-consistency with labeled data and training an auxiliary GNN model to estimate the correctness of its responses to questions. Experiments demonstrate that the proposed approach improves confidence calibration significantly across several datasets compared to baseline methods. We acknowledge that one limitation of our approach is the computation time of sampling multiple answers from LLMs. However, this cost can be effectively alleviated by parallel sampling. In the Appendix A.2, we provide the details of the sampling time. In future work, we aim to extend the framework to incorporate the data uncertainty coming from ambiguous questions and also explore the multi-step confidence calibration in the chain-of-thought framework.
Our approach relies on a consistency graph to calibrate confidence.

Large Language Models (LLMs) are powerful tools that deliver valuable information across a wide range of applications. However, the misinformation they may generate can lead to significant societal issues. Our calibration model evaluates the accuracy of an LLM’s responses, thereby enhancing their reliability. When integrated into real-world systems, our model would enable LLMs to abstain from answering uncertain queries or allow users to decide the level of trust they place in the responses. We anticipate that our research will positively influence the responsible deployment and use of LLMs in society.

## Appendix A Appendix

### A.1 Hyperparameters and Model configurations

Model hyper-parameters:

Our model used three GCN layers; typically, the embedding dimension was $256,512,and1024$ for three GCN layers. For the training process, we used the binary cross-entropy loss with a decaying learning rate that reduced the learning rate by $0.9$ if the validation loss did not improve $10$ epochs (with an initial learning rate of $10^{-}4$ and a minimum learning rate of $10^{-7}$). The optimizer was Adam with $\beta_{1}=0.9$ and $\beta_{2}=0.98$. The batch size was 32. For the rephrased prompts, we set $k=3$, $n=30$, so for each rephrased question, we sampled ten answers. While calculating the ECE, we divide the confidence into $B=10$ bins.

Evaluation Setup:

For each question, we evaluate the confidence prediction corresponding to the most likely answers from the LLM response. The setup is consistent with the baseline methods.

Graph construction:

For each question, we prompt the LLM to give 30 answers, and the temperature for LLM is set to be 0.6. For each answer, the SentenceBert model Reimers & Gurevych () is used to get each answer’s embedding. The cosine similarity between each answer’s embedding is taken as the edge weight of the graph. We apply the K-Means clustering method to cluster similar semantic responses. The maximum cluster number is set as 3.

### A.2 Computational cost

We performed all experiments on NVIDIA A100 GPUs with 80GB of memory. Generating 30 responses using the Llama3 and Vicuna models for 6000 questions from CoQA and TriviaQA data required up to 4 hours, with an average of approximately 2 seconds per question. The CoQA dataset demanded more processing time due to the longer contextual information in the input. The time can be shortened by parallel sampling.

### A.3 Additional Cases

To better understand our method intuitively, we have collected a few examples to show the difference between our algorithm and APRICOT.

To summarize our observation here:

1. Multiple responses to the same question does reveal the LLM’s confidence in its answers.
2. The LLM’s self-evaluation of confidence is often much higher than it should be – the LLM is overconfident about its responses.
3. The chain-of-thought responses used by ApriCoT add some information to make each answer more complete and reasonable in the spirit of 1, but it mainly adds the information within one response, not as much information as the multiple responses used by ours.

Example 1:

Question: Who plays Captain Jack Sparrow’s father Edward Teague in the Pirates of the Caribbean films?

True answer:: Keith Richards

LLM response: David Schofield

More responses from the LLM: Martin Klebba. Keith Richards, Geoffrey Rush, Martin Klebba. Keith Richards. Martin Klebba. David Schofield. (only list 7 responses here to save space)

GCC-estimated confidence: 0.23

CoT response: David Schofield,

Self-evaluation: $80\%$ (with the further question ”Please provide your confidence in the answer only in percent (0-100 %):”)

ApriCoT-estimated confidence: 0.79

Example 2:

Question: In which film will you find the Rodger Young?

True answer:: Starship Troopers

LLM response: The Bridge on the River Kwai.

More responses from the LLM: The Greatest Story Ever Told. The Best Years of Our Lives. The Bridge on the River Kwai. The Best Years of Our Lives (1946). 1949’s Battleground. The Best Years of Our Lives.

GCC-estimated confidence: 0.22

CoT response: All the President’s Men.

Self-evaluation: 95%

ApriCoT-estimated confidence: 0.81

Example 3:

Question: BS is the international car registration of which country?

True answer:: Bahamas.

LLM response: Germany.

More responses from the LLM: Bahamas. Bahrain. Bangladesh. Bahamas. Belgium. Bahamas. Germany. Bhutan. Belgium.

GCC-estimated confidence: 0.34

CoT response: Belgium

Self-evaluation: 98%

ApriCoT-estimated confidence: 0.61

### A.4 Additional Visualizations

Besides the cases we show in the previous section. Here, we present several case examples and visualize the response patterns. We performed dimension reduction of LLM’s responses to different questions and then plotted their embeddings to the 2-dimensional space.
Fig 4 shows the responses generated by Llama3 as an example. From the figure, we observe that answers with higher confidence levels tend to cluster closely together, indicating consistency and reliability in these responses. In contrast, answers with lower confidence levels exhibit greater diversity, reflecting a broader range of possibilities. This behavior aligns well with our initial assumption, demonstrating that higher confidence responses are more consistent, while lower confidence responses capture a wider variety of potential answers.

Figure: Figure 4: Visualization of the generated response patterns
Refer to caption: x12.png

### A.5 Additional results

Additional reliability plots We showed all reliability diagrams for Llama3 for TriviaQA in Fig. 5 and CoQA dataset in Fig. 6. To summarize the trends, we observe that Platt scaling narrows the range to the middle value. Verbalized uncertainty cannot generate a wider range of confidence values. GraphSpecral with Platt tends to generate a wider range of confidence values, but the bias can not be improved across all cases, resulting in the bar height not following the diagonal line closely. Our model can predict a wider range of confidence values and achieve better calibration in all settings, with the auxiliary consistency graph and clustering features contributing to improved calibration overall.

Figure: (a)
Refer to caption: extracted/5973685/sections/figures/3.png

Figure: (a)
Refer to caption: x15.png

Additional baseline results In Table 5, we showed the performance of the baseline method under varying training sizes. As the number of training data decreases, the ece will drop from 0.096 to 0.165.

**Table 5: Performance under varying Training Sample Sizes for the baseline methods(Apricot)**
| # of Training Samples | ECE | AUROC | Brier |
| --- | --- | --- | --- |
| 100 | 0.165 | 0.611 | 0.229 |
| 300 | 0.133 | 0.634 | 0.211 |
| 500 | 0.112 | 0.695 | 0.204 |
| 1000 | 0.105 | 0.722 | 0.192 |
| 4000 | 0.096 | 0.743 | 0.187 |

### A.6 Prompting strategy

Here, we showed the prompts we used to generate the rephrasing questions.

You are a helpful assistant. I have a question that
I would like to see it rephrased in multiple ways.
Please take the original question and generate
several rephrased versions while maintaining the
same meaning. The question can only have
one direct answer. Here is the original question:
…. Please provide four distinct rephrases of
the question.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1.	Are the confidence values of answers in the same cluster the same?

2.	When verifying the ability of the proposed method on OOD in Section 4.3, I recommend that the author add other datasets of tasks that are quite different from the training data to verify the generalization ability, such as Math
soundness: 2
strengths: 1.	The paper presents a novel method to confidence calibration in large language models by leveraging graph neural networks and the consistency among multiple model responses. 

2. The manuscript is well-organized and the experiments are comprehensive.
summary: To enhance the calibration performance of LLMs, this paper proposes to combine the LLM’s self-consistency with labeled data and train an auxiliary model to estimate the correctness of its responses to questions. Experiments demonstrate that the proposed method outperforms baselines in confidence calibration on two datasets.
weaknesses: 1. The premise proposed in this paper is if LLMs give similar response. Then there is less uncertainty, and these responses tend to have a high probability of being correct.  It is common to use the idea of self-consistency to guide the selection of the most appropriate answer from multiple answers, but it may not be reasonable to use it as a probability to evaluate the correctness of an answer. For example, the LLM may exhibits generate the same wrong answer when sampling multiple answers. In this case, is the proposed method still effective? 

2. This paper trains another model to calibrate the correctness of the answers generated by LLM. What is the difference between this auxiliary model and the "reward model" in the LLM evaluation work, which is used to evaluate the quality of responses? If there is a difference, the author should point out the difference with this type of work; if it is similar, the author should add a description of these works and performance comparison in related work and experiments.

3. This work uses semantic similarity to cluster different answers. Even so, answers within the same cluster may still be completely different answers. Because they just exhibit the similar reasoning paths but gives the different final answers. This phenomenon is also mentioned in paper [1]. But according to Equation 3, the author calculates the correctness probability of each cluster using the ratio of numbers. Why? How to explain the reasonability?
[1] SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales

4. In Method Section, the details how the training dataset was created are unclear. For example, for the same question, will the confidence information of each cluster be input into the GNN? It is recommended that the author add a paragraph to describe the complete process of constructing GNN training data.

5. This paper constructs training data based on the results of self-consistency. The author should add a comparison with the performance of the simplest self-consistency method to estimate confidence in the baselines.

6. Section 4.3 lacks analysis of DDL results.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: * The paper considers an important problem (LLM confidence estimation).
* A diverse set of baseline methods are included (although I would have liked more information about how they were tuned compared to the baseline).
* The approach is applicable to “black box” models since it only relies on being able to sample from the model, rather than requiring access to the activations or full predictive distribution.
* The idea of using a GNN to calibrate a set of responses given a measure of semantic similarity between them is appealing.
summary: This paper considers the problem of producing meaningful uncertainty estimates given samples from a large language model. The approach relies on repeated sampling the LLM and forming semantically equivalent clusters of responses. These clusters are converted to a fully-connected similarity graph based on SBERT similarity between the responses contained in those clusters, which is the input to a supervised post-hoc calibration graph neural network (GNN). The node classification problem is to predict for each response if it is correct. The GNN does not take as input the SBERT embedding vectors to avoid dependency on textual information, but uses the cluster IDs and the similarity between each response. The clustering is obtained using the k-means algorithm. The appendix says that 30 responses are sampled and that K is set to 3 for k-means. The robustness of the approach is further (marginally) improved by forming multiple prompts for each question to increase the variety of response; further variety is associated with greater uncertainty. The proposed approach is compared to a diverse set of baseline methods on two standard QA tasks and is found to perform favourably in terms of calibration performance. The approach performs well with only small amounts of training samples.
weaknesses: * The use of a fixed number of samples and K for K-means seems like it could introduce some issues. If there’s no/little ambiguity won’t this still return three different clusters? At least some more sensitivity analysis should be done for K but this seems like a fundamental issue with the approach since choosing the number of clusters is underspecified in general.
* The quality of the approach seems like it depends strongly on the semantic similarity used. SBERT is quite an old approach and it’s unclear how well it generalises to various settings. I would have liked to see more analysis of the choice of semantic similarity. 
* The evaluation is conducted on two public QA benchmark datasets from 2017 and 2019, which are almost certainly in the training dataset for many LLMs. How does this impact performance?
* There is no meaningful discussion of the limitations.
* Regarding the writing, it should be made more apparent what data is treated as "calibration data" vs. "validation data" vs. "test data"
* Overall, my biggest concern with the paper that the evaluation is limited to two fairly "easy" QA datasets. This makes it difficult to gauge how general the proposed approach is.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: refer to the weaknesses
soundness: 3
strengths: 1.	The topic of the paper is important. A well calibrated confidence of LLM outputs can benefit a lot of domains.
2.	The model exhibits strong generalizability across different datasets and LLMs, showing robustness against domain shifts, which is valuable for real-world applications.
3.	The paper includes sensitivity analyses and comparisons across various configurations, demonstrating the stability and effectiveness of the method under different setups and highlighting the performance benefits over established baselines.
summary: The paper presents a graph-based method for confidence calibration for LLMs, aiming to improve reliability by estimating response correctness through consistency. A graph neural network (GNN) is trained on a similarity graph of multiple LLM responses to a question, predicting correctness based on response consistency without processing language directly. Experimental results show substantial improvements over baseline methods in calibration performance and out-of-domain generalization.
weaknesses: 1.	The requirement for sampling multiple responses and constructing similarity graphs for each query can introduce substantial computational costs, limiting scalability for real-time or resource-constrained applications.
2.	The motivation is not clear. First, how to choose the value of \tau in Eqn. is an extra challenge. Second, after having the correctness label, why do we need to leverage a GNN for classification? I think it is a typical text classification task. The motivation of choosing GNN is not clear. 
3.	The reliance on Sentence-BERT embeddings for similarity computations could introduce biases and inconsistencies if embeddings do not fully capture semantic equivalence, potentially affecting calibration accuracy.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Figure 1 is not very informative, it does not provide the intuition why the proposed method chose to use GNNs
2. In line 163-164, it is supposed to provide proof here: theoretical studies, empirical studies, or examples 
3. the use of the ROUGE metric for judging the correctness is problematic. ROUGE does not account for semantic meanings and is sensitive to length and word orders. Therefore, it is not an appropriate metric to evaluate whether two responses are semantically equivalent 
4. CoQA and TriviaQA are short-answer datasets. It is useful to test on datasets with long-form answers such as TruthfulQA, which can show the generality of the proposed method
5. in line 63-64, " A response consistent with more answers tends to have a higher likelihood of being correct". A high consistency is an indicator of high confidence rather than high correctness.
soundness: 2
strengths: 1. the paper is easy to follow 
2. This work develops a graph-based method to determine the semantic uncertainty, which is interesting
3. better performance in OOD settings
summary: This work proposes a GNN-based approach to calibrate confidence scores of LLM responses. The proposed method is an extension of the Semantic Uncertainty,  a sampling-based method that estimates uncertainty by using the semantic distribution.
The main distinction of the proposed method is that it treats each response as a node and applies GNNs to obtain the logit. Next, cross-entropy loss is introduced to calibrate the node score. Experimental results on short-text datasets CoQA and TriviaQA show that the new method can outperform other baselines and lead to a better generality in OOD settings.
weaknesses: * **Main concern**:  the scientific contribution of this work is limited. The proposed method is an extended work of the semantic uncertainty with GNNs
* writing and presentation should be improved. Some claims appear too arbitrarily without any proof. Figure 1 is not very informative. (see questions below)
* the proposed method requires labels, but other baselines such as self-checkgpt and semantic uncertainty require no further training
* the use of the ROUGE to evaluate semantic equivalence is problematic since it does not capture semantic meaning well; in the experimental part, only short-answer datasets are used. It is important to include long-form generation dataset