Graph-based Confidence Calibration for Large Language Models
Authors: Yukun Li
Sijia Wang
Lifu Huang
Liping Liu
Venue: Accepted by TMLR ()

Abstract: Reliable confidence estimation is essential for enhancing the trustworthiness of large language models (LLMs), especially in high-stakes scenarios. Despite its importance, accurately estimating confidence in LLM responses remains a significant challenge. In this work, we propose using an auxiliary learning model to assess response correctness based on the self-consistency of multiple outputs generated by the LLM. Our method builds a consistency graph to represent the agreement among multiple responses and uses a graph neural network (GNN) to estimate the likelihood that each response is correct. Experiments demonstrate that this method has strong calibration performance on various benchmark datasets and generalizes well to out-of-domain cases.

Published in Transactions on Machine Learning Research (5/2025)
Graph-based Confidence Calibration for Large Language Models
Yukun Li
yukum. Cettmlii@tufts.de
Deportment of Computer Science Tufts University
Sijia Wang
stjiauangg@ut..c edu
Deportment Computer Science
Virginia Tech
Lifu Huang
Yahuang@@edaviss. edu
Department of Computer Science Virginia Tech, University of California, Davis
Li-Ping Liu
liping.luutuuff Huu Wuwtuufs...... edu
Department of Computer Science Tufts Universily
Reviewed OpenReview: https: // opwnrevie...eeee... net/ formmo?m? id= BDPvuD5FTg
Abstract
Reliable confidence estimation is essential for enhancing the trustworthiness of large lan- guage models (LLMs), especially in high-stakes scenarios. Despite its importance, rately estimating confidence in LLM responses remains significant challenge. In this work, we propose using an auxiliary learning model to assess response correctness based on the self-consistency of mnlltpple outputs generated by the LLM. Our method builds a consistency graph to represent the agreement among multiple responses and uses a graph neural network (GNN) to estimate the likelihood that each response correct. Experiments demonstrate that this method has strong calibration performance on various benchmark datasets and generalizes well to out-of-domain cases.
1 Introduction
In recent years, large language models (LLMs) have demonstrated remarkable capabilities across various natural language processing tasks such as question answering (Wei et al., 2022; Shen et al., 2023; Zheng et al., 2023; Qin et al., 2023; Singhal et al., 2023), text summarization (Tang et al., 2023; Deroy et al., 2023; Williams, 2023; Wang
Tam al., 2023; Roit et al., 2023), and even creative writing (Gómez-Rodríguez
et al., 2024; Deng al., 2024). Despite their impressive performance, LLMs often give incorrect responses in question-answeringg tasks. One particularly important challenge lies in calibrating the confidence levels of LLM-generated responses (Kuhn et al., 2022; Ulmer al., 2022; Van Landeghem al., 2022; Vazhentsev et al., 2023; Ulmer et al., 2024). Accurate confidence estimation is vital for deploying LLMs in the real world, as it enables users to gauge the reliability of the model's predictions and make informed decisions accordingly. On the contrary, miscalibrated confidence may lead to over-reliance on incorrect responses or unnecessary skepticism toward the correct ones. For example, misleading response may steer a patient in the harmful direction when making health decisions; it may also cause an investor make impulsive financial choices.

---

Published in Transactions on Machine Learning Research (5/2025)
In this work, we focus on calibrating LLMs' confidence to better reflect the correctness of their responses. This task is challenging in several aspects. First, due to LLMs' superior ability to generate text, mistakes in their response often occur at the semantic level, making them hard to detect even for humans. There are methods using an auxiliary Language Model (c.g., DeBERTa (He et al., 2020)) to verify whether the LLM's response appropriately answers the question (Ulmer el al., 2024). Since the LLM supposed to be much stronger than the LM, the LLM should be able to avoid most mistakes that can be detected by an LM; so this type of method may omit a significant fraction of wrong answers. Second, is hard to detect mistakes from the LLM's internal working mechanism. Because the LLM uses many hidden layers to process the information, it is hard to discern the signal from a small number of hidden units. Even if this is possible, it is not easy to apply this type of method to black-box LLMs.
Recently, there has been some progress in quantifying the model's confidence in its own responses through consistency among the outputs generated by the model itself (Chen & Mueller, 2023; Lin et al., 2024). These approaches demonstrate a strong correlation between an LLM's self-consistency and the actual correctness of its responses. However, because these methods depend hand-crafted features, they often fail to accurately calibrate confidence, resulting in mismatch between predicted confidence levels and the true accuracy of the answers. This raises an important research question: can we improve confidence calibration by learning from patterns of consistency across the LLM's own responses?
In this work, we propose an auxiliary learning model to improve confidence calibration. We begin by constructing similarity graphs from the LLM's multiple responses to the same questions, where graph edges represent the degree of agreement between responses. We then train separate calibration model using graph neural networks (GNNs) to predict the correctness of each response. The key insight is that consistency among responses carries strong signals of correctness for instance, a response that aligns well with many others is more likely to be accurate. Importantly, our model operates solely on response consistency and does not analyze the actual language content. This work focuses on practical scenarios where correctness reflects alignment with the training data, and docs not address the case where training data itsolf contains consistent but incorrect information. The latter case is a more challenging scenario investigaled by ongoing research (Biester et al., 2024; Shi et al., 2023; Krishnan & Wu, 2019).
We further investigate the problem of transferring calibration model across different question domains, which is crucial when larget domains lack sufficient training data. Despile its importance, this problem has received limited attention in the literature. Our study demonstrates that the proposed auxiliary calibration model can generalize to new domains with minimal performance degradation. This generalization is from the observation that self-consistency can serve as a broadly applicable signal for confidence, enabling the learning model that relies solely on self-consistency to achieve strong transfer performance.
We evaluate the performance of the proposed method with an extensive empirical study that includes four datasets from different question domains. Empirical results show that method achieves strong per- formance. Besides the improved calibration performance, our model enhances the ranking of an LLM's responses. The study has also tested our model and competing models in out-of-domain settings. The results show that the proposed method shows robust performance when generalizing to new domains.
In summary, our main contributions are:
A learning-based GNN framework: We propose a learning- based framework leveraging GNNs to calibrate confidence values of LLMs' responses.
Enhanced calibration performance: We conduct an extensive empirical study to evaluate the proposed method and show that it substantially outperforms recent methods in confidence calibration across several widely used benchmark datasets.
Improved out-of-domain generalizability: We investigate the scenario of oft-of-domain (OOD) confidence calibration and show the superior performance of the proposed method in this setting.
2

---

Published in Transactions on Machine Learning Research (5/2025)
2
Related Work
Due to the urgent need improve the reliability of LLMs, confidence estimation and calibration for these models have become active areas of research. Existing research in LLM uncertainty quantification can be summarized into two main categories: uncertainty quantification and confidence calibration (Geng et al., 2023). Confidence estimation for short responses (e.g., for multi-choice or yes-no questions) is generally less complicated than for long responses (Ye et al., 2024). For a brief response, the LLM's output logits are informative about its confidence; the easy comparisons of responses to the true answer facilitate both calibration and evaluation. Confidence estimation for long responses cannot simply depend on LLM's s output logits (Duan et al., 2023; Bakman al., 2024) because the logits indicate more about the probability of text and less about the semantics behind it. There are also methods using the internal state of an LLM (Ren et al., 2022; Beigi al., 2024), but it is not always available to have such information about the LLM interface.
Another approach is to check the LLM's consistency in its responses. Kotelanski et al. (2023) demonstrate that repeated sampling and consistency chocks across multiple outputs can serve as reliable proxies for model confidence. Manakul et al. (2023) generate multiple responses from the LLM and check the consistency between responses using various methods, including querying the LLM. Chen Mueller (2023) combine the consistency between responses and the LLM's self-reflection certainty quantify the uncertainty. Kuhn et al. (2022) consider confidence from semantic equivalence and proposes method based on clustering of responses. Lin et al. (2024) organize responses in graph with their pairwise semantic similarity and then extract graph statistics for confidence estimation. ZZhang et al. (2024) examine methods of comparing responses via entailment and cont contratiction relationships. These studies highlight the importance of semantic consistency in ranking an LLM's responses. However, manually designed features are limited in their ability to capture the full extent of self-consistency among LLM responses, leading to poor calibration performance.
To better calibrate the confidence estimation, some methods directly use correctness labels in their calibration procedures. Mielke et al. (2022) train a calibrator to predict the correctness of a response for given question. With similar idca, Ulmer et al. (2024) train language model (c.g., DeBERTa) based on question-response pairs to predict the probability of responses' correctness. Based on SelfCheckGPT (Manakul et al., 2023) and JARC (Tian et al., 2023), Chen et al. (2024) train supervised models to reduce grouping losses and improve the confidence estimation. The method by Liu et al. (2024) uses an LLM's latent representations to predict the correctness of responses. Detommaso et al. (2024) use the "multicalibration" technique calibrate the probability of correctness. Fadeeva al. (2023) offer a detailed comparative study of various confidence estimation methods, providing empirical evidence their effectiveness across different tasks. However, these studies have not sufficiently exploited response consistency to predict the probabilities of the responses being correct.
3 Method
Our ultimate goal is to quantify the probability of the correctness of response from an LLM. Since the LLM can give a correct answer with different phrases, we need to consider the probability that the response is semantically correct.
Background: The formulation semantic equivalence (Kuhn et al., 2022) provides a framework for our analysis. Let R be the space of all possible responses. Given a question g, the space R is divided into a set Cy of semantic classes: R Ucee, C and C' C'nC = 0 for any two different semantic classes C² E Cg. Two 72 r2
responses rı,
C in the same equivalent class are considered as the same semantic response: if one the correct answer, the other is correct well. Under an LLM, the semantic response C has probability:
p(Clg) Σ p(rlg).
(1)
rEC
Here p(rlg) is the probability of a single response from the LLM. We also say that p(Clq) the LLM's confidence in the semantic response C.
3

---

Published in Transactions on Machine Learning Research (5/2025)
However, it is non-trivial to define the equivalent class, and we will discuss the approximation later. To estimate p(C]g), one approach is through semantic similarities between response samples of an LLM for the same question 9. Let (ri....,TT) be n responses from the same question 4, and they form & clusters C₂ = - C.) by their semantic similarity. We can use natural language inference (NLI) systems to predict the relationships (e.g,, entailment and contradiction) bet responses and derive their similarity.
We assume that each cluster Ö is from different semantic class C, then p('(q) can be approximated by
p( 101
(2)
From the cluster probabilities, the uncertainty of the LLM on the question q is estimated as the entropy of the empirical distribution over clusters (Kuhn et al., 2022), and the confidence of response r, C is estimated as |Cl/n (assuming similarity values are binary) (Lin al., 2024).
Now, we depart from the sctup of semantic classes and consider the correctness of responses. Let C* be the correct semantic answer to question q. Without knowing C*, a common assumption is that the model's confidence p(Clg) in the semantic response C reflects the correctness. For example, p(C(1) is approximately the probability of correctness:
p(Cx CC* |CK/
(3)
With this assumption, the more certain the model about semantic response, the more likely the response is correct. Conversely, a wide variation in the LLM's responses indicates low confidence in all responses ri and low accuracy. This pattern is also found in previous studies (Kuhn al., 2022; Lin et al., 2024).
While a positive correlation exists between the LLM's confidence and the probability of correctness, the two quantities are unlikely to have the equal relationship shown in (3). Therefore, we need further calibration to estimate the probability of correctness.
people Caltomiaa more days in year than we used lo?
14
people
luy ssci
more days in year
SBERT
emblrs)
Mij
(a) Edge weight
The overaee mumber of workdays year has not changed significanlly in Caliioonii in socent years
simes (emb(rr) (ri), emb(r.))
emblry)
Yes, people Califoriia weck more days in year Gan faey used beaause of the elimination of daylight-saviagg saving time
Chustering
Califonnia work did 1979
days per
No, people in Californio do not woek more dagss In a year that they used to
(b) Noda feature with ClusterID
Prompt LLM
Construct Graph with (a) and (b)
Obtain confidence with GNN
High confiderice score Law confidencze score
GNN
G E,)
The overall framework of our confidence calibration model. Given an input question, our approach first generates multiple responses from the LLM and construets similarity-weighted graph based these responses. This graph serves as the input for the GNN model, which calibrates the confidence of the LLM responses. In the weighted graph, the edge weight wij is defined as (emb((( emb(r;)), where 1.j = 1 1,...,n. n. A higher weight indicates greater similarity between the responses. We also use cluster memberships as the node features to enhance the performance.
Figure 1:
4

---

Published in Transactions on Machine Learning Research (5/2025)
3.1 Confidence calibration graph learning problem
Now, we set supervised learning problem and train model to calibrate the confidence of the correctness of responses. We first consider the correctness labels of the LLM's responses. In the supervised setting, we have a ground-truth answer a to the question Then we use a to assign correctness labels to sampled responses (r1, Tu. for the same question q.
In our work, we have two approaches to label these responses. In the first approach, we use the ROUGE similarity. Specifically, we compute the ROUGE similarity simm(re, a) between a sampled response and the correct answer to decide the correctness label.
y; 1simmp r;) 2 7). i I.....
(4)
Here 1[-1 is one if the condition is true or 0 otherwise. As shown in previous studies (Lin & Och, 2004) and our own study, the ROUGE metric is reasonably accurate in measuring semantic similarity between short sentences. We follow the previous work, and set r 0.3 (Kuhn et al., 2022).
In the second method, we utilize the LLM to generate correctness labels. Specifically, we provide the question q and the standard answer as the context, then ask whether the response r; answers the question q- The response from the LLM is then used as the label for rj. We denote the procedure as
y lm,,(g,a,r.))
(5)
We provide the prompt for labeling in the Appendix F.
Both methods are automatic and can scale up to large datasets. To guaranteed the labeling quality, we also include a relatively small set of manual labels in our study. For cach question, human labeler inspocts the question q and the true answer a, and then assign the correctness label yı to response ri.
We then consider the input to the calibration model. We form similar graph G over responses to encode information about their consistency. The graph contains the clustering structure of responses and likely further useful information to predict the correctness of responses. The graph G = (V, E, w) is a fully connected graph, with the node set V consisting of n responses and the edge weight Wiy being the similarity between the pair of responses (ri,r;), r;). We compute the similarity from the two responses" embeddings. particular, we first use the Sentence- BERT model (Reimers & Gurevych, 2019) to compute the two responses* vector representations and then compute the cosine similarity
wiy = simeos simeos(emb(r)) (emb(r:), emb(r,)), i,j = I....,n.
(6)
Here, emb(-) represents the embedding function.
Then, we treat the problem as a node classification problem (Xiao et al., 2022). In particular, we run a GNN gun(.) to predict the probability of cach response being correct
P gnng(G).
(7)
Here p [0, 1]" contains the probabilities for n responses being correct.
To provide clustering information to the GNN, we first run the K-means clustering algorithm the responses' embeddings and assign cluster ids from 0 to K - 1 based on the order from largest to smallest (ties are randomly broken). Then, we feed each response's cluster membership as a one-hot feature input to the GNN. Therefore, the GNN's predictions are purcly based on the relationships between responses in somantic space. We choose NOT to feed in the embedding vectors of responses to avoid the GNN's dependency on textual information. This helps the GNN to generalize to questions from different domains. The overall framework is shown in Fig 1.
The main purpose of the learning model is to calibrate P. One approach is to minimize the cross-entropy loss of p against correctness labels. The loss computed from the question q is
lg = Σ yi log P. + (1- - ys) log(1 1 p:)
(8)
5

---

Published in Transactions on Machine Learning Research (5/2025)
Note that the loss is consistent marginally since the loss is minimized when p. p(us/G). An alternative approach is to minimize the squared error (y: -P)); from which get similar performances. We choose the cross-entropy loss in our work. A further consideration is to explicitly consider the similarity between p; and Py given the response similarity Wij. We leave such exploration to the future.
3.2 Improve the estimation through multiple prompts
It is well known that the syntactic form of question influences responses and introduces additional variance. The variance of question forms from users may pass to the variance of responses and cause even diverse analyze
responses due to the lack of confidence. To account for question variances in real applications, the LLM's responses to multiple prompts derived from the same question. These responses are treated as answers to the same semantic question. We then apply the same method as before to predict the correctness of each response.
In particular, we rephrase the original question q into different forms {qi.... 9 while maintaining the original sentence's semantic meaning. We employ a multiple rephrasod questions strategy for answer sam- pling. Specifically, we prompt the GPT-4 to give & different but. with the same meaning rephrased questions for the given question q. Then, we sample n/k responses from the LLM for each rephrased question and still get a total of n responses, from which the confidence calibration is the same as we have described above. For questions about which the LLM is less certain, the model is more likely to produce diverse responses. In this scenario, confidence calibration is more accurate because the model's uncertainty becomes more apparent.
4 Experiments
We evaluate the calibration performance of our proposed framework by comparing it against baseline meth- ods. We also test their performances the OOD setting, where their calibration parameters are decided on one dataset, but they are tested on a different dataset. We further analyze the quality of automatic labels and the sensitivities of our method under different Sypter-paramed settings.
4.1 Dataset and experiment setup
Dataset:
We conduct experiments on four datasets: (1) CoQA (Reddy et al., 2019), open-book conver- sational question answering task; (2) TriviaQA (Joshi et al., 2017), a commonsense QA task. (3) Truth- fulQA (Lin et al., 2022a), comparably more challenging dataset for factual QA tasks. and (4) HotpotQA, a qucstion answering dataset that requires models to find and combine information from multiple passages to answer complex questions. We repeat the experiments 10 times, each with a different train/val validatonn split and test the performance on the test set.
Baselines: We compare our methods with the following basclines. Length-normalized sequence likelihoods (Seq. likelihood) (Malinin & Gales, 2021; Kuhn et al., 2022) is standard measure for confidence. This method calculates the likelihood of each sequence and normalizes it by the length of the sequence to provide a fair comparison between different lengths of sequences. Platt scaling (Platt, 1999), a variant of the sequence likelihood baseline, applies Platt scaling to the raw likelihoods.. GraphSpectral (Lin et al., 2024) uses the graph theory to estimate the confidence. Then we also include post-hoc uncertainty calibration, GraphSpectral+1soo and GraphSppetral+PPltt into the baseline methods. Self-check GMa(Ma et al., 2023) checked the consistency between responses querying the LLM. Verbalized Uncertainty (Lin et al., 2022b; Tian et al., 2023; Xiong et al., 2024) generates verbal statements about the model's confidence in its predictions. Verbalized Qual maps the confidence percent (Verbalized %) into numerical values. APRICOT (Ulmer el al., 2024), a supervised method, fine-tunes the Deberta language model to predict confidence scores for LLM outputs. Furthermore, we also include the baseline applying two post-hor uncortainty calibration methods, APRICOT+Iso and APRICOT+Platt, to adjust the confidence scores obtained by Apricot. We performed all the baseline experiments utilizing the open-source codebase and used the default. hyperparameters.

---

Published in Transactions on Machine Learning Research (5/2025)
Graph construction: For each question, generate 30 candidate answers through LLM prompting. Fach generated response is then processed using the Sentence-BERT (SBERT) model (Reimers & Gurevych, 2019) to obtain the answer's high-dimensional embeddings. To quantify the semantic alignment between all responses, we compute the cosine similarity between every pair of answer embeddings. These scores are then utilized as edge weights in our similarity graph, where each node represents an individual answer, and the odges weights signify the degree of semantic relation between them.
Model hyper-parameters. To ensure our model can capture complex and abstract features at each layer, our model comprises three GNN layers, with embedding dimensions of 256, 512, and 1024 for the first, second, and third layers, respectively. The initial learning rate was set to 10-4. If the validation loss did not show improvement over ten consecutive epochs, the learning rale was reduced by a factor of 0.9. The optimization was performed using the Adam optimizer, configured with hyperparameters B1 = 0.9 and B2 = 0.98. The batch size was 16.
LLMs: We assess our confidence calibration method on two LLMs with good performance: Llama3-8B (Maama 2024), and Vicuma-7b-v1.5 (Vicuna) (Zheng et al., 2024).
Labcling the data: To obtain the correctness label for CoQA and TriviaQA datasets, we followed previous work (Kuhn et al., 2022) and used the ROUGE-L metric for labeling. For the TruthfulQA dataset, given its focus on factual correctness and longer answers, we employed GPT4 (Potsaweel, 2024; Liu et al., 2023; Badshah & Sajjad, 2024) to generate the labels. In addition, we conducted experiments on a smaller subset of data using human- annotatd labels to validate the reliability of our automatic labeling process. Details of this manual annotation study are provided in Section4.4.
Evaluation metrics: The evaluation metrics include Expectation Calibration Error (ECE), Brier Score, and AUROC, Specifically, (1) ECE quantifies the consistency between the prediction error and the uncertainty of the prediction. An ideal calibration curve should exhibit a lower ECE. It measures the consistency between the prediction error and the confidence of the prediction. Specifically, the confidence interval is grouped into fixed bins, and the average of the difference between the confidence and error in each bin is compared. Formally, ECE is calculated as ECE- Σm ne lace(b) conf where ny is the number of predictions in bin b, N is the total number of data points and acc(b) and con are the accuracy and confidence of bin b, respectively. (2) Brier Score (Brier, 1950), which is the mean squared difference between predicted probabilities and the actual binary results. Lower Brier Scores indicate better performance. (3) AUROC to indicate the models' discriminatory ability.
Further experimental configurations and prompting strategy are provided in the Appendix. The experiments are conducted on NVIDIA A100 GPUs with 80GB of memory.
4.2 Experiment results
For the Tlama3 model, the confidence calibration performance on TriviaQA is shown in Table For the TriviaQA dataset, it can be observed that the likelihood method performs poorly on the calibration error (ECE and Brier Score) and AUROC due to unreliable model prediction probability (Zhang et al., 2024). Platt scaling improves the ECF post-calibration and enhances the model's discriminative ability, resulting in higher AUROC results. However, this method cannot capture the semantic equivalence among answers, leading to sub-optimal performance. The Verbalized and Verbalized Qual prompts LLM to output confidence for their answers, improving AUROC by - 5% compared with the likelihood baseline. However, it faces the overconfidence issue; thus, the calibration errors are still high. The GraphSpectral method can produce good confidence estimations, but its calibration performance is poor. Even with the addition of techniques such as Isotonic Calibration or Platt Scaling, this issue can only be partially mitigated. The auxiliary DeBERTa method combines the LLM outputs, Chain-of-TT Chain-of-Thougght (CoT) outputs, and verbalized confidence to finc-tune the DeBERTa model for predicting confidence. Our method captures the prediction confidence based on the graph structure of LLM's responses in semantic space and achieves better ECE results. The ECE is reduced from 0.07 to 0.03 and improves the AUROC from 0.72 to 0.86 compared with the bascline calibration methods. The experiment results on TruthfulQA, HotpotQA and CoQA for the Llama3 model are shown in Table 1. These results show a similar trend, with our model achieving superior performance in confidence calibration compared to the baseline methods.
7

---

Published in Transactions on Machine Learning Research (5/2025)
Table 1: Comparison of confidence calibration performance on TriviaQA, CoQA, TruthfulQA and HotpotQA dataset for Llama3
T
CoQA
TruthfaQQA
HotposQA
Method
Brier]
AUROC!
ECR,
AUROC1
Brier]
ATTROC
FCF.)
Reiee]
AUROCI
CrppSpprttal (CS)
(M12
1002
.10 000
.732
110.
1022
1977 1012
+
.172 0017
253
GS + ho
167 +: M01
1822 6. (0022
(008 /002
302 X0M
1792 6. 1038
024 #: 1822
190 + MES
1977 (012
1986 1837
1303 1012
215 4: 000
1187 :: 1121
Platt
.00
361
042.4
015
.977
.013
11111
3300 .014
SellehrekGP7
312
182 (05)
.167 (000
200 6000
/033 0027
128.4 010
362 M2S
5/6
3056 1830
(853 1014
6733
100
Siy Biallhod
5961 005
500 (00
200 .000
382 001
.57 .026
175 4. 000
155. 038
569
0015
052 1. 009
/13 .018
651 002
105. 011
+ 5012
(002 .002
.208 2000
050 0009
0037
1025
+:
1200 (0120
6031
1424
Vertadioal Qud
0002
.000
.302 .001
(681
037
022
.016
3358 .012
025
Vedtalioal R
201 4 001
.03 a: .008
.083 =: .000
423 3: IM220
1602 1. .027
216 4: 002
510 4. 033
.73 .025
.31 4: 1035
.310 .014
1072 000
2204 025
APRICOTT
145 + 1877
773 + (03
/014 /X06
113 600
225 +: 00:00
132 4: 0035
101 4: 003
1637 003
052 4: 011
.371 0018
.8023 .011
0814 009
A
182 à 012
.723 à (003
.270 0. .004
:271 .000
.731 4: /022
/07 4: 003
200 d: 103
JA37 .024
1130 d: K11
.350 .012
.823 d: .011
A1734: J022
FPRiCOT+Flatt
(75
1371
.011
Oun
.130 .000
.804 .002 .035
1121 800
.758
.005
.151
739 011
.087*:003
.000
OuntMiiti pronpts)
.141= + JM72
.803 = 0012
.000 = .008
.118 .000
.776 4. 0012
005 + 0077
177 + 0035
.736 1: 007
1030 + 003
142 .000
321 #: 002
.021 1: .006
Table 2: Comparison of confidence calibration performance on TriviaQA, CoQA, TruthfulQA and HotpotQA dataset for Vicuna
TrivisQA
CoQA
TrushfulQA
HolpoiQA
Mathod
Brier.
AUROCT
ECE
Brier]
AUROCT
ECE]
Brier]
AUROCT
ECEL
Brier]
AUROC1
ECEL
GraphSportaal (CS)
196 800
7002= DIG
112
277 004
100+
202+:011
28 306
647+
22
162=
072
4733 008
2022 100
GS Iso
1966 000
J522 MIG
246::002
03:4
29/4.088
0822 004
16km (088
613 0088
06 022
68 - Platt
177+ 000
702+ 005
067-000
228+ 002
606+0004
056+
307+ 007
647+ 000
183+-003
160 049
673+ 008
073-040 040
Self-deaPPP
.35.1 001
113
221.1:044
.68.1.009
1321.010
28111,002
5521 (08
308.1.07
285
66211 187
187
Sog liciioood
485+ 002
561+ 002
4700+020
900++012
652+0002
102+0060
345+
187+ 000
205+-012
403-220
630 070
223-930
Platt
342e 002
001
016
.00++0015
065-+0001
105++0008
288
687+
191+:000
212= 0100
(30
250
Verbaliad Qua
303+ 002
631+000 007
078+-014
455+ 097
406++000 000
.009+-001
471++003
487+ 063
.018+0005
220 140
662+
142-140
Verbolized %
.02 1.001
505
.9221.025
.3991.003
321...09
3801.002
5661.00
387..017
312__033
6831
003
APRICOT
1962 000
006
1078
.150:::004
21424:006
.0/0:::009
.1972..007
200L.002
118
1662=000
28225.022
JV 074
APRICOT+
187+ 000
.783+ 005
000+0004
100++005
742++006
054+-007
.197+-0077
700
087+0008
142- 001
778++ 022
073-073
APRICOT Platt
186+
.760=100G
1100+0005
27424:002
045
234+b006
20040002
085+t.00
360m=00
.783 0.22
042=042 042
Ouzs
.1002 000
.816:10022
.08:20004
.1811:001
.7511.000
.0322.004
2022.003
.77442.001
00000000
.102=0000
.7912.002
.022.0002
(Mulii poompts)
.1652.000
895
.0551.003
.168:1.001
0011
776331.00
03301.006
20021.044
.7811.01
0631.004
.1312.000
Furthermore, we also compare the confidence calibration performance for the Vieuna model on the TriviaQA, CoQA, TruthfulQA and HotpotQA datasets. The results are summarized in Table 2. Our model consistently improves the calibration error compared to the baseline methods. Both GraphSpectral and our method have a similar assumption that the consistency level between responses indicates the confidence levels of these responses. However, GraphSpectral uses simple graph statistics to measure the confidence level of responses and could not capture complex relationships between response patterns and confidence levels (e-g- patterns beyond clustcring structures). As a comparison, by framing the problem as a learning problem, our method has better opportunities to discover such relationships and provides a better calibration performance. Self- Check GPT uses its own evaluation on whether the context supports the answers and heavily relies the LLM model's capability to do self-reflection, which can also be hallucinated. Thus the generated confidence scores are not calibrated well with empirical accuracy.
We present the reliability diagrams for all methods on TriviaQA to better understand the model improvement. The reliability diagram is created by binning responses into 10 bins according to their confidence values. Then the frequency of correctness is also computed for each bin. With an ideal calibration, the confidence value in each bin should match the frequency of correctness. At the same time, good calibration model
8

---

Published in Transactions on Machine Learning Research (5/2025)
Actunny
Accnasy
Accuracy
Accurscy
66%
7
0.4
92
02-
0.4
62
044
0.2
04
L4
0.
10
00
0.4
044 Confidence
1.0
00
0.
0.4
06
00
66
2.
Confidence
Confidence
(a) Seq. Likelihood.
(b) Platt scaling
(c) GraphSpectral
(d) GS + Platt scaling
1.0
14
0.8
0.
06
0.6
0
Accuracy
0.4
10:3
0.24
0.6
622
0.6
04
08
1.0
Corfidence
Confidence
Confidence
Confidence
(e) Verbalized Qual.
(f) APRICOT
(g) PPRt
(h) Our method
Figure 2: Reliability diagrams for different methods using 10 bins each for Vicuna on TriviaQA The color, as well as the percentage number within each bar, indicates the proportion of total responses contained each bin. Larger values represented by colors closer to purple, and the height indicates the ratio of correct ones. We prefer wide spread of responses in different bins (strong ability differentiate responses) and bin heights along the diagonal line (accurate calibration). Our model outperforms others with a broader bin spread and better alignment with the diagonal for calibration accuracy.
should differentiate responses according to their confidence levels and give a wide distribution of confidence values in these bins.
The reliability diagram is shown in Fig. 2. (We also show other reliability diagrams for the different hods for Llamas on TriviaQA and CoQA in the Appendix E). The figure presents the reliability diagrams for different methods. In these diagrams, both the color intensity and the percentage nunbers within each bar represent the proportion of total responses that fall into each respective bin. Specifically, larger propor- tions are depicted with colors closer to purple, while the height of cach bar indicates the ratio of correct predictions within that bin. Our framework achieves a broad spread of responses across the bins, showing good differentiation capabilities; at the same time, the bar heights closely follow the diagonal line, indicat- ing a good calibration performance. However, baseline methods cannot reach the same performance, The likelihood-bace confidence methods exhibit significant overconfidence, indicating many responses rated with high confidence are actually wrong. The Auxiliary DeBERTa (APRICOT) method, which integrates LLM outputs, Thaonghf-Thouuht (CoT) outputs, and verbalized confidence to train an auxiliary DeBERIa, model, enhances the AUROC. However, it still experiences some overconfidence issues, potentially caused by the inherent overconfidence in the input verbalized confidence scores. Furthermore, the baseline methods' relia- bility diagrams revealed that this method frequently assigned high confidence scores to incorrect predictions, deviating markedly from the ideal calibration represented by the diagonal line. For example, the verbalized method's predictions in the highest confidence bins (80-90%) were significantly below the corresponding empirical accuracy, indicating a tendency to overestimate the certainty of its outputs.
4.3 Confidence calibration in the OOD setting
Domain shift poses significant challenges for deploying machine learning models real-world scenarios where data variability expected. To comprehensively assess the robustness and generalization capabilities of our proposed model compared to baseline methods, we conducted series of out-of-domain (OOD) evaluations.
9

---

Published in Transactions on Machine Learning Research (5/2025)
Table 3: Evaluation the OOD setting. Models are trained on the TriviaQA from Llama3 responses and
tested on out-of-domain datasets
Datasct
Method
Brier
AUROC
ECE
GraphSpectral platt)
0.17
0.72
0.095
Llama3 CoQA
Apricot
0.24
0.59
0,154
Ours
0.13
0.77
0.077
GraphSpectral platt)
0.32
0.63
0.324
Llama3 TruthfulQA
Apricot
0.25
0.54
0.197
Ours
0.23
0.66
0.16
CraphSpoctral platt)
0.24
0.53
0.07
Vicuna Trivia
Apricot
0.19
0.76
0.13
Ours
0.17
0.81
0.07
GraphSpectral(w platt)
0.35
0.55
0.26
Vieuna CoQA
Apricot
0.24
0.59
0.08
Ours
0.22
0.73
0.10
Experiment sctup: We evaluate the confidence calibration of different approaches under out-of-domain setlings. We have two experiment configurations: out-of-domain dataset OODD, and out-of-domain LLMs (OODL). For OODD, we train the confidence calibration model on TriviaQA from Llama3 responses and test it on CoQA Llama3 and TruthfulQA Llama3 answers. For OODL, we use the same training data from Llama3 but test the Vicuna model's responses on the TriviaQA and CoQA datasets. We compare our model with the Apricot and GraphSpectral (with Platt scaling) methods.
Results and Analysis: Table. 3 shows the OOD performance of the bascline methods. The OOD ex- periment results revealed that our model maintained high level of performance across tested domains. Specifically, the model demonstrated consistent. calibration, evidenced by low ECE values and strong discriminative ability, reflected in high AUROC scores on in-domain and OOD datasets. For example, while the model achieved an ECE of 0.03 and an AUROC of 0.86 on TriviaQA (in-domain), it maintained an ECE of 0.077 and an AUROC of 0.77 on CoQA. Furthermore, the Brier scores across domains remained within acceptable ranges, demonstrating reliable probabilistic predictions even when faced with unfamiliar data distributions. The relatively small increase in ECE and slight decrease in AUROC for OOD datasets suggest that while there is some degradation in performance, the model retains substantial robustness and accuracy. This is primarily because similarity graph patterns are highly invariant to the data distribution. Specifically, our model employs the consistency graph and the clustering feature that does not alter with data distribution shifts, enabling it to maintain stable performance across different datasets.
In contrast, Apricot typically relies on specific dataset features, which leads to poor performance in OOD scenarios. Furthermore, calibration methods like the Platt scaling can improve the confidence calibration in-domain, but their calibration elfect.iveness remains limited under domain shift scenarios. This is because this calibration technique mainly adjusts the output probabilities but does not fundamentally address the biases introduced by feature representation changes across distributions.
4.4 Checking the quality automatic labels
In our previous experiment, we used an automated method to assign the correctness labels to responses by comparing them to true answers with ROUGE-L scores. One question is whether these labels are reliable. manually annotate the labels of a small set of LLM-generated responses and examine
In this experiment, the accuracy of labels generated from ROUGE-L scores. We also use these manual labels to evaluate top- performing algorithms.
To label the data manually, a human labeler checks the true answer of a question and labels the correctness of a response generated by the LLM. Then, automatic labels are compared to manual labels to get the accuracy value. Limited by our resources, we label 600 questions for each of the three datasets, TriviaQA, CoQA, and Hotpot Table 4 provides breakdown of the accuracy across different datasets. The results
10

---

Published in Transactions on Machine Learning Research (5/2025)
Table 4: The accuracy of labels generated from ROUGE-L scores. The high accuracy indicates the reliability
of automatic labels, as well our evaluation above.
TriviaQA
CoQA
HotpotQA
Accuracy of automatic labels
0.96
0.92
0.89
Table 5: Evaluation with manual labels (Llama3)
Table 6: Evaluation with manual labels (Vicuna)
Dataset
Method
AUROC
ECE
Brier
Dataset
Method
AUROC
ECE
Brier
+latl
0.79
0.050
0.15
GSaPlat
0.78
0.053
0.18
TriviaQA
Ppricot+Plaatt
0.77
0.059
0.13
TriviaQA
+pricot+
0.77
0.052
0.17
Ours
0.81
0.037
0.12
Ours
0.80
0.032
0.
GS | Platt
0.74
0.056
0.12
GS Platt
0.69
0.049
0.19
CoQA
Apricot+Platt
0.72
0.062
0.11
CoQA
Platc
0.72
0.057
0.15
Ours
0.76
0.023
0.09
Ours
0.77
0.032
0.12
Platt
0.76
0.088
0.19
GS+
0.68
0.065
0.
HotpotQA
Plaicot+Platt
0.78
0.052
0.18
HotpotQA
Apricot+
0.77
0.076
0.14
Ours
0.81
0.030
0.15
Ours
0.78
0.041
0.12
indicate that labels from ROUGE-L scores fairly accurate: the lowest accuracy is 0.89. The results are also consistent with previous studies (Kuhn et al., 2022).
We further evaluated our model and two strong baselines (GraphSpectral+PPite and Apricot++Platt) with manual labels. As shown in Table 5 and Table 6, our model still outperforms the baselines, which is consistent with our previous evaluation results with automatic labels.
4.5 Sensitivity analysis
In this subsection, we conducted several sensitivity analyses of our model.
Number of training samples We conducted experiments examine the relationship between performance tested our model performance the Llama3 TriviaQA and the amount of training data. Specifically, dataset and varied the training size from 100 to 4000. The results are displayed in Table 7. We observed that the model's performance does not drop significantly with the reduced training data. These experimental results indicate that the model performs well with limited data availability, demonstrating its applicability in real-world scenarios where only smaller datasets are available. We also tested the baseline performance, the results are shown in Appendix E.
Hyperparameter sensitivity We conduct the sensitivity analysis of our model's calibration error perfor- mance concerning two key configurations: the number of sampled answers used to construct the graph and the number of Graph Convolutional Network (GCN) layers in the GNN model. The results are displayed in Fig. 3. The experiments are conducted using the Llama3 model on the TriviaQA dataset. For Fig. 3 (a) experiments, we varied the number of sampled answers from 10 to 50 while keeping other configurations and hyperparameters fixed, as described in the experimental setup. We observe that increasing the number of sampled answers slightly improves performance, which then stabilizes. In Fig. 3(b), the sensitivity to the number of GCN layers indicates that model remains stable with to 4 layers, with the best performance observed at 3 layers.
5 Conclusion and Future Work
In summary, in this work, we proposed an effective strategy of calibrating the confidence an LLM's responses by learning an auxiliary GNN model on the self-consistency pattern among responses to the same questions. Experiments demonstrate that the proposed approach improves confidence calibration significantly across several datasets compared to baseline methods. Our calibration model enhances the reliability of LLMs by evaluating response accuracy, enabling them to abstain from uncertain queries and empowering users to
11

---

Published in Transactions on Machine Learning Research (5/2025)
Table 7: Performance under varying
0.0250
0.020
Training Sample Sizes
0.0225
0.018
0.0200
0.0175
Tr. size
ECE
AUROC
Brier
0.0150
0.0125
100
0.082
0.812
0.201
0.0100
0.010
300
0.062
0.836
0.187
0.0075
0.008
500
0.059
0.842
0.181
0.0050
0.006
1000
0.047
0.853
0.177
3
20
50
Number of GCN Layers
Number of Sampled Answers
4000
0.029
0.860
0.14
Figure 3: Sensitivity analysis of our model
determine trust levels, thereby promoting responsible deployment in society. However, there are instances where an LLM might be highly confident an incorrect semantic response, resulting in a consistency graph similar that of correct answer. In such cases, our calibration model may not provide an accurate confidence estimation. Without straightforward solution to this problem, we leave the study of this issuc to the future.
Broader Impacts: Our work's advancement in calibrating the confidence levels of LLM responses carries important social implications: by accurately estimating the likelihood that a response is correct, an LLM can choose abstain from answering difficult questions or signal its level of confidence to help users make more informed decisions. We believe this work will enhance the trustworthiness of LLMss and contribute positively to their responsible use in society.
Acknowledgment
We thank all reviewers and the action editor for their insightful feedback. Li-Liping Liu's work was supported by NSF Award 2239869. We would like to thank Xu Han and Jacob Boerma for their help in labeling the data used in this work.
References
Sher Badshah and Hassan Sajjad. Refereecc-guided verdict: Llms-as-judges in automatic evaluation of free-form text. arXiv preprint 0rXiv:2408.000 092355, 2024.
Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. Mars: Meaning-aware response scoring for uncertainty estimation generative 1lms. arXiv preprint ar orXic:2402.11756 2024.
Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-T'ee Tien Lu, and Lifu Huang. Internalinspector : Robust confidence estimation in lims through internal states. Xixe preprint or orXiv:2406. 12053, 2024.
Fabian Biester, Mohamed Abdelaal, and Daniel Del Gaudio. LImelcan: Context-aware tabular data cleaning via llm-generated ofds. In European Conference Advances in Dalabases and Information Systems, pp. 68-78. Springer, 2024.
Glenn W. Brier. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78 (1):1, January 1950. doi:
Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness, 2023.
Lihu Chen, Alexandre Perez-Lebel, Fabian M Suchanck, and Gaäl Varoquaux. Reconfidencing llms from the grouping loss perspective. Xixio preprint 072.022002.04057, 2024.
12

---

Published in Transactions on Machine Learning Research (5/2025)
Zekun Deng, Hao Yang, and Jun Wang. Can ai write classical chinese poetry like humans? an empirical study inspired by turing test, 2024.
Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. How ready are pre-trained abstractive models and Ilms for legal case judgement summarization?, 2023.
Gianluca Detommaso, Martin Bertran, Riccardo Fogliato, and Aaron Roth. Multicalibration for confidence scoring in Ilms, 2024.
Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv preprint rrXii:2307.01373 2023.
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et al. Lm-polygraph: Uncer- tainty estimation for language models. arXiv preprinl arXiv:231 1. 1.07383, 2023.
Jiahui Geng, Fengru Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Tryna Gurevych. A survey of language model confidence estimation and calibration. arXiv preprint orXw::3311005295, 2023.
Carlos Gódez-Rodriguez and Paul Williams. A confederacy of models: a comprehensive evaluation of LLMs creative writing. In Houda Bouamor, Juan Pino, and Kalika Bali (cds.), Findings of the Association for Compulational Lingwislies: EMNLP 2023, pp. 14504-14528, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1 /22//22/v1222333 (o0....fniingeeeml...666 URL https: pettss//aclaate /al.aatolll org/2023. f indingg-emmlp.965 966.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0330 03654, 2020.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Yen Kan (eds.), Proccedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v11 /v/ P17-1147. URL https: btttps//accanttonn http://accannooogyoorrreeoeeeeee wwtp://-1g///////../
Maia Kotelanski, Robert Gallo, Ashwin Nayak, and Thomas Savage. Methods to estimate large language model confidence. arXiv preprint (XXx:2312.03733 2023.
Sanjay Krishnan and Eugene Wu. Alphaclean: Automatic generation of data cleaning pipelines. arXiv preprinl ar Xiv: 14..11666 11827, 2019.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncer- tainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2022.
Chin-Yor Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest com- mon subsequence and skip-bigram statistics. In Proceedings of the 42nd ansuual meeling of the association for computational linguistics (ACL-04), pp. 605-612. 2004.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human false- hoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anaual Meeling of the Association for Compulational Linguistics (Volume 1: Long Pupers), pp. 3214 3252, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022. acl-long.220 229. URL attps://aclanthhooggy org.lloooo.or///// 2tgg2022.2o acl acll-oon.22 229.
Stephanic Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncortainty in words. Transactions on Machine Learning Research, 2022b. ISSN 2835-8856. URL https://operrevier... forum?id=8ssk2u1
13

---

Published in Transactions on Machine Learning Research (5/2025)
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification
for black-box large language models. Transactions on Machine Learning Research, 2024. URL https:
Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. Uncertainty estimation and quantification for llms: A simple supervised approach. arXiv preprint orXiv:2404.15900 2024.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In Inter- national Conference on Learning Represenlations, 2021.
Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucination
detection for generative large language models. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023.
Meta. Meta llama 3. https://git git. cow/meta- aaaaeetttaaaaaaaaaaaaaaaaaaaaaaa 2024.
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' over- confidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857 872, 2022.
John C. Platt. Probabilistic outputs for support vector machines and comparisons regularizod likelihood methods. In Advances in Large Moryin Classifiers, pp. 61-74. MIT Press, 1999.
Potsaweel. Truthfil-ga-Ilm-judges
2024.
URL httpss//higggngffaeecoo// htt.:///.......//taaaatttttttaae
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, and Jie Zhou. WebCPM: In- teractive web search for Chinese long-form question answering. In Amna Rogers, Jordan Braderraber, and Naoaki Okazaki (eds.), , Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 8968 8988, Toronto, Canada, July 2023. Association for Com- putational Linguistics. doi: 10.18653/v1/2023.ed lonclllon.4999 acl-long.c 499.
URL https://cclantholl htwphhoooooyo.o//2222222 org/2023.
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answer- ing challenge. Transactions of the Association for Compulational Linguistics, 7:249 266, 2019. doi: 10.1162/tacl 00266. URL httpp://acclanthollog. 10011/1-111001111111..10011
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ith International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); pp. 3982-3992, 2019.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Takshminarayanan, and Peter J Liu. Out-of-distribution detection and seloctive gencration for conditional languago models. In The Eleventh International Conference on Learning Represenlations, 2022.
Paul Roit, Johan Ferret, Lior Shani, Roce Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Leonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos Garea, Piotr Stanczyk, Nino Vicillard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pictquin, and Idan Szpcktor. Factually consistent summarization via reinforcement learning with textual entailment feedback. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Popers), pp. 6252-6272, 6272, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/v1. /cl2223.acc-oomm / //aclanthology www.:/2..rg/..2..... acl- acl-long.... 344.
URL https:
14

---

Published in Transactions on Machine Learning Research (5/2025)
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Y. Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Flan-moe: Scaling instruction- finetuned language models with sparse mixture of experts. CoRR, abs/2335.117700 2023. doi: 10.48550/
ARXIV STTTV.330.11477 URL https: .opiiiooo wtg/10.44550//rrt....... 4455008rXiii 2305. 14705.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.1000 16789, 2023.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. Towards expert-level medical question answering with large language models, 2023.
Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. Evaluating the factual consistency of large language models through news summarization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 5220 5255, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1 /1/2023.2222222 http:///////2....nnnnn/////// findings-acl.322 ac1.322. URL https: http:::////laahhologg// 2rtpr/hoooogyyo//22223..3/ org/2023. .ores.finn findings-acl.33 a22.
Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan Nestor, Ali Soroush, Pierre Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin Rousseau, Chunhua Weng, and Yifan Peng. Evaluating large language models on medical evidence summarization. npj Digital Medicine, 6, 08 2023. doi: 10.1038/541746--023, 020-0099-77
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafacl Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023. URL bttps://opeereweee
aet/fforumiddrgfaaCrrmm
Dennis Ummer Jes Frellsen, and Christian Hardmeier. Exploring predictive uncertainty and calibration in NLP: A study on the impact of method & data scarcity. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2707- 2735, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1 www....nniing-cmml...9000 URL
www.e//..r........ee//.. fiw.inisiiiiiinnnneeeee emnlp. 198.
Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. Calibrating large language models using their generations only. arXiv preprint arx:::2 05979, 2024.
Jordy Van Landeghem, Matthew Blaschko, Bertrand Anckaert, and scalable predictive uncertainty in text classification. IEEF Access, ACCESS.2002..368877.
Moens. Benchmarking 2022. doi: 10.1109/
Artem Vazhentsev, Gleb Kuzmin, Akim Tsvigun, Alexander Panchenko, Maxim Panov, Mikhail Burtsev, and Artem Shelmanov. Hybrid uncertainty quantification for selective text classification in ambiguous tasks. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11659-11661 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: /0.18633/11/2023.acll 652. URL https://aclanthol//
org/ 222320222222 acl moll-ong.... 652.
Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, Yibin Liu, Jialong Wn, Shengwei Ding, Long Li, Zhiwei Huang, Xinle Deng, Teng Yu, Gangan Ma, Han Xiao, Zixin Chen, Danjun Xiang, Yunxia Wang, Yuanyuan Zhu, Yi Xiao, Jing Wang, Yiru Wang, Siran Ding, Jiayang Huang, Jiayi Xu, Yilihamu Tayier, Zhenyu Hu, Yuan Gao, Chengfeng Zheng, Yueshu Ye, Yihang Li, Lei Wan, Xinyue Jiang, Yujic Wang, Siyu Cheng, Zhule Song, Xiangru Tang, Xiaohna Xu, Ningyu Zhang, Huajun Chen, Yuchen Eleanor Jiang, and Wangchunshu Zhou. Weaver: Foundation models for creative writing, 2024.
15

---

Published in Transactions on Machine Learning Research (5/2025)
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2022. URL
Shunxin Xiao, Shiping Wang, Yuanfei Dai, and Wenzhong Guo. Graph neural networks in node classification: survey and evaluation. Machine Vision and Applications, 33(1):4, 2022.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian Hc, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL
Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking llss via uncertainty quantification. arXiv preprint orXiv:2401. 12794, 2024.
Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. Luq: Long-text uncertainty quantification for llms. Niv preprint arXiv:2408. 20279, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging llm-as-a-judge with mi-bench and chatbot arena. Aduances in Newral Information Processing Systems, 36, 2024.
Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. Why does chatgpt fall short in providing truthful answers?, 2023.
A Hyperparameters and Model configurations
Model hyper-parameters:
Our model used three GCN(or GLN) layers; typically, the embedding dimension was 256, 512, and 1024 for each layers. For the training process, we used the binary cross-entropy loss with a decaying learning rate that reduced the learning rate by 0.9 if the validation loss did not improve 10 epochs (with an initial learning rate of 10-4 4 and a minimum learning rate of 10-?). The optimizer was Adam with 1 0.9 and 83 = 0.98. The batch size was 16, 32. For the rephrased prompts, we set k = 3, 30, so for each rephrased question, sampled ten answers. While calculating the ECE, divide the confidence into B 10 bins.
Evaluation Sctup:
For each question, we evaluate the confidence prediction corresponding to the most likely answers from the LLM response. The setup is consistent with the baseline methods.
Graph construction:
For each question, we prompt the LLM to give 30 answers, and the temperature for LLM is set to be 0.6. For each answer, the SentenceBert model Reimers & Gurevych (2019) is used to get each answer's embedding. The cosine similarity between each answer's embedding is taken as the edge weight of the graph. We apply the K-Means clustering method to cluster similar semantic responses. The maximum cluster number is set. as 3.
B Computational cost
We performed all experiments on NVIDIA A100 GPUs with 80GB of memory. Generating 30 responses using the Llama3 and Vicuna models for 6000 questions from CoQA and TriviaQA data required up to hours, with an average of approximately 2 seconds per question. The CoQA dataset demanded more processing time due the longer contextual in information in the input. The time be shortened by parallel sampling.
16

---

Published in Transactions on Machine Learning Research (5/2025)
C
Additional Cases
To better understand our method intuitively, we have collected a few examples to show the difference between our algorithm and APRICOT.
To summarize our observation here:
1. Multiple responses to the samne question does reveal the LLM's confidence in its answers. The LLM's self-evaluation of confidence is often much higher than should be the LLM is overconfident about its responses. The teain-of-thoughit responses used by ApriCoT add some information to make each answer more complete and reasonable in the spirit of 1, but it mainly adds the information within one response, not as much information as the multiple responses used by ours.
Example 1:
Question: Who plays Captain Jack Sparrow's father Edward Teague in the Pirates of the Caribbean films?
True answer:: Keith Richards
LLM response: David Schofield
More responses from the LLM: Martin Klebba. Keith Richards, Geoffrey Rush, Martin Klebba, Keith Richards. Martin Klebba. David Schofield. (only list 7 responses here to save space)
GCC-estimated confidence: 0.23
CoT responsc: David Schofield,
Self-evaluation: 80
ApriCoT-estimmteed confidence: 0.79
Example 2:
Question: In which film will you find the Rodger Young?
True answer:: Starship Troopers
LLM response: The Bridge the River Kwai.
More responses from the LLM: The Greatest Story Ever Told. The Best Years of Our Lives. The Bridge the River Kwai. The Best Years of Our Lives (1946). 1949's Battleground. The Best Years of Our Lives.
GCC-ostimated confidence: 0.22
CoT response: All the President's Men.
Solf-ovaluation: 95
ApriCoT-estimated confidence: 0.81
Example 3:
Question: BS is the international car registration which country?
True answer:: Bahamas.
LLM response: Germany.
More responses from the LLM: Bahamas. Bahrain. Bangladesh. Bahamas. Belgium. Bahamas. Germany. Bhutan. Belgium.
GCC-estimated confidence: 0.34
CoT response: Belgium
Self-evaluation: 98
17

---

Published in Transactions on Machine Learning Research (5/2025)
ApriCoT-cstimated confidence: 0.61
D Additional Visualizations
Besides the cases we show in the previous section. Here, we present several case examples and visualize the response patterns. We performed dimension reduction of LLM's responses to different. questions and then plotted their embeddings to the 2-dimensional space. Fig shows the responses generated by Llama3 as an example. From the figure, we observe that answers with higher confidence levels tend to cluster closely together, indicating consistency and reliability in these responses. In contrast, answers with lower confidence levels exhibit greater diversity, reflecting broader range of possibilities. This behavior aligns well with our initial assumption, demonstrating that higher confidence responses are more consistent, while lower confidence responses capture a wider variety of potential answers.
visalization the responses distibutioon
15-
10-
Component:
03-
-13
Component
Figure 4: Visualization of the generated response patterns
E
Additional results
Additional reliability plots We showed all reliability diagrams for Llama3 for TriviaQA in Fig. 5 and CoQA dataset in Fig. 6. To summarize the trends, observe that Platt scaling narrows the range to the middle value. Verbalized uncertainty cannot generate wider range of confidence values. GraphSpecral with Platt tends to generale wider range of confidence values, but the bias can not be improved across all cases, resulting in the bar height not following the diagonal line closely. Our model can predict wider range of confidence values and achieve better calibration in all scttings, with the auxiliary consistency graph and clustering features contributing to improved calibration overall.
Additional baseline results In Table 8, we showed the performance of the baseline method under varying training sizes. As the number of training data decreases, the will drop from 0.096 to 0.165.
Table 8: Performance under varying Training Sample Sizes for the baseline methods( methods(Apricot)
# of Training Samples
ECE
AUROC
Brier
100
0.165
0.611
0.229
300
0.133
0.634
0.211
500
0.112
0.695
0.204
1000
0.105
0.722
0.192
4000
0.096
0.743
0.187
18

---

Published in Transactions on Machine Learning Research (5/2025)
0.-
Accurccy
Accuracy
0.4
06
Accuracy
6.
Sara
4.5
4.07
2
0.2
10.0
0.8
Confidence
(a) Seq. likelihood.
(b) Seq.
likelihood +
(c) GraphSpectral.
(d) GraphSpectral Platt.
Platt scaling.
Accuracy
0.44
Accuracy
Accuracy
0.4
7
171%
0.2
200
3.5 Confidence
Confiddrnc
Confidence
Confidence
(e) Verbalized Qaul
(f) Apricot.
(g) Apricot + Platt scal- ing.
(h) Ours
Figure Reliability diagrams for different methods using 10 bins each for Trivia from Llama3 model responses. The color and the percentage number within each bar indicate the ratio of responses contained in each bin. Larger values are represented by colors closer to purple.
F Prompting strategy
Here, we showed the prompts to generate the rephrasing questions.
Prompts for rephrasing questions
You are a helpful assistant. 1 have a question that I would like to see it rephrased in multiple ways. Please take the original question and generate several rephrased versions while maintaining the same meaning, and the question can only have direct answer. Here is the original question: .... Please provide four distinct rephrases of the question.
The prompts for labeling:
Prompts for labeling
You will be provided with question, reference answer, and student's answer. Please evaluate the student's answer based on the reference answer and provide your score for the student's answer in the format: "Score: Assign a score of 0) for incorrect and 1 for correct. For example, "Score: 0" or "Score: 1.. Do not include any additional information. Question: f...} Student answer: {...)
Reference answer: {...) Now, please enter your score. Score:
G Sensitivity and ablations
G.1 Sensitivity to accuracy metric
In this section, we evaluate the sensitivity of the threshold of accuracy metric of our models. From the results, it show that method is relative insensitive of the threshold.
19

---

Published in Transactions on Machine Learning Research (5/2025)
02-
338
3,2
33.33
Acoura
Accuracy
AcCirsty
Accuragy
02
Contidence
Confidence
Confidence
Confidence
(a) Seq Likelihood
(b) Platt scaling
(c) GraphSpectral
(d) GS + Platt scaling.
1.0
16
0.8
PEAN
0.5
Accuracy
0.5
Accumcy
Accuracy
100
5.3
0/4
6.5
66
10
63
62
34
0.6
0.
0.
0.2
64
0.6
0.3
10
3.0
02
64
9.5
3.5
1.0
Confidence
Confidence
Canfidince
Confidence
(e) Verbalized Qual
(f) Apricot
(8) Apricot Platt scal- ing.
(h) Ours
Figure 6: Reliability diagrams for different methods using 10 bins each for CoQA from Llama3 model responses. The color and the percentage number within each bar indicate the ratio of responses contained in each bin. Larger values are represented by colors closer to purple.
Table 9: The results of using different threshold
AUROC
ECE
Threshold
Apricot
GraphSpectral
Ours
Apricot
GraphSpectral
Ours
0.3
0.72
0.79
0.82
0.074
0.076
0.023
0.5
0.70
0.76
0.80
0.091
0.083
0.031
G.2
Use the sentence embedding feature the node feature
We also provide the comparison with using sentence embedding feature as the node feature. We tested this method on TriviaQA. We got Brier scores 0.21, AUROC 0.75, and ECE values 0.11. The results indicate that GNN with sentence embedding as the node feature can produce worse results than our proposed approach. We see clear overfitting issues when GNN uses semantic foatures: the validation quickly shoots up after the initial dip. We conclude that GNN using semantic features could not generalize test data.
G.3
Use the Rouge similarity as the weight of the similarity graph
We provide the results of using Rouge-L as the weight of the similarity graph as shown in Table 10. For the dataset with long-fore form answers (e.g., TruthfulQA), the performance much worse than using the clusterID feature, From the results, we conclude Rouge-L sensitive to the length of the responses.
Table 10: The reuslts of using Rouge similarity graph
ECE
AUROC
Brier
TriviaQA
0.022 + 0.006
0.836 + 0.000
0.122 + 0.000
CoQA
0.021 + 0.007
0.795 + 0.001
0.110 + 0.000
TruthfulQA
0.035 + 0.005
0.632 + 0.014
0,221 + 0.002
20


Reviews:
[{"review_id": "Cfb375ariY", "reviewer": "Reviewer_EEyf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* the paper is well written and organized.\n* the experiments show improvements in confidence calibration.\n\nWeaknesses:\n- statistical significance of improvements is unclear.\n\nQuestions:\n- which of the results in table 1 and table 2 are statistically significantly better than baselines?\n\n- the work is presented as providing an estimate of correctness.  However --- as acknowledged in the conclusion --- there are many reasons why the LLM might be confident but wrong, such as bad or wrong training data or a significant distribution shift in the model prompt.  It would be useful to find an alternative phrase early in the paper, perhaps something like a confidence estimate \"regarding the model alignment (with the training data)\" or \"reliability relative to the training data\".\n\n- a sensitivity analysis over rephrasings of \"the original question q in k different forms\" would be useful.  How strongly does the effectiveness of the method depend on the approach?\n\n- Lin et al. (2024)'s GraphSpectral approach, as described in the related work sounds similar to this proposal.  What are the graph statistics used by them and how do the learned GNN's responses improve on it?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for estimating the confidence of an LLM generated answer. The approach measures the pair-wise similarity among multiple LLM generated responses, and uses the resulting similarity graph as the input to a trained graph neural network to estimate probability of a correct response.  In experiments on multiple benchmarks, the proposed method shows improved results with regards to calibration accuracy, including in OOD settings. The paper also presents sensitivity analyses, exploring the impact of training data size and hyperparameter sensitivity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- add statistical significance to experimental results\n- add a sensitivity analysis over rephrasing methods", "broader_impact_concerns": "The paper might be improved with a brief broader impact statement describing the risks of overreliance on AI models. Confidence estimates can help people avoid overreliance, but only if they are correct, and that is worth noting in the context of a broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "k2gyguC2Px", "reviewer": "Reviewer_zZ4u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper attempts to address a very important problem of confidence calibration of LLMs", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Confidence estimation for large language models (LLMs) is crucial for ensuring their reliability, especially in real-world applications. Existing methods face challenges in detecting semantic-level mistakes and quantifying uncertainty. This work proposes a novel graph-based confidence calibration method that leverages consistency among an LLM’s multiple responses to a question. A similarity graph is constructed based on the consistency of LLM-generated responses. A Graph Neural Network (GNN) is trained to predict correctness probabilities without processing actual language content. The approach significantly outperforms existing methods across multiple benchmark datasets by using response consistency for confidence estimation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There is something off with equation 1.  For the same semantic response C, there can be numerous r when we consider the domain of natural language responses.  Consequently, p(C|q) = \\Sum_r∈C p(r|q) can lead the p(C|q)  to be greater than 1 if p(r | q) is estimated using something like softmax. I think there is some assumption here that the space of all possible responses is somehow known and can be enumerated. For natural language responses without some syntactic restriction (such as the count of tokens), one can't make such an assumption. \n\nUsing ROUGE similarity to compute semantic similarity is odd. It would be useful to consider some other alternatives such as embedding in eqn 4 and empirically evaluate what measure is a good choice. \n\nSimilarly, for equation 6 in method 2, why not also consider ROGUE metric when measuring similarity between r_i and r_j?\n\n\"We choose NOT to feed in the embedding vectors of responses to avoid the GNN’s dependency on textual information\" - once again, this choice must be studied through ablation. Just relying on how the responses are split across clusters using 1-hot embedding would remove significant semantic content and rely on the distribution of semantic dispersion of the output to help predict confidence. \n\n\" We employ a multiple rephrased questions strategy for answer sampling. Specifically, we prompt the GPT-4 to give k different but with the same meaning rephrased questions for the given question q. Then, we sample n/k responses from the LLM for each rephrased question and still get a total of n responses\" - how sensitive is the method to the use of different LLMs for this prompt rewriting? \n\nOverall, the method presented in the paper appears rather simplistic and an incremental extension of Lin et al 2024. Without sufficient empirical study, including the suggestions above, it is not clear if the claims in the paper are well-justified.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "y7WxWClUay", "reviewer": "Reviewer_ity6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The graph-based confidence calibration approach is a novel approach.\n2. The proposed method outperforms baseline confidence calibration approaches across multiple datasets (TriviaQA, CoQA, and TruthfulQA).\n3. The model holds good calibration performance even in OOD settings.\nWeaknesses:\n1. The method assumes that consistency among responses correlates with correctness. However, LLMs can be consistently wrong (systemic bias), especially on factual or ambiguous questions (e.g., TruthfulQA). The paper acknowledges this limitation but does not propose a concrete mitigation strategy.\n2. The datasets primarily focus on factual QA, but confidence calibration is also crucial in reasoning-intensive tasks (e.g., math problems, multi-hop reasoning).\n3. Using ROUGE similarity to assign correctness labels is problematic, as it may not capture deep semantic correctness. You apply GPT-4 to generate labels on TruthfulQA datasets. Why didn't you use it for other datasets?\n4. Constructing response graphs for every question requires 30 LLM queries, significantly increasing computational costs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a Graph Neural Network (GNN)-based model to estimate response correctness. The key idea is that responses with higher consistency across multiple LLM-generated answers are more likely to be correct. The paper introduces a graph-based method for confidence calibration using response consistency. The authors claim that their approach outperforms existing methods in confidence estimation and OOD generalization.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please count how often systemic bias occurs and discuss potential solutions in the paper.\n2. Please try to apply this method to complex reasoning tasks/datasets (e.g., MATH, StrategyQA, HotpotQA, etc.)\n3. Please try to analyze efficiency trade-offs and explore potential optimizations (e.g., reducing the number of sampled responses).\n4. In Tables 1 and 2, please keep the left aligned on the \"Method\" column to clarify the comparison.\n5. At the end of page 8, should \"Platting scaling\" be \"Platt scaling\"?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HFFMwSro6J", "reviewer": "Reviewer_Z1en", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe authors conduct comprehensive experiments on multiple datasets, including CoQA, TriviaQA, and TruthfulQA, demonstrating improvements over prior baselines in 3 calibration metrics: ECE, Brier Score, and AUROC. The paper also focuses on out-of-domain generalization, demonstrating greater generalizability of the method in the 3 calibration metrics.\n\n## Weaknesses\n\nThe method is too trivial. It simply applies a Graph Neural Network to a dataset that was processed using a non-novel technique. The overall approach lacks originality, as it only takes existing ideas of consistency-based confidence calibration and reformulates them with a GNN. The justification for using GNNs is weak, as the authors do not provide evidence that alternative methods such as clustering or weighted averaging would not work just as well or better. There is no discussion on why a graph representation is fundamentally necessary for this problem, and the claimed benefits are not strongly substantiated.\n\nThe correctness labels are post-hoc generated using ROUGE similarity and GPT-4, which introduces bias and allows the authors to tune the label generation process to artificially improve their method’s performance. There is no independent ground truth, making the evaluation unreliable. The choice of ROUGE as a similarity metric is also questionable because it operates at the word level rather than capturing true semantic similarity. The authors do not clarify why why it was chosen over more semantic-aware methods like SBERT similarity.\n\nWhile the paper compares its approach to some prior consistency-aware methods, such as GraphSpectral and Self-Check GPT, it does not provide a thorough discussion of their limitations. The authors claim that their approach improves upon these methods, but they do not analyze in depth why prior methods fall short in calibration quality or whether the improvements observed in their results are statistically significant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method for confidence calibration in large language models by constructing a similarity graph over multiple responses from an LLM to the same question and using a Graph Neural Network to predict correctness probabilities. The authors claim that their approach leverages response consistency to improve confidence estimation outperforms existing methods in calibration metrics such as ECE, Brier Score, and AUROC, and improves out-of-domain generalization compared to prior work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical Changes (Required for Acceptance Recommendation): \n\nClarify the issue of post-hoc correctness labeling. The authors should either use human-annotated correctness labels or show robustness by testing across multiple correctness thresholds. Justify the need for a graph-based approach by providing evidence that a GNN meaningfully improves over alternative consistency-based aggregation methods. Provide an empirical comparison to other confidence calibration methods, such as GraphSpectral, SelfCheckGPT, and APRICOT, and compare performance in both in-domain and out-of-domain settings\n\n## Suggested (Non-Critical) Improvements: \n\nExplain why ROUGE is used instead of SBERT similarity, and provide experimental evidence supporting this choice if it is indeed the best. Discuss broader implications of miscalibrated confidence in LLMs, including real-world risks and failures in domains such as medicine, law, and education. Include a discussion on why previous methods did not fully explore the potential of response consistency and explicitly state what problems their method overcomes that previous approaches failed to address.", "broader_impact_concerns": "NO", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a GNN based method for calibrating the confidence of LLMs on top of multiple resampled answers, with the goal of  improving the uncertainty estimation for the correctness of the selected answer. The work builds on previous self-consistency papers such as  (Chen & Mueller, 2023; Lin et al., 2024) and (Ulmer et al., 2024) and suggests to add a GNN for improving the confidence prediction. Overall, this is a focused study that introduces a new method and empirically evaluates it across in-domain and out-of-domain datasets with two LLMs. It will be interesting to expand this direction further with long output answers and more capable LLMs.

The final paper should include all the additional results and clarifications discussed from the discussions below.
Decision: Accept (poster)