Graph-based Confidence Calibration for Large Language Models

Authors: Yukun Li, Sijia Wang, Lifu Huang, Liping Liu
Primary area: foundation or frontier models, including LLMs
Keywords: Language Models; Uncertainty Calibration

Abstract:
One important approach to improving the reliability of large language models (LLMs) is to provide accurate confidence estimations regarding the correctness of their answers. However, developing a well-calibrated confidence estimation model is challenging, as mistakes made by LLMs can be difficult to detect. We propose a novel method combining the LLM's self-consistency with labeled data and training an auxiliary model to estimate the correctness of its responses to questions. This auxiliary model predicts the correctness of responses based solely on their consistent information. To set up the learning problem, we use a weighted graph to represent the consistency among the LLM's multiple responses to a question. Correctness labels are assigned to these responses based on their similarity to the correct answer. We then train a graph neural network to estimate the probability of correct responses. Experiments demonstrate that the proposed approach substantially outperforms several of the most recent methods in confidence calibration across multiple widely adopted benchmark datasets. Furthermore, the proposed approach significantly improves the generalization capability of confidence calibration on out-of-domain (OOD) data.

Decision: reject
Reviewer scores: 6, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730467178559

thread_type:
official_review

<thread_object>
creator: reviewer_rphj

<content>
summary:
This work proposes a GNN-based approach to calibrate confidence scores of LLM responses. The proposed method is an extension of the Semantic Uncertainty,  a sampling-based method that estimates uncertainty by using the semantic distribution.
The main distinction of the proposed method is that it treats each response as a node and applies GNNs to obtain the logit. Next, cross-entropy loss is introduced to calibrate the node score. Experimental results on short-text datasets CoQA and TriviaQA show that the new method can outperform other baselines and lead to a better generality in OOD settings.

strengths:
1. the paper is easy to follow 
2. This work develops a graph-based method to determine the semantic uncertainty, which is interesting
3. better performance in OOD settings

weaknesses:
* **Main concern**:  the scientific contribution of this work is limited. The proposed method is an extended work of the semantic uncertainty with GNNs
* writing and presentation should be improved. Some claims appear too arbitrarily without any proof. Figure 1 is not very informative. (see questions below)
* the proposed method requires labels, but other baselines such as self-checkgpt and semantic uncertainty require no further training
* the use of the ROUGE to evaluate semantic equivalence is problematic since it does not capture semantic meaning well; in the experimental part, only short-answer datasets are used. It is important to include long-form generation dataset

questions:
1. Figure 1 is not very informative, it does not provide the intuition why the proposed method chose to use GNNs
2. In line 163-164, it is supposed to provide proof here: theoretical studies, empirical studies, or examples 
3. the use of the ROUGE metric for judging the correctness is problematic. ROUGE does not account for semantic meanings and is sensitive to length and word orders. Therefore, it is not an appropriate metric to evaluate whether two responses are semantically equivalent 
4. CoQA and TriviaQA are short-answer datasets. It is useful to test on datasets with long-form answers such as TruthfulQA, which can show the generality of the proposed method
5. in line 63-64, " A response consistent with more answers tends to have a higher likelihood of being correct". A high consistency is an indicator of high confidence rather than high correctness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730583778579

thread_type:
official_review

<thread_object>
creator: reviewer_qgfu

<content>
summary:
The paper presents a graph-based method for confidence calibration for LLMs, aiming to improve reliability by estimating response correctness through consistency. A graph neural network (GNN) is trained on a similarity graph of multiple LLM responses to a question, predicting correctness based on response consistency without processing language directly. Experimental results show substantial improvements over baseline methods in calibration performance and out-of-domain generalization.

strengths:
1.	The topic of the paper is important. A well calibrated confidence of LLM outputs can benefit a lot of domains.
2.	The model exhibits strong generalizability across different datasets and LLMs, showing robustness against domain shifts, which is valuable for real-world applications.
3.	The paper includes sensitivity analyses and comparisons across various configurations, demonstrating the stability and effectiveness of the method under different setups and highlighting the performance benefits over established baselines.

weaknesses:
1.	The requirement for sampling multiple responses and constructing similarity graphs for each query can introduce substantial computational costs, limiting scalability for real-time or resource-constrained applications.
2.	The motivation is not clear. First, how to choose the value of \tau in Eqn. is an extra challenge. Second, after having the correctness label, why do we need to leverage a GNN for classification? I think it is a typical text classification task. The motivation of choosing GNN is not clear. 
3.	The reliance on Sentence-BERT embeddings for similarity computations could introduce biases and inconsistencies if embeddings do not fully capture semantic equivalence, potentially affecting calibration accuracy.

questions:
refer to the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688636263

thread_type:
official_review

<thread_object>
creator: reviewer_e4j6

<content>
summary:
This paper considers the problem of producing meaningful uncertainty estimates given samples from a large language model. The approach relies on repeated sampling the LLM and forming semantically equivalent clusters of responses. These clusters are converted to a fully-connected similarity graph based on SBERT similarity between the responses contained in those clusters, which is the input to a supervised post-hoc calibration graph neural network (GNN). The node classification problem is to predict for each response if it is correct. The GNN does not take as input the SBERT embedding vectors to avoid dependency on textual information, but uses the cluster IDs and the similarity between each response. The clustering is obtained using the k-means algorithm. The appendix says that 30 responses are sampled and that K is set to 3 for k-means. The robustness of the approach is further (marginally) improved by forming multiple prompts for each question to increase the variety of response; further variety is associated with greater uncertainty. The proposed approach is compared to a diverse set of baseline methods on two standard QA tasks and is found to perform favourably in terms of calibration performance. The approach performs well with only small amounts of training samples.

strengths:
* The paper considers an important problem (LLM confidence estimation).
* A diverse set of baseline methods are included (although I would have liked more information about how they were tuned compared to the baseline).
* The approach is applicable to “black box” models since it only relies on being able to sample from the model, rather than requiring access to the activations or full predictive distribution.
* The idea of using a GNN to calibrate a set of responses given a measure of semantic similarity between them is appealing.

weaknesses:
* The use of a fixed number of samples and K for K-means seems like it could introduce some issues. If there’s no/little ambiguity won’t this still return three different clusters? At least some more sensitivity analysis should be done for K but this seems like a fundamental issue with the approach since choosing the number of clusters is underspecified in general.
* The quality of the approach seems like it depends strongly on the semantic similarity used. SBERT is quite an old approach and it’s unclear how well it generalises to various settings. I would have liked to see more analysis of the choice of semantic similarity. 
* The evaluation is conducted on two public QA benchmark datasets from 2017 and 2019, which are almost certainly in the training dataset for many LLMs. How does this impact performance?
* There is no meaningful discussion of the limitations.
* Regarding the writing, it should be made more apparent what data is treated as "calibration data" vs. "validation data" vs. "test data"
* Overall, my biggest concern with the paper that the evaluation is limited to two fairly "easy" QA datasets. This makes it difficult to gauge how general the proposed approach is.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730816630569

thread_type:
official_review

<thread_object>
creator: reviewer_fhwk

<content>
summary:
To enhance the calibration performance of LLMs, this paper proposes to combine the LLM’s self-consistency with labeled data and train an auxiliary model to estimate the correctness of its responses to questions. Experiments demonstrate that the proposed method outperforms baselines in confidence calibration on two datasets.

strengths:
1.	The paper presents a novel method to confidence calibration in large language models by leveraging graph neural networks and the consistency among multiple model responses. 

2. The manuscript is well-organized and the experiments are comprehensive.

weaknesses:
1. The premise proposed in this paper is if LLMs give similar response. Then there is less uncertainty, and these responses tend to have a high probability of being correct.  It is common to use the idea of self-consistency to guide the selection of the most appropriate answer from multiple answers, but it may not be reasonable to use it as a probability to evaluate the correctness of an answer. For example, the LLM may exhibits generate the same wrong answer when sampling multiple answers. In this case, is the proposed method still effective? 

2. This paper trains another model to calibrate the correctness of the answers generated by LLM. What is the difference between this auxiliary model and the "reward model" in the LLM evaluation work, which is used to evaluate the quality of responses? If there is a difference, the author should point out the difference with this type of work; if it is similar, the author should add a description of these works and performance comparison in related work and experiments.

3. This work uses semantic similarity to cluster different answers. Even so, answers within the same cluster may still be completely different answers. Because they just exhibit the similar reasoning paths but gives the different final answers. This phenomenon is also mentioned in paper [1]. But according to Equation 3, the author calculates the correctness probability of each cluster using the ratio of numbers. Why? How to explain the reasonability?
[1] SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales

4. In Method Section, the details how the training dataset was created are unclear. For example, for the same question, will the confidence information of each cluster be input into the GNN? It is recommended that the author add a paragraph to describe the complete process of constructing GNN training data.

5. This paper constructs training data based on the results of self-consistency. The author should add a comparison with the performance of the simplest self-consistency method to estimate confidence in the baselines.

6. Section 4.3 lacks analysis of DDL results.

questions:
1.	Are the confidence values of answers in the same cluster the same?

2.	When verifying the ability of the proposed method on OOD in Section 4.3, I recommend that the author add other datasets of tasks that are quite different from the training data to verify the generalization ability, such as Math
</content>
</thread_object>

</thread>
<thread>
creation date:
1733291921218

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers and ACs,  
We would like to thank all the insightful comments and valuable suggestions provided by all reviewers. In the following, we would like to summarize the contributions and revisions of this paper again.

**Contributions**:

**[High-level insight]** This paper proposes a graph-based confidence calibration method for LLMs that uses a similarity graph to encode relationships among multiple responses to the same question. By employing a GNN to calibrate response correctness, the method significantly improves confidence calibration performance and generalization to out-of-domain data compared to baseline approaches. This method is novel and appealing (*R fhWK*, *R e4j6*, and *R RPHJ*), has strong performance and generalizability (*R fhWK*, *R e4j6*, *R RPHJ*, and *R qGFu*), and is applicable to “black box” models (*R e4j6*).

**Responses and Revisions**

- **[For reviewer fhWK]**  
  - **Clarifications:** We have provided a detailed analysis in the response, and it resulted in a positive adjustment in scoring.  
  - **Extended experimental analysis:** Additional experiments have been conducted on more challenging TruthfulQA datasets and demonstrate the generalizability of our method to diverse datasets.    
- **[For reviewer e4j6]**  
  - **Clarification on hyperparameters:** We have added more details on the choices of hyperparameters.   
  - **Additional experiments:** Additional experiments have been conducted on the more challenging TruthfulQA dataset and demonstrated better performance.   
  - We hope our further experiments and clarification have been helpful.  
- **[For reviewer qGFu]**
  - **Regarding the Computation:** We conducted a sensitivity analysis on the impact of the number of sampled answers, and the results demonstrate that the approach maintains strong performance even with fewer sampled answers. Additionally, computational efficiency can be improved through parallel sampling.
  - **Motivation for choosing GNN:** We have provided a detailed analysis of why we chose the GNN in the responses. Our method utilizes the topology of the similarity graph and does not rely on the semantic meaning. Furthermore, by utilizing the GNN’s message-passing mechanism, it can gradually integrate information across the entire graph, resulting in improved calibration.  
We hope our further clarification has been helpful.
- **[For reviewer RPHJ]**
  - **Additional experiments:** We conducted additional experiments on the more challenging TruthfulQA dataset, which demonstrated improved performance and led to an increase in scoring.
  - **Figure Clarity:** We have revised Figure 1 to provide the intuition as to why the proposed method chose to use GNNs.

Thanks again for your efforts in reviewing our work, and we hope our responses can address any concerns about this work.
Thanks, Authors
</content>
</thread_object>

</thread>


Metareview: