Calibrated Decision-Making through Large Language Model-Assisted Retrieval

Authors: Chaeyun Jang, Hyungi Lee, Seanie Lee, Juho Lee
Primary area: foundation or frontier models, including LLMs
Keywords: calibration, RAG, LLMs

Abstract:
Recently, large language models (LLMs) have been increasingly used to support various decision-making tasks, assisting humans in making informed decisions. However, when LLMs confidently provide incorrect information, it can lead humans to make suboptimal decisions. To prevent LLMs from generating incorrect information on topics they are unsure of and to improve the accuracy of generated content, prior works have proposed Retrieval Augmented Generation (RAG), where external documents are referenced to generate responses. However, traditional RAG methods focus only on retrieving documents most relevant to the input query, without specifically aiming to ensure that the human user's decisions are well-calibrated. To address this limitation, we propose a novel retrieval method called Calibrated Retrieval-Augmented Generation (CalibRAG), which ensures that decisions informed by the retrieved documents are well-calibrated. Then we empirically validate that CalibRAG improves calibration performance as well as accuracy, compared to other baselines across various datasets.

Decision: reject
Reviewer scores: 6, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730061786146

thread_type:
official_review

<thread_object>
creator: reviewer_46au

<content>
summary:
The paper proposes CalibRAG, which extends Band et al. 2024 to the RAG use case, where LLM generates calibrated generations for better decision making. The main component is to train a forecasting function f that provides a confidence estimation of the query and a retrieved document.  Experiments are conducted on 4 existing QA datsets with baselines that were not specifically designed for RAG or reranking methods without considering calibration.

strengths:
1. The topic of confidence estimation in RAG systems (for decision making) is an interesting extension of existing work. 

2. In experiments, baselines include both calibration methods and reranking methods, which are two good perspectives to study.

weaknesses:
1. The reviewer feels the writing of the paper is not clear.  The paper does not clearly describe the use case (LLM-assisted decision making). More specifically, almost half of abstract and intro are about very basic concepts about LLM and RAG. The discussion of calibration is also very general and does not talk about anything specific in this work. By the end of abstract and intro, the reviewer is not clear what the paper wants to do. Section 2.1 then immediately starts with Band et al. 2024 and the notations become confusing since the problem setup is unclear. The authors seem to assume the readers know Band et al. 2024 very well. In fact, the reviewer has to read that paper to understand what the problem setup is. The authors are suggested to refer to the paper on how to set up the problem.

2. The novelty and technical contribution of this work looks borderline. More specifically, extending to RAG is meaningful, as indicated above. The work heavily builds upon Band et al. 2024. The arguments of the weaknesses of Band et al. 2024 look reluctant expect for the RAG part. To the reviewer, the real major contribution of the work is to the extension to RAG use case, which is not very technically deep (but again, still meaningful). 

3. Many technical details seem to be missing. For example, there's pretty much no details about the cross-encoder baseline. The LLM-rerank baseline is also a very sensitive method. Without more details. the experimental results are not very trustworthy to the reviewer.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730470676387

thread_type:
official_review

<thread_object>
creator: reviewer_fxeq

<content>
summary:
This work focus the problem that the confidently incorrect information provided by LLMs may mislead human decision-making process. The target is to calibrate confidence of LLM responses with the correctness of decision-making based on these responses. The proposed CalibRAG framework, which contains synthetic supervision data generation and training, show notable improvement in confidence calibration and reranking tasks.

strengths:
- It is a important research topic to calibrate LLM confidence in a retrieval-augmented generation (RAG) system. Confidence calibration of RAG is highly relevant for building trustworthy LLM systems, as in practical applications, LLMs are typically used with input knowledge.
- The proposed method demonstrates significant improvements in the uncertainty calibration task. The experimental setup includes diverse datasets, multiple baselines, and several standard evaluation metrics.

weaknesses:
- The paper appears to equate "the confidence of LLM response" with "the relevance between the query and the document."
	- During training (Section 3.2), the forecasting function relies solely on information from the query and the document, excluding the actual LLM-generated guidance ($z$). Since the LLM-generated guidance may omit or alter important information from the document, using only the query and document information to substitute for the LLM-generated guidance is unreasonable. Therefore, the proposed method seems more akin to training a specialized rerank model rather than calibrating the confidence of the LLM response.
- The reranking experiments are not entirely fair due to the use of additional fine-tuning data compared to baseline methods. Moreover, the synthetic supervision data uses the correctness of user decisions (surrogate model in this paper) as the supervision signal, which results in a smaller gap and aligns more closely with the testing procedure compared to baseline methods.
- The results in Section 4.3 do not convincingly that the LLM can effectively simulate human decision-making. In the 0-20 and 40-60 bins, the agreement rate is only approximately 60%, indicating that a substantial portion of the data cannot be simulated by the LLM. Considering that in Table 2, the proposed method improves accuracy by about 2% compared to LLM-rerank on the BioASQ dataset, the approximately 30% disagreement between the LLM and human decision-making is significant.
- Figure 3 should be a vector image (import as pdf/svg) to ensure high quality.

questions:
* Are there any fundamental differences or novel issues in confidence calibration for Retrieval-Augmented Generation (RAG) compared to calibration in generation models without retrieval augmentation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730728324690

thread_type:
official_review

<thread_object>
creator: reviewer_wbip

<content>
summary:
This paper proposes the Calibrated Retrieval-Augmented Generation (CalibRAG) framework. CalibRAG allows an LLM using RAG to not only select relevant information to support user decision-making but also provide confidence levels associated with that information by utilizing a forecasting function, ensuring well-calibrated decisions based on the retrieved documents.

strengths:
1. Proposes a new dataset by creating labels that indicate whether decisions made using retrieved documents correctly answer the questions, essential for training the forecasting function.

2. The proposed method outperforms existing uncertainty calibration baselines across various tasks involving RAG context using the Llama-3.1 model in decision-making scenarios.

weaknesses:
1. Many important baselines in RAG that have the same motivation with this paper are missing in experiments. This paper is actually aims to improve the robustness of RAG, which can select useful documents and prevent interference from noisy documents. So many important baselines such as RetRobust [1], Self-RAG [2], CRAG [3] should be discussed and compared but this paper overlooks them.

2. Technical contributions are limited. This paper uses a surrogate model that predicts the probability of whether the user’s decision based on the guidance provided by RAG will be correct. Many existing methods are similar to this such as Self-RAG [2] and CRAG [3]. I cannot learn any insightful points from this paper that is distinguishable enough from existing methods.

3. The writing should be improved. The Introduction section has so many contents about the existing methods that have been well-known in RAG area while there is too little contents about the methods and contributions for this paper.

4. The term "Decision Making" is too broad to describe only the evidence selection task in open-domain question answering. It is typically used in a much wider context.

[1] Making Retrieval-Augmented Language Models Robust to Irrelevant Context

[2] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

[3] Corrective Retrieval Augmented Generation

questions:
Please refer to the details in the Weakness section:

1. Human decision-making is far more complex than described in this paper. How can we assume that if LLMs can handle document selection, they are also capable of decision-making?

2. How do other related baselines perform?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731170375109

thread_type:
official_review

<thread_object>
creator: reviewer_tasw

<content>
summary:
In this paper, the authors build upon Band et al.'s (2024) work on linguistic calibration for long-form generation by designing an LLM-based Calibrated Retrieval-Augmented Generation framework. This framework ensures that decisions informed by the retrieved documents are well-calibrated. The authors validated the effectiveness of their approach in four QA datasets.

strengths:
1. The model design in the paper appears well-constructed and reasonable.
2. Introducing a confidence score in LLM responses can effectively support human decision-making.

weaknesses:
1. Section 2 lacks coherent organization, which limits its ability to effectively link the relevant background to the authors’ contributions. For example, the three issues identified at the end of Section 2.1 are not clearly addressed in the proposed approach. It would be helpful to clarify how this approach resolves each of these problems. Additionally, when formalizing the problem in Section 2.1, the authors should explicitly outline its relationship to previous work. Without this context, researchers in automated QA may struggle to understand the distinction between the original question and the open-ended query, as well as the necessity of introducing the open-ended query. Additionally, Section 2.2 should provide a clearer explanation of the calibration error, as it is challenging to intuitively grasp how performance differences impact decision-making outcomes when interpreting Figure 1. Moreover, while Figure 1a illustrates the effectiveness of calibRAG, Figure 1b does not, and despite the presentation of calibRAG’s results in Figure 1a, there is no corresponding description in the text.
2. The authors should expand the discussion on open-ended queries. Specifically, is it necessary for users to create these queries, or could they be generated by the LLM itself? What impact would LLM-generated queries have on the outcomes? Additionally, examining how variations in open-ended queries influence model performance would add valuable insights.
3. The authors incorporate confidence scores in long-form generation to enhance human decision-making. However, there is insufficient empirical analysis to demonstrate this benefit—specifically, a comparison of human judgment accuracy with and without confidence scores is missing. Automated evaluation alone is inadequate to validate this claim. Moreover, while the authors prompt the LLM to express uncertainty using either an integer scale from 0 to 10 or linguistic terms of certainty, the impact of these differing methods on model performance remains unclear. Further analysis is needed to clarify how these approaches influence outcomes.
4. The paper lacks comprehensive experimental validation. Evaluating the model on the complete BEIR benchmark would provide a more thorough assessment. Additionally, how does the model perform in non-zero-shot scenarios? For instance, is the hyperparameter setting—particularly the number of retrieved passages (K)—consistent across datasets, or does it vary? Stage 3 in Section 3.4 requires a more detailed description, specifically regarding how the query is reformulated in the experiments. Furthermore, how is the predefined threshold ϵ selected, and what effect does its value have on performance? The dataset verified by human annotators is too limited to robustly support the conclusions drawn from the experimental analysis. It would also be beneficial for the authors to showcase the performance of different models on human-annotated data.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732106358464

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1)

comment:
We thank all reviewers for their valuable and constructive comments. **We are eager to engage in the review process and address the suggested areas for improvement to further strengthen the quality of this work.**

--- 

Before reviewing your respective rebuttals, we kindly ask all reviewers to read the following:

The ultimate purpose of calibrating LLMs is to **minimize the risks associated with user decisions** that rely on LLM-generated outputs. In highly sensitive domains such as medicine, confidently providing incorrect information can have severe results. Thus, the practical goal of calibration is to **ensure the model does not express high confidence in uncertain information**, enabling users to reconsider or postpone their decisions. This aligns with the concept of "decision calibration" as defined by Band et al.

However, recent open-source LLMs that incorporate retrieval-based methods have shown increased calibration error. As illustrated in Table 1-(b), existing calibration methods not designed for retrieval contexts lead to higher calibration error, even when retrieval contexts improve overall performance. Moreover, as shown in Table 1-(a), current RAG approaches fail to retrieve documents that help decision-making. 

To address these challenges, we propose a novel calibration method tailored to RAG scenarios, with two primary objectives:  

1. **Identify and retrieve documents that support decision-making**  

2. **Provide calibrated confidence scores that reflect uncertainty appropriately**  

To achieve these goals, we designed a system where the LLM as an encoder, paired with a classifier head explicitly models **the accuracy of user decisions** based on the LLM-generated responses. This classifier is trained to maximize **proper scoring rules** (e.g., likelihood), ensuring theoretically sound calibration. Unlike Band et al., who neither addressed RAG scenarios nor tuned classifiers for likelihood-based probability calibration, our approach introduces a fundamentally distinct methodology. 

In addition, existing RAG variants (e.g., selfRAG, cRAG) primarily focus on **efficient operations** or providing **rich, contextually relevant information** for question answering, rather than directly addressing decision calibration. These approaches often define correctness based on utility scores (which may be unrelated to factual accuracy) or query-document relevance, relying on heuristic-driven and conceptually ambiguous metrics. Consequently, they fail to satisfy the theoretical requirements of proper scoring rules and differ fundamentally from our definition of calibrated confidence.

This study highlights the necessity of **calibration designed specifically for RAG** in LLM research and introduces a **simple yet effective solution**. Our method is compatible with any LLM encoder and utilizes synthetic training data explicitly crafted for this purpose, which we have made publicly available along with our code. As the **first approach to provide practical and theoretically grounded decision calibration in RAG scenarios**, we aim to establish a new standard for LLM research and its applications.

---
**We sincerely appreciate the constructive feedback once again, as it significantly enhances the robustness of our paper. We have summarized specific revision points at the end of each response to the reviewers' questions. If you find these points agreeable, we will incorporate them into the manuscript and notify you accordingly.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732435694014

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
The Release of the First Revised Manuscript.

comment:
https://openreview.net/pdf?id=nNQmZGjEVe

---

Here is how the feedback has been addressed in the manuscript. Due to the limited time, we incorporated the requested changes in advance:

* Updates to address the issues in Section 2 are highlighted in red within Section 3. Specifically:
    * (tAsW W1, W2)
    * (fXeq W1)
    * (46aU W2)
* The following changes have been applied to the tables in Appendix B:
    * Table 10: (tAsW W9)
    * Table 11: (tAsW W6)
    * Table 12: (tAsW W5)
    * Table 13: (tAsW W10, wbiP W2, fXeq W2)
    * Table 14: (wbiP W1)
* Additionally, adjustments were made to Appendix C:
    * Table 15: (tAsW W8)
* Replaced Figure 3 with a PDF image (fXeq W3).
---

We hope these updates meet the reviewers' expectations and look forward to receiving further feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new retrieval method, named Calibrated Retrieval-Augmented Generation (CalibRAG), to ensure the decisions informed by the retrieved documents are well-calibrated. The authors have performed empirical experiments to demonstrate that CalibRAG improves both the calibration performance and accuracy.

Overall, this paper introduces some interesting ideas. However, the technical contribution of the proposed method seems limited. Some experimental settings are not fair enough (e.g., the rerankig experiments). The experimental analysis is not suffcient enough. Some important baselines are missing.