When to Trust LLMs: Aligning Confidence with Response Quality (2024)

Abstract: Despite the success of large language models (LLMs) in natural language generation, much evidence shows that LLMs may produce incorrect or nonsensical text. This limitation highlights the importance of discerning when to trust LLMs, especially in safety-critical domains. Existing methods often express reliability by confidence level, however, their effectiveness is limited by the lack of objective guidance. To address this, we propose CONfidence-Quality-ORDer-preserving alignment approach (CONQORD), which leverages reinforcement learning guided by a tailored dual-component reward function. This function integrates quality reward and order-preserving alignment reward functions. Specifically, the order-preserving reward incentivizes the model to verbalize greater confidence for responses of higher quality to align the order of confidence and quality. Experiments demonstrate that CONQORD significantly improves the alignment performance between confidence and response accuracy, without causing over-cautious. Furthermore, the aligned confidence provided by CONQORD informs when to trust LLMs, and acts as a determinant for initiating the retrieval process of external knowledge. Aligning confidence with response quality ensures more transparent and reliable responses, providing better trustworthiness.

heading: 1 Introduction
text: Large Language Models (LLMs) have excelled in natural language understanding and generation (Brown et al., 2020; Anil et al., 2023). However, mounting evidence indicates that LLMs generate incorrect or nonsensical text, including fabricated citations or incorrect medical information, risking errors in critical applications (Ji et al., 2023; Zhang et al., 2023b; Agrawal et al., 2023; Du et al., 2023; Cohen et al., 2023). The urgent question is:
* Corresponding author
When can we trust LLMs? Addressing this concern is essential to prevent the uncritical acceptance of misleading information and to guide decisions on when to rely on LLMs versus when to seek external knowledge.
Recently, researchers have explored prompting LLMs to output calibrated confidence alongside text (Tian et al., 2023) for determining LLMs’ reliability. The essence of confidence calibration is to ensure that the confidence expressed corresponds with the correctness of the response, which is critical for the model’s transparency and trustworthiness. While classification tasks benefit from ground truth labels to calibrate predicted probabil-
5984
ities against actual correctness (Guo et al., 2017), generative tasks confront the challenge of calibration without clear ground truths. Current calibration methods for text generation rely on heuristics that consistency among multiple responses (Xiong et al., 2023) or the top-k responses facilitate calibration (Tian et al., 2023). However, these assumptions often bear little relevance to the intrinsic quality of the responses. Therefore, these strategies frequently result in a misalignment between the expressed confidence and the actual quality of the response, shown in Figure 1. The critical issue is the absence of a gold standard for confidence that directly reflects response quality, leaving models incapable of guiding confidence levels aligned with response quality.
In this study, we explore a Reinforcement Learning (RL) framework to tackle this challenge, designing reward functions that align confidence levels with response quality. This framework takes advantage of the adaptability afforded by the diverse reward functions in RL to bridge the gap between confidence and response quality (Guo et al., 2021; Ziegler et al., 2019). A direct reward approach involves rewarding language models for well-aligned response-confidence pairs while penalizing misaligned ones, where the reward model is fine-tuned on the constructed training data. However, this method can inadvertently encourage language models to take shortcuts. Specifically, such a strategy may inadvertently encourage the generation of lower-quality responses paired with correspondingly reduced confidence (empirically demonstrated in Section 3).
To address this issue, it is imperative to devise a reward function that encourages the generation of accurate and well-aligned responses. We propose CONfidence-Quality-ORDer-preserving alignment approach, called CONQORD (sounds as Concord), designing a dual-component reward strategy focusing on response quality and confidence alignment. The reward model comprises:: (i) A quality reward model that rates the response quality. (ii) An order-preserving alignment reward model encourages an ordinal relationship consistency between confidence and quality rating, while penalizing ordinal discrepancies. This order-preserving reward fosters careful self-calibration and adaptability to various contexts. The order-preserving nature of the reward minimizes the impact of outliers. By integrating the quality reward model with the alignment reward function, we apply the Proximal Pol-
icy Optimization (PPO) algorithm to harmonize verbalized confidence with response quality, thus preventing the model from becoming cautious.
We conduct experiments using four foundational models, including LLAMA-2 7B, Zephyr 7B, Mistral 7B, and LLAMA-2 13B, across two datasets: NQ and TruthfulQA. Experimental results demonstrate that our CONQORD substantially improves the alignment performance between confidence levels and the quality of responses without inducing excessive caution. Moreover, we evaluate the practicality of CONQORD’s calibrated confidence in adaptive retrieval task (Asai et al., 2023; Ding et al., 2024), where confidence scores are used to guide the activation of external knowledge. Our experiments confirm that CONQORD’s confidence alignment reliably dictates the trustworthiness of LLM outputs. CONQORD contributes to making the model-generated responses not only more transparent but also more reliable, through its refined confidence calibration.

heading: 2 Related Works
text: Confidence, or uncertainty, refers to the degree of certainty or assurance that accompanies a prediction or decision made by a model (Geng et al., 2023). The calibration of confidence is essential for the reliability of machine learning systems, as it ensures that predicted probabilities match the true likelihood of outcomes as closely as possible (Guo et al., 2017; Minderer et al., 2021). While in traditional classification tasks, this involves aligning predicted probabilities with actual ground truth labels, the task becomes more challenging for generative models due to the inherently ambiguous nature of the ground truth (Gawlikowski et al., 2021; Liu et al., 2023).
Confidence Elicitation in LLMs Confidence elicitation in LLMs aims to gauge the certainty of responses without modifying the model or accessing its internals (Geng et al., 2023). Mielke et al. (Mielke et al., 2022) proposed an external calibrator, which requires access to the model’s internals, often impractical. Lin et al. (Lin et al., 2022a) introduced verbalized confidence, prompting models to declare their certainty. Yet, they focused on fine-tuned models and did not explore zero-shot scenarios. Zhou et al. (2023) examined confidence in prompt design but did not provide explicit confidence measures to users.
Confidence Calibration in LLMs Calibration in LLMs is an emerging new research direction. Tian et al. (2023) investigate verbalized approaches to calibrate the confidence of LLMs. They direct LLMs to produce the top k predictions for a query, where each prediction is paired with a distinct probability value that reflects the model’s confidence in the accuracy of that prediction. Xiong et al. (Xiong et al., 2023) proposed a hybrid method by combining verbalized numbers and consistency-based scores for benchmarking.
Limitations and Challanges Prior methods depend on heuristic presumptions that assume consistency across multiple samples or posit that recalling top-k samples aids in confidence calibration. Nonetheless, these methods suffer from a lack of appropriate direction. This deficiency stems from the lack of a definitive ground truth standard for confidence that aligns precisely with the quality of the answers, presenting significant obstacles to the accurate alignment of confidence estimates.

heading: 3 Confidence Alignment via RL
text: We adopt a Reinforcement Learning (RL) framework to tackle the challenge of lacking a groundtruth standard for confidence assessment. Different from Supervised Fine-Tuning (SFT), which depends on labeled data, RL offers a more adaptable solution by allowing any indicator as a reward. We follow previous studies (Ramamurthy et al., 2023; Wu et al., 2023), regard text language generation as a Markov Decision Process (MDP) while the remaining elements of MDP are listed in Appendix A. In the rest of this section, we elaborate on the reward strategy.

heading: 3.1 Preliminary Alignment Approach
text: We introduce a preliminary alignment approach (PreApproach) to align confidence with response quality utilizing a reward model. The reward model is fine-tuned on tailored training data consisting of question-response-confidence tuples, to recognize and incentivize confidence alignment.
Data construction for fine-tuning reward model. The construction of training data begins with the generation of a dataset containing tuples in the format: <question, response, confidence score>. This dataset is derived from the existing RLHF dataset (Bai et al., 2022). For each instance in the original dataset, we create two new samples, one
TruthfulQA NQ Dataset
0.0
0.1
0.2
0.3
0.4
0.5
0.6
E C E
ECE↓ LLAMA2-7b Preliminary
(a) Expected Calibration Error (ECE)
0.430
0.435
0.440
0.445 Accuracy↑
LLAMA2-7b PreApproach
0.25
0.30
0.35
0.40 A cc ur ac y
TruthfulQA NQ Dataset
0.220
0.225
0.230
0.235
0.240
(b) Accuracy
Figure 2: Comparison between vanilla LLAMA-2 7b and PreApproach on TruthfulQA and NQ. Although PreApproach provides better calibration (lower ECE), PreApproach suffers a performance decline.
with a high confidence (such as 0.9) and another with a low confidence (such as 0.1). These samples are assigned scores based on the following criteria:
• Chosen: Alignment between response quality1 and confidence. High-quality responses with high confidence; low-quality responses with low confidence.
• Rejected: Misalignment. Good response with low confidence; bad response with high confidence.
We concatenate the response and its confidence, with examples provided in Appendix C, and use these data to train a reward model to discern that ‘chosen’ (alignment) is preferable to ‘rejected’ (misalignment). The reward model’s loss is computed using Equation 1. We then employ this reward model to direct LLM fine-tuning through Proximal Policy Optimization (PPO) (Schulman et al., 2017; Zheng et al., 2023), with details in Section 4.2.

heading: 3.2 Results and Discussions
text: We conduct experiments to compare the confidence alignment and response quality between PreApproach and vanilla LLAMA-2 7B (Touvron et al., 2023). More details of the experimental settings, including hyperparameters, datasets, and metrics, can be found in Section 5.1.
The experimental results are illustrated in Figure 2 (a), which shows that PreApproach attains a lower Expected Calibration Error (ECE) (Guo
1We view the responses labeled as ‘chosen’ in the original dataset as high-quality responses and those labeled as ‘rejected’ in the original dataset as low-quality response (Bai et al., 2022).
et al., 2017) than LLAMA-2 7B, signifying improved confidence alignment.
However, we observe a trade-off, illustrated in Figure 2 (b), where achieving alignment gain is at the expense of diminished response accuracy. This decline may stem from the RL fine-tuning process not focusing on improving response quality, leading to a shortcut where language models tend to produce lower-quality responses with lower confidence. The challenge arises from attempting to fine-tune a single reward model to evaluate both response quality and confidence alignment, which can inadvertently favor responses that meet alignment criteria but lack quality. Thus, there is a clear need for an approach designed to explicitly enhance both response quality and confidence calibration, thereby mitigating the unintended reduction in accuracy while pursuing confidence alignment.
4 CONfidence-Quality-ORDerpreserving Alignment Approach
In this section, we propose CONfidence-QualityORDer-preserving alignment approach via reinforcement learning, namely CONQORD.

heading: 4.1 Dual-component Reward Strategy
text: To achieve quality and alignment, we decouple the reward function into two components to assess the two objectives, separately:
1. Quality reward function for assessing response quality.
2. Alignment reward function for assessing the consistency between the response quality and verbalized confidence stated by LLM.
This strategy is based on the notion that an accurate assessment of response quality is a prerequisite for aligning it with confidence. Hence, we decouple the evaluation of response quality. By fine-tuning a quality reward model that accurately judges response quality, we can streamline the process of aligning quality rewards with confidence, thus facilitating reward model fine-tuning.
Quality reward. We develop a quality reward model to evaluate the response quality. To train this model, we utilize Reinforcement Learning from Human Feedback (RLHF) datasets, ensuring that the high-quality response receives a higher score than the low-quality ones. We employ a binary
ranking loss (Touvron et al., 2023) as LQ:
LQ = −log(σ(RQ(x, yh)−RQ(x, yl))) (1)
where RQ(·) denotes quality reward, x refers to the input prompts, yh and yl denotes the high-quality and low-quality responses, σ refers to the sigmoid function.
Order-preserving alignment criterion. For confidence alignment, we first introduce an orderbased criterion, which preserves a consistent order relationship between verbalized confidence and response quality. Specifically, for any pair of samples, i and j, an desired relationship between tuples (xi, yi, ci) and (xj , yj , cj) should preserve the order:
ci ≤ cj ⇐⇒ RQ(xi, yi) ≤ RQ(xj , yj), (2)
where ci denotes the golden confidence for sample i. This criterion is grounded in the intuition that a higher quality response should be accompanied by a higher stated confidence.
Order-preserving alignment reward. Guided by the above criterion, we propose an orderpreserving alignment reward function RA:
RA(xi, yi, ci)
= ∑
j ̸=i (ci − cj) · (RQ (xi, yi)−RQ (xj , yj))
(3) The reward function is defined as the sum of the products of pairwise differences in confidence and corresponding reward scores for all samples. This design inventively rewards the alignment of confidence with the quality of responses, thereby enforcing a direct proportionality between a sample’s stated confidence and its actual quality. It penalizes any deviations from this alignment.
This function promotes an environment that motivates participants to calibrate the quality of their responses to align with their expressed confidence levels, thereby improving the accuracy of responses. It prioritizes relative comparison over absolute measures, encouraging meticulous selfassessment and offering adaptability across various contexts. Moreover, the order-preserving nature of the reward function is robust to outliers, ensuring that the system maintains its integrity even in the presence of anomalous data points.
Overall Reward. The overall reward function RO consists of both quality reward RQ and orderpreserving alignment reward RA, which can be summarized as follows:
RO(xi, yi, ci)
=RQ(xi, yi) + α ·RA(xi, yi, ci), (4)
where α is the hyper-parameter balancing the quality reward and order-preserving alignment reward.

heading: 4.2 RL Fine-tuning LLM
text: To improve confidence alignment, we train LLM using the reinforcement learning (RL) framework (Wu et al., 2023; Touvron et al., 2023), employing our dual-component reward 4 as an approximation of the golden reward and the vanilla pre-trained LLM as the policy π for optimization. During this phase, our objective is to optimize the following functions:
argmax π Ex∼D,ŷ∼π[R(ŷ | x)], R(ŷ | x) = RO(ŷ | x)
− βDKL(πθ(ŷ | x) ∥ π0(ŷ | x)), (5)
where R is the final reward function containing a penalty term for diverging from the original policy π0. We iteratively improve the policy by sampling prompts x from D and outputs ŷ from the policy π and adopt Proximal Policy Optimization (PPO) (Schulman et al., 2017), an actor-critic RL algorithm, to improve our objective.

heading: 4.3 Comparison with PreApproach
text: We analyze the difference between the PreApproach in Section 3 and CONQORD in Section 4.
PreApproach manually assigns confidence scores to construct samples for fine-tuning reward model data. This process is susceptible to introducing bias. For instance, the prevailing methodology may inadvertently condition the reward model to perceive confidence as a binary choice, with probabilities often anchored to extreme values such as 0.1 or 0.9.
In contrast, our CONQORD method introduces an order-preserving alignment reward function that circumvents this issue by not requiring explicit confidence specification. This approach inherently reduces bias, as it eliminates the need to pre-defined confidence levels, instead allowing the model to infer confidence in a more nuanced and unbiased manner.
Therefore, CONQORD is more robust and generalizable compared with the PreApproach, which is empirically demonstrated in Section 5.5.

heading: 5 Experiments
text: In this section, we evaluate the alignment performance of our proposed CONQORD on benchmark datasets. Further hyperparameter analysis and case studies are also provided.

heading: 5.1 Experimental Settings

heading: 5.1.1 Datasets
text: We conduct experiments on two tasks, hallucination evaluation and question-answering. For hallucination evaluation, we utilize TruthfulQA (Lin et al., 2022b), which contains 817 questions spanning 38 categories designed to test language models’ tendency to mimic human falsehoods (Zhang et al., 2023a). For question-answering, we adopt the widely-used Natural Questions (NQ) dataset (Kwiatkowski et al., 2019), which comprises real anonymized, aggregated queries issued to the Google search engine, forming a questionanswering dataset. We randomly sample 500 examples from the dev set due to the cost consideration of running experiments.

heading: 5.1.2 Baselines
text: We compare our CONQORD with three baselines on four foundation models, including LLAMA-2 7B, LLAMA-2 13B (Touvron et al., 2023), Zephyr 7B (Tunstall et al., 2023), and Mistral 7B (Jiang et al., 2023). The baseline methods are as follows:
• Vanilla elicits verbalized confidence to directly request them to output a confidence score ranging from 0 to 1.
• Top-k: Tian et al. (Tian et al., 2023) prompt LLMs to generate the top K predictions for a query, each accompanied by an explicit probability that represents the model’s confidence in its prediction.
• CoT+Agg: Xiong et al. (Xiong et al., 2023) leverage the Chain-of-Thought (Wei et al., 2022) prompting strategy. This strategy has been demonstrated to be effective in inducing reasoning processes in LLMs.
All the prompts to induce confidence are listed in Appendix B.

heading: 5.2 Evaluation Metric
text: To evaluate the alignment of the verbalized confidence and response quality, we employ widely used Expected Calibration Error. We also utilize the Pearson Correlation coefficient and Spearman Rank Correlation Coefficient for alignment assessment:
• Expected Calibration Error (ECE) (Guo et al., 2017): ECE is defined as the average (squared) error between the average accuracy and confidence within each bin, where each error is weighted by the fraction of samples falling within the bin.
• Pearson Correlation Coefficient (PCC) (Cohen et al., 2009): PCC evaluates the linear relationship between two data sets, calculated as the covariance of the variables normalized by the product of their standard deviations.
• Spearman’s Rank Correlation Coefficient (SRCC) (Sedgwick, 2014): SRCC determines the rank-based correlation between two variables, evaluating the extent to which their relationship can be modeled by a monotonic function.
• Accuracy: We instruct GPT-4 (OpenAI, 2023) to calculate the accuracy score of generated responses by comparing them with ref-
erence responses using prompt-based instructions (see Appendix B).

heading: 5.2.1 Setup
text: In our study, we employ the foundational model architecture for the reward, reference, value, and actor models. The fine-tuning of the reward model utilizes the Helpful & Harmless dataset (Bai et al., 2022). Across all foundation models, the AdamW optimizer is chosen as the optimization algorithm. We set the KL penalty coefficient, β, to 0.005, aligning with the parameters used in prior research (Touvron et al., 2023). In our primary experiments, we select an α value of 0.4. We apply a weight decay of 0.1 and maintain a constant learning rate of 10−6. During each iteration of Proximal Policy Optimization (PPO), we process batches of 32 samples and perform a single gradient update per mini-batch. Experiments are conducted on eight 80G A100 GPUs.

heading: 5.3 Confidence Alignment Evaluation
text: We conduct experiments to demonstrate the effectiveness of our CONCORD on TruthfulQA and NQ datasets. Table 1 and Table 2 illustrate the performance of aligning methods under four foundation models (LLAMA-2 7B, Zephyr 7B, Mistral 7B, and LLAMA-2 13B) on the TruthfulQA and NQ datasets. Ideally, a model with lower ECE values, higher Pearson and Spearman coefficients, and
lower p-values demonstrates stronger confidence alignment.
We observe that our CONQORD generally exhibits strong confidence alignment, as evidenced by the best Expected Calibration Error (ECE) across most of the datasets and model sizes. This suggests that predictions are closely aligned with actual qualities. CONQORDD maintains the foundational model’s performance while significantly improving calibration, unlike PreApproach, which causes a notable performance drop. However, when compared to the CoT prompt that enhances performance, our method still has room for improvement, which is a focus for our future research efforts In contrast, the Vanilla method displays the highest ECE and lowest correlation metrics, indicating a lack of quality-confidence alignment. The performance of the Top-k and CoT+Agg methods varies depending on the dataset and the model size, neither achieves the same level of alignment as CONCORD. Despite the CoT+Agg method achieving marginally higher Pearson and Spearman correlations in the 7B model on the NQ dataset, CONQORD’s calibration superiority is evident given its consistently low ECE.
In a nutshell, the experimental results demonstrate CONCORD’s superiority in confidence alignment. CONCORD’s strong calibration indicates a more reliable correlation between the model’s expressed confidence and the actual accuracy of
its responses. Such findings reveal CONCORD’s potential in applications requiring accurate confidence estimations.

heading: 5.4 Adaptive Retrieval Experiments
text: To evaluate the effectiveness of the calibrated CONQORD model in enhancing retrieval augmentation performance, we conduct experiments utilizing the confidence scores generated by the calibrated model. These scores serve as a basis for determining whether external retrieval should be employed. Specifically, when the CONQORD model outputs a low confidence score, we utilize input questions for retrieval to incorporate external knowledge and assist the model in generating accurate responses. Conversely, when the confidence score is high, we
consider it unnecessary to introduce retrieval at that stage, as the model itself is capable of directly producing high-quality responses.
We conduct experiments on both the TruthfulQA and NQ datasets, and the results are presented in Figure 3. Firstly, we observe that the calibrated model produces higher-quality responses for highconfidence outputs, demonstrating that our model effectively aligns with the confidence and response quality. Secondly, by selecting suitable confidence thresholds (0.8 for TruthfulQA and 0.7 for NQ), we find that utilizing retrieval augmentation for low-confidence responses significantly improves response accuracy. However, introducing retrieval augmentation for high-confidence responses may lead to unexpected performance degradation. For instance, there is a certain performance loss of the RAG model on the TruthfulQA dataset, which we attribute to the introduction of misleading questions through retrieval, resulting in additional information noise. Therefore, choosing an appropriate confidence threshold in practical applications enables us to fully leverage the model’s inference capability while minimizing unnecessary retrieval and avoiding noisy information.

heading: 5.5 Hyper-Parameter Analysis
text: We analyze the sensitivity of our CONQORD to the hyper-parameter α from two perspectives: response quality and alignment effectiveness. The performance refers to the accuracy, which is evaluated by GPT-4. The alignment effectiveness refers to the widely-used ECE. We vary the α in
{0.0, 0.2, 0.4, 0.6, 0.8, 1.0}. As shown in Figure 4, ECE is decreased with the increases of α, which validates the effectiveness of alignment reward RA on aligning the verbalized confidence and response quality. The accuracy is observed to be insensitive to changes in α.
Besides, we also show the performance of PreApproach in Section 3 for comparison. We observe that across a range of α values, CONQORD consistently outperforms PreApproach with respect to response quality and the alignment between quality and confidence. This observation demonstrates the superiority of the dual-component reward function implemented in CONQORD.

heading: 5.6 Case Study
text: Our case study focusing on TruthfulQA serves as an illustrative example of how our calibrated model, CONQORD, effectively aligns verbalized confi-
dence with the actual quality of responses generated by LLMs. The examples presented in Table 3 reveal that traditional baseline methods frequently overstate confidence, which can mislead users and undermine trust in LLMs. In contrast, CONQORD exhibits a marked enhancement in this respect, calibrating confidence scores to closely correspond with response quality. Unlike baseline methods might assign unnecessarily high confidence to lowquality responses, CONQORD judiciously adjusts confidence levels, ensuring that high confidence is indicative of high-quality responses. This calibration is a pivotal advancement in bolstering the trustworthiness of generated content.

heading: 6 Conclusion and Future Work
text: In this paper, we propose a confidence-qualityorder-preserving alignment approach (CONQORD), which marks a significant step forward in the domain of confidence alignment for LLMs. CONQORD is a reinforcement learning method with a well-designed dual-component reward strategy, containing both quality reward and order-preserving alignment reward functions. Specifically, the alignment reward encourages LLM to generate higher confidence with higher quality scores. Experiments have demonstrated that our CONQORD not only achieves better alignment performance between confidence and quality but also preserves the quality of the model’s responses. Furthermore, the aligned confidence provided by CONQORD can serve as a determinant for initiating external knowledge.
We view confidence alignment as a promising research direction, with significant potential for advancing the field. Key research questions include enhancing response quality alongside alignment accuracy, leveraging aligned confidence as a supervisory signal for self-reflection and model improvement, and extending experimentation to additional downstream applications. We are enthusiastic about exploring directions further and plan to conduct additional investigations in the future.
Limitations
This paper proposes a confidence calibration method based on reinforcement learning with human feedback (RLHF) to align the verbalized confidence with actual response quality. However, it is important to acknowledge the limitations of this research. Firstly, the proposed method is primarily
applicable to open-source models, as it relies on adjusting the model’s weight parameters for calibration. For commercial closed-source models, where access to the weight parameters is restricted, the proposed method may not be suitable. Additionally, due to practical constraints and experimental costs, this study only conducted experiments on the 7B or 13B foundation model. Therefore, the generalizability of the proposed method to large parameter scales (such as 70B) remains unexplored and is left for future work. It is crucial to investigate the effectiveness and applicability of the proposed approach across a broader range of models to establish its wider practical utility.

heading: Acknowledgement
text: This work is funded by the National Key R&D Program of China (2022YFB3103700, 2022YFB3103701), the Strategic Priority Research Program of the Chinese Academy of Sciences under Grant No. XDB0680101.

heading: A Text Generation Environment
text: We provide a reinforcement learning environment for text generation in LLM.
Our study concentrates on text-generation tasks. For each task, we are provided with a collection of input prompts, denoted as D = {xn}Nn=1. We follow the framework of (Ramamurthy et al., 2023; Wu et al., 2023)) to model language generation as a Markov Decision Process (MDP), represented by the tuple ⟨S,A,R, P, γ, Tmax⟩, where V is a finite vocabulary set.
In this MDP, an episode commences with a randomly chosen prompt x = (x1, x2, . . . , xl), where each xi is an element of V . The episode concludes either when the sequence generation surpasses the maximum time step Tmax or when an end-of-sequence token is produced. The state space is denoted by S, with the initial state s0 = (x1, x2, . . . , xl) belonging to S . An action at ∈ A, which is a token generated by the policy language model Pθ at time t, is selected from V , with a0 signifying the beginning of the sequence. The transition function P extends the current state st by appending the action at, resulting in the sequence (x1, x2, . . . , xl, a0, a1, . . . , at−1). This iterative process is maintained until the final time step T ≤ Tmax, yielding the output sequence y = (a1, . . . , aT ).
The reward function R assigns real-valued rewards based on the criteria detailed in Section 5, providing feedback at each step and upon reaching time T . The policy model Pθ may be initialized using a pre-trained language model and potentially further refined through supervised fine-tuning using task-specific examples. The specifics of the reward function are elaborated upon later in the text.

heading: B Prompt
text: B.1 Prompts for GPT evaluation The prompt for GPT evaluation is:
Determine whether the given Answer and Reference for a specified Question are semantically equivalent. If the Answer matches or aligns with the Reference’s meaning, or if the Answer is included in the Reference, rate it as 1.0. If not, rate it as 0.0. Provide only 0.0 or 1.0, with no other choice or explanation. \n### Question: ${QUESTION}. \n### Answer: ${OUTPUT_ANSWER}. \n###
Reference: ${GOLDEN_ANSWER}. \n### Rating:
B.2 Prompts for Inducing the Verbalized Confidence
Vanilla & Ours Please provide the answer and the Confidence (0.0 to 1.0) that the answer is truthful and informative for the Question.\n### Question: who played will on as the world turns?\n \n ### Answer: Jesse Soffer.\n### Confidence: 0.9.\n### Question: how many episodes in season 2 breaking bad?\n### Answer: 5\n### Confidence: 0.2.\n### Question: Note that to ensure a fair comparison, our CONQORD also adopts the above prompt to induce confidence.
Top-k Provide your 2 best answers and the Confidence that each is correct (0.0 to 1.0) for the following question. Give ONLY the task output description of your guesses and probabilities, no other words or explanations. The format is: ### Question: \n### Answer-1: Yes. Bruno Richard Hauptmann was sentenced to death for the kidnapping.\n### Confidence-1: 0.9.\n### Answer-2: No.\n### Confidence-2: 0.0.\n\n ### Question:
CoT+Agg Please provide the answer and the Confidence (0.0 to 1.0) that the answer is truthful and informative for the Question. Let’s think it step by step.\n\n ### Question: Was the Lindbergh kidnapping ever solved?\n### Explanation: Bruno Richard Hauptmann was sentenced to death for the kidnapping. \n### Answer: Yes.\n### Confidence: 0.9.\n\n ### Question:
B.3 Prompts for RAG Given the following question and
the retrieved evidence, please generate a truthful and informative answer. Only answer, no explanations.\n\n\n### Question: question\n\n\n### Retrieved Evidences: rag\n\n\n### Answer:

heading: C Examples of Constructed Tuple in PreApproach
text: Chosen:
• ### Question: ${QUESTION}. ### Answer: ${GOOD_ANSWER}. ### Confidence: 0.9.
• ### Question: ${QUESTION}. ### Answer: ${BAD_ANSWER}. ### Confidence: 0.1.
Rejected:
• ### Question: ${QUESTION}. ### Answer: ${GOOD_ANSWER}. ### Confidence: 0.1.
• ### Question: ${QUESTION}. ### Answer: ${BAD_ANSWER}. ### Confidence: 0.9.

heading: Future Work
text: We are enthusiastic about exploring directions further and plan to conduct additional investigations in the future. Limitations
This paper proposes a confidence calibration method based on reinforcement learning with human feedback (RLHF) to align the verbalized confidence with actual response quality. However, it is important to acknowledge the limitations of this research. Firstly, the proposed method is primarily
applicable to open-source models, as it relies on adjusting the model’s weight parameters for calibration. For commercial closed-source models, where access to the weight parameters is restricted, the proposed method may not be suitable. Additionally, due to practical constraints and experimental costs, this study only conducted experiments on the 7B or 13B foundation model. Therefore, the generalizability of the proposed method to large parameter scales (such as 70B) remains unexplored and is left for future work. It is crucial to investigate the effectiveness and applicability of the proposed approach across a broader range of models to establish its wider practical utility