Reinforcement Learning for Better Verbalized Confidence in Long-Form Generation

## Abstract

Abstract Hallucination remains a major challenge for the safe and trustworthy deployment of large language models (LLMs) in factual content generation. Prior work has explored confidence estimation as an effective approach to hallucination detection, but often relies on post-hoc self-consistency methods that require computationally expensive sampling. Verbalized confidence offers a more efficient alternative, but existing approaches are largely limited to short-form question answering (QA) tasks and do not generalize well to open-ended generation.
In this paper, we propose LoVeC ( Lo ng-form Ve rbalized C onfidence), an on-the-fly verbalized confidence estimation method for long-form generation. Specifically, we use reinforcement learning (RL) to train LLMs to append numerical confidence scores to each generated statement, serving as a direct and interpretable signal of the factuality of generation. Our experiments consider both on-policy and off-policy RL methods, including DPO, ORPO, and GRPO, to enhance the model calibration. We introduce two novel evaluation settings, free-form tagging and iterative tagging , to assess different verbalized confidence estimation methods. Experiments on three long-form QA datasets show that our RL-trained models achieve better calibration and generalize robustly across domains. Also, our method is highly efficient, as it only requires adding a few tokens to the output being decoded.

## 1 Introduction

While large language models (LLMs) demonstrate impressive performance across a wide range of tasks [1, 2, 3], one of their most critical limitations is hallucinations [4, 5].
When faced with unfamiliar or uncertain input, LLMs often generate fabricated or incorrect content. These hallucinations pose a significant barrier to the real-world deployment of LLMs [6, 7, 8, 9, 10], especially in high-stakes domains such as medicine, law, and finance, where factual inaccuracies can have serious consequences [6, 7, 8].

Reliable confidence and uncertainty estimation is thus crucial for improving the trustworthiness and practical applicability of LLMs.
Accurate confidence or uncertainty scores enable effective downstream strategies such as selective question answering [6], retrieval-augmented generation [11], LLM cascading [12], and fact re-ensembling [13], thereby enhancing model robustness and utility.
Following the definitions by Lin et al. [14], uncertainty refers to the variability or dispersion in the model’s predictions given only the input query. In contrast, confidence is defined with respect to both the input and the specific generated output, capturing how certain the model is about that particular response.
While much prior research on confidence estimation has focused on short-form question answering (QA) tasks, long-form QA (with outputs exceeding 100 words) is generally more common and better aligned with real-world applications [6, 7, 8, 9].

Recently, there has been growing interest in confidence estimation methods for long-form outputs [6, 7, 8, 9, 12, 13, 15, 16]. However, existing approaches are often post-hoc and computationally inefficient. Many rely on generating multiple samples for consistency checking [6, 7, 13], or require an additional model (e.g., GPT-4 [17]) to extract atomic claims [15, 16]. In contrast, verbalized confidence offers a potentially more efficient alternative, as it avoids both multiple sampling and auxiliary models. However, verbalized confidence remains underexplored in the context of long-form text generation, and it is unclear whether it can provide well-calibrated confidence estimates.

Figure: Figure 1: Overview of our two evaluation settings. In Free-form Tagging, the model generates both the answer and confidence score suffix. In Iterative Tagging, the model is given a fixed response and assigns confidence scores sentence-by-sentence.
Refer to caption: x1.png

To address these challenges, we propose LoVeC (Long-form Verbalized Confidence),
an on-the-fly verbalized confidence estimation method that generates confidence scores alongside long-form factual statements in a single decoding pass (Contribution #1).
Specifically, we apply a reinforcement learning (RL)-based approach that enables LLMs to produce well-calibrated confidence estimates during text generation (Figure [1](https://arxiv.org/html/2505.23912v1#S1.F1)). Compared to supervised fine-tuning (SFT), RL enables direct optimization toward task-specific reward signals, aligning model behavior with desired outcomes beyond token-level likelihoods [18, 19]. Moreover, RL does not require fine-grained token-level annotations, which are often expensive or unavailable in practice [20, 21]. We explore both on-policy and off-policy RL to generate confidence scores during long-form generation.

Another key challenge in confidence calibration lies in the fair and rigorous evaluation of different models and methods. To this end, we propose two novel evaluation settings for verbalized confidence estimation in long-form generation (Contribution #2; illustrated in Figure [1](https://arxiv.org/html/2505.23912v1#S1.F1)): free-form tagging and iterative tagging. In free-form tagging, the model is prompted with a question and generates a complete answer with verbalized confidence suffix. Since different models may produce different outputs under this setting, direct comparison can be challenging. To address scenarios where a fixed long-form response is required, we introduce iterative tagging, a novel setting in which the model is provided with a fixed answer and tasked with assigning confidence scores sentence-by-sentence.

Our experiments on Llama-3-8B-Instruct [22] and Gemma-2-9B-It [23], evaluated across three in-domain and out-of-domain long-form QA datasets, demonstrate improved calibration in both iterative and free-form tagging. Our analysis further shows that LoVeC is highly efficient, achieving a 20$\times$ speedup compared to state-of-the-art methods, and generalizes well to short-form QA tasks.

## 2 Related Work

Confidence/Uncertainty Estimation in Long-form Generations.
Previous research on confidence and uncertainty estimation has primarily focused on multiple-choice or short-form question answering [14, 24, 25, 26, 27, 28, 29, 30, 31]. Recently, there has been increasing interest in confidence and uncertainty estimation for long-form generation.
Huang et al. [12] introduce a unified calibration framework applicable across text generation tasks by comparing the distributions of both correctness and confidence scores.
Zhang et al. [6] propose LUQ, an uncertainty estimation method designed for long-form generation at both the sentence and passage levels. This approach requires sampling multiple responses, making it computationally expensive.
Several studies [7, 13, 15, 16, 32] explore post-hoc methods that estimate claim-level uncertainty in long-form outputs. While these approaches offer finer-grained confidence estimates, they typically rely on GPT-based claim extraction, leading to high computational cost [7, 13, 15].
In contrast, we propose an on-the-fly verbalized confidence estimation method that generates confidence scores alongside long-form factual statements in a single decoding pass. Our methods do not need additional sampling or API calling, making it more efficient and scalable.

Verbalized Confidence Estimation.
Teaching LLMs to verbalize their confidence in short-form generation has been widely explored through two main approaches: prompt-based and training-based methods.
Prompt-based methods [29, 30] design prompts encouraging models to express uncertainty but do not improve internal calibration, often resulting in overconfidence.
Training-based methods fine-tune models with specialized datasets to explicitly express uncertainty during generation.
A line of work encourages models directly to abstain to answer by saying “I don’t know” when encountering unfamiliar inputs [33, 34, 35].
Another line of work trains models to produce two-part responses: first generating an answer, followed by an explicit confidence estimate [36, 37, 38, 39, 40].
However, extending verbalized uncertainty to long-form generation remains challenging, as multiple aspects may vary in certainty within a single response.
Recent work addresses this problem by tightly coupling uncertainty cues with the generated output. LoGU [8] trains models to flag uncertain claims during generation, and Band et al. [41] propose linguistic calibration by embedding expressions such as “I believe” or “I am 70% uncertain” into the text. Although both approaches improve human interpretability, they lack machine interpretability, making post-processing and integration with downstream tasks more difficult. In contrast, our method produces structured outputs by appending numerical confidence tags to each sentence, offering greater flexibility and interpretability.

Reinforcement Learning for Confidence Estimation.
Reinforcement learning (RL) is gaining traction for fine-tuning LLMs [19, 42]. When target behaviors can be sampled from the base model, recent work suggests RL often outperforms supervised fine-tuning (SFT) in aligning model outputs [43, 44]. Confidence estimation via RL remains a relatively new direction, primarily explored in short-form settings. Proximal policy optimization (PPO) based methods such as RewardingDoubt [40] and SaySelf [37] have achieved strong results in QA tasks, outperforming SFT-based techniques like R-tuning [45]. Extending RL to long-form confidence remains limited. LoGU [8] employs direct preference optimization (DPO) [18] to guide ordinal phrase generation, while Band et al. [41] use PPO to calibrate user-facing answers. In our work, we explore multiple algorithms, sucha as DPO, odds ratio preference optimization (ORPO), group relative policy optimization (GRPO) [46] to add a bounded numerical scale after each statement as verbalized confidence during the generation.

## 3 Preliminaries

In this section, we introduce the preliminaries of confidence estimation in long-form generation.

Primary Goal. In long-form confidence estimation, the primary objective is to align confidence scores with the factuality of the generated output [6, 7, 8, 12, 13, 15, 16]. The focus on factuality is mainly for two reasons: (1) hallucinations remain a significant challenge in LLMs, and confidence estimation can effectively indicate potential hallucinations during generation; (2) the factuality of a sentence can be objectively assessed, enabling a more quantitative and consistent evaluation compared to subjective criteria such as creativity or coherence [7, 8].

Granularity. Formally, given an input query $q$, an LLM parameterized by $\theta$ generates a response $y=\pi_{\theta}(q)$. Confidence estimation can be performed at various granularities depending on whether the confidence score is assigned at the level of atomic claims (a short sentence conveying a single piece of information) [7, 13, 15], for each sentence [6, 10], or the whole passage [6, 12]. For sentence-level confidence estimation, the response $y$ is defined as:
$y=\pi_{\theta}(q)$ consisting of a sequence of sentences $\mathbf{s}$ and corresponding confidence scores $\mathbf{c}$:

$$ $y=\{(s_{1},c_{1}),(s_{2},c_{2}),\dots,(s_{n},c_{n})\}=\{(s_{i},c_{i})\}_{i=1}^ {n}.$ (1) $$

where $s_{i}$ represents the $i^{th}$ sentence and $c_{i}\in[0,1]$ denotes the corresponding confidence score,
representing the estimated probability of factual correctness; higher values indicate greater confidence.

Factuality Evaluation. Each sentence $s_{i}$ is assigned a factuality score $f_{i}\in[0,1]$, reflecting its actual factual accuracy. These factual scores $\mathbf{f}$ are obtained by prompting an oracle verification model $\mathcal{O}$ with suitable supporting evidence $E$ pertinent to the query $q$:

$$ $\mathbf{f}=\texttt{FactCheck}(\mathcal{O},q,E,\mathbf{s}).$ (2) $$

Confidence Evaluation. To evaluate these confidence scores, the objective is to ensure the confidence scores $c_{i}$ generated by the model are well-calibrated and closely align with the independently determined factuality scores $f_{i}$. This calibration requirement is expressed as:

$$ $\forall i\in\{1,2,\dots,|\mathbf{f}|\},c_{i}\approx f_{i},|\mathbf{f}|=| \mathbf{c}|$ (3) $$

Various metrics can be applied to measure this alignment. We discuss more details in Section [5](https://arxiv.org/html/2505.23912v1#S5).

## 4 Long-form Verbalized Confidence

### 4.1 Confidence Estimation via RL

We formulate the task of verbalizing confidence as a sequential decision-making problem on top of language generation. A LLM operates as the policy $\pi_{\theta}$, parameterized by $\theta$. The objective of the policy is to assign confidence scores to its generated factual statements, such that these scores align with independently verified factuality assessments. Notably, hallucination is not penalized as generation errors; instead, the model is expected to assign low confidence scores to hallucinated statements, thereby facilitating hallucination detection.

We estimate confidence at the sentence level, rather than at the passage or atomic-claim level. Sentence-level estimation balances interpretability, computational efficiency, and alignment with natural language structure. Compared to passage-level estimation, it allows for finer-grained assessment. Compared to atomic-claim-level methods, it avoids extra decomposition steps and produces outputs that are more easily interpreted by humans. Moreover, using numerical confidence scores supports flexible post-processing without affecting text fluency or factual content, unlike methods that embed confidence markers (e.g., “I believe”, “I am uncertain”) directly into the output. For evaluation, we introduce two task settings: free-form tagging and iterative tagging.

Free-form Tagging.
We study a setting in which the policy model $\pi_{\theta}$ produces factual statements along with their associated confidence estimates in a single generation pass. As shown in the left part of Figure [1](https://arxiv.org/html/2505.23912v1#S1.F1), in this formulation, the action space includes all possible factual statements $s$ and corresponding confidence values $c$, spanning the model’s full vocabulary. The model outputs a sequence of sentence–confidence pairs, $y=\{(s_{1},c_{1}),(s_{2},c_{2}),\ldots,(s_{n},c_{n})\}$, by maximizing the following objective, where $t$ is the $t^{\text{th}}$ output token in sequence $y$:

$$ $y_{t}=\underset{y_{t}}{\mathrm{argmax}}\;\pi_{\theta}(y_{t}|y_{<t},q)$ (4) $$

This free-form setting gives the model full generative freedom to balance content generation with calibrated confidence expression. For example, we can use the output confidence to further constrain model to decode only high confidence statements in on-stake domains such as medicine, law, etc.

Iterative Tagging.
We also evaluate models in a controlled setting where the content is fixed and only the confidence scores are predicted. This setting is motivated by use cases where the generation cannot be altered and provides a consistent basis for model comparison. As shown on the right in Figure [1](https://arxiv.org/html/2505.23912v1#S1.F1), given a query $q$ and a base language model $\pi_{\text{base}}$, we first generate a static output $y_{\text{base}}=\{s_{1},s_{2},\ldots,s_{n}\}$. The policy model $\pi_{\theta}$ is then asked to assign confidence scores $c_{i}\in\{0,1,\ldots,10\}$ for each sentence, conditioned on the query and previously tagged pairs:

$$ $c_{i}=\underset{c}{\mathrm{argmax}}\;\pi_{\theta}(\{q,(s_{1},c_{1}),(s_{2},c_{ 2}),\ldots,(s_{i-1},c_{i-1}),s_{i}\},c)$ (5) $$

By decoupling content generation from confidence estimation, this setting ensures fair comparison across models and only requires models to generate confidence scores. In contrast to free-form tagging, it avoids the confounding effects of content variation on confidence evaluation.

Why RL.
RL offers key advantages over SFT for confidence estimation. SFT often requires dense annotations. However in our task, confidence score will only be presented after each statement, providing weak and sparse supervision signal. By contrast, RL can learn from sparse, delayed rewards [47, 48]. In addition, compare to SFT, RL models the ordinal relation between confidence scores (e.g., $10$ is more confident than $9$) directly with the training reward, offering direct optimization for consistency between confidence and factuality scores. RL also naturally supports credit assignment across different statement and confidence pairs. Thus, we adopt RL to train models for confidence estimation. To explore the trade-offs between performance and computational cost, we implement both on-policy and off-policy variants, making it well-suited for this sequential calibration problem with sparse rewards derived from external factuality evaluators. Specifically, we study three representative algorithms, GRPO [46], DPO [49], and ORPO [50], alongside other baseline methods, providing a comprehensive survey.

### 4.2 On-Policy Design

Given a data point $d=(q,E)\sim\mathcal{D}$, containing a query $q$ and the evidence $E$ for verification, the output sequence $y=\{(s_{i},c_{i})\}_{i=1}^{n}$ can be sampled from $y=\pi_{\theta}(q)$. Given an oracle model $\mathcal{O}$, we can obtain the ground truth factuality $\mathbf{f}=\texttt{FactCheck}(\mathcal{O},q,E,\mathbf{s})$. In our setting, the core design challenge for on-policy RL lies in constructing a reward signal that encourages aligning the model’s predicted confidence scores $\mathbf{c}$ with the factual correctness of each statement $\mathbf{f}$.

Figure: Figure 2: GRPO Reward Function
Refer to caption: x2.png

Intuitively, we want to reward the model when the confidence $c_{i}$, and correctness $f_{i}$ for each statement $s_{i}$ are close (e.g., high correctness - high confidence and vice versa), and penalize the model when they are far part (e.g., low correctness, high confidence, vice versa). We use a log-base reward as it imposes stronger penalties to miscalibration comparing to simple linear and quadratic losses. The log-base reward is more appropriate for risk-sensitive applications where confidence must reflect true correctness likelihood. We design this confidence reward $r^{\text{conf}}$ for an output $y$ using binary cross-entropy loss as below, where $\lambda$ is the scaling factor, $R_{max}$ is the normalizing factor and $\odot$ is the Hadamard product. Figure [2](https://arxiv.org/html/2505.23912v1#S4.F2) provides the visualization of our reward function.

$$ $r^{\text{conf}}=\lambda\cdot\frac{1}{n}\mathbf{1}^{\top}\left(\mathbf{1}+\frac {\mathbf{f}\odot\log(\mathbf{c})-(\mathbf{1}-\mathbf{f})\odot\log(\mathbf{1}- \mathbf{c})}{R_{max}}\right)$ (6) $$

Both confidence $c_{i}$ and factuality $f_{i}$ are normalized from integers in $\{0,1,\dots,10\}$ to real numbers in $[0,1]$ for numerical stability in our implementation. In practice, we combine the confidence reward with other subordinate objectives (e.g., informativeness, format reward) to ensure model is accurately expressing confidence while retaining the quality of generation, with more details in Appendix [B](https://arxiv.org/html/2505.23912v1#A2).

We instantiate this on-policy setup using the GRPO algorithm [46], a recent on-policy method adapted from PPO [51]. Given a dataset $\mathcal{D}=\{d_{1},d_{2},\ldots,d_{N}\}$, where each data point $d=(q,E)$, we sample a group of output trajectories $\mathbf{y}=\{y_{1},y_{2},\ldots,y_{G}\}$ from the current policy $\pi_{\theta_{\text{old}}}$ and obtain the group reward $\mathbf{r}=\{r_{1},\dots r_{G}\}$. Then we calculate the averaged advantage $\hat{A}_{j}(\pi_{\theta},\pi_{\text{old}},y_{j},E)$ by computing the reward using fact-checking for policy update. The GRPO loss is defined as below, with $\beta$ as the KL-regularization factor. We provide implementation details in Appendix [B](https://arxiv.org/html/2505.23912v1#A2).

$$ $\displaystyle\mathcal{L}_{\text{GRPO}}(\theta)=$ $\displaystyle-\mathbb{E}_{(q,E)\sim\mathcal{D},\,\{y_{j}\}_{j=1}^{G}\sim\pi_{ \theta_{\text{old}}}(\mathbf{y}\mid q)}\Bigg{[}\frac{1}{G}\sum_{j=1}^{G}\left( \hat{A}_{j}(\pi_{\theta},\pi_{\text{old}},y_{j},E)-\beta\,\mathbb{D}_{\text{KL }}[\pi_{\theta}\|\pi_{\text{ref}}]\right)\Bigg{]}$ (7) $$

### 4.3 Off-Policy Design

For off-policy RL, we focus on preference learning [52]. To construct the preference-pair data $(q,y_{w},y_{l})\sim\mathcal{D}_{\text{pref}}$, for each query $q$, we need to construct a winning output $y_{w}$ and a losing output $y_{l}$. We first probe the model’s $\pi_{base}$ initial knowledge, using query $q$ from $(q,E)\in\mathcal{D}$ to elicit the initial response from the model. The response $y_{\text{base}}=\pi_{\text{base}}(q)$ only contains factual statements $y_{\text{base}}=\{s_{1},s_{2},\ldots,s_{n}\}$ as we have not taught the model to generate formatted confidence yet. Similarly, we generate the factual correctness score $\mathbf{f}=\texttt{FactCheck}(\mathcal{O},q,E,y_{base})$ and augment the preference-pair dataset $\mathcal{D}_{\text{pref}}$ for off-policy training, as detailed in Algorithm [1](https://arxiv.org/html/2505.23912v1#alg1).

Figure: Algorithm 1 Generating Preference Pair Dataset via Fact-Checked Confidence Scores

For off-policy methods, we use DPO and ORPO to demonstrate the implementation for preference based training [49, 50]. For DPO algorithm, we first finetune the original model for format following on $y_{w}$ with SFT only, to acquire $\pi_{\text{SFT}}$. We then perform training with the standard DPO objective as below, with $\beta$ to regularize the model’s behaviour with respect to the reference model $\pi_{\text{SFT}}$.

$$ $\mathcal{L}_{\text{DPO}}(\theta)=-\mathbb{E}_{(q,y_{w},y_{l})\sim\mathcal{D}_{ \text{pref}}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}\mid q)}{ \pi_{\text{SFT}}(y_{w}\mid q)}-\beta\log\frac{\pi_{\theta}(y_{l}\mid q)}{\pi_{ \text{SFT}}(y_{l}\mid q)}\right)\right]$ (8) $$

The ORPO algorithm updates output preferences jointly with standard SFT, using the following objective, where $\lambda$ is a weighting hyperparameter. Notably, this method does not require a reference model.

$$ $\mathcal{L}_{\text{ORPO}}(\theta)=\mathbb{E}_{(q,y_{w},y_{l})\sim\mathcal{D}_{ \text{pref}}}\left[\mathcal{L}_{\text{SFT}}(\theta)+\lambda\cdot\left(-\log \sigma\left(\log\frac{\textbf{odds}_{\theta}(y_{w}\mid q)}{\textbf{odds}_{ \theta}(y_{l}\mid q)}\right)\right)\right],\text{where}$ (9) $$

$$ $\textbf{odds}_{\theta}(y\mid q)=\frac{P_{\theta}(y\mid q)}{1-P_{\theta}(y\mid q)}$ (10) $$

## 5 Experiments

### 5.1 Experiments Setup

Datasets. We use three datasets for evaluation. Among them, we split WildHallucination (WildHallu) for training and testing, while the other two datasets are used for testing only: (1) WildHallu: It contains 7919 entities mined from user-chatbot conversations collected in the wild. We divide the original dataset [53] into training, development, and test sets with a 8:1:1 ratio. (2) Bios: It consists of 183 human-annotated entities related to people on Wikipedia from FActScore [54], covering a wide range of popularity levels. It has been widely used for evaluating both long-form factuality and uncertainty [6, 7, 13]. (3) PopQA [55]: Following Jiang et al. [13], we use the long-form version of PopQA, which comprises entities across diverse topics such as people, cities, movies, and companies.

Fact-checking. Both Bios and PopQA provide corresponding Wikipedia pages as evidence. For WildHallu, the dataset authors provide the top-10 Google Search results for each entity. During fact-checking, we input the content to be verified alongside the collected evidence, following the pipelines described in [6, 53, 54]. Specifically, we use GPT-4o to obtain more accurate judgments. We conduct additional human annotation to double check this pipeline in Appendix [C](https://arxiv.org/html/2505.23912v1#A3). The detailed prompting strategy is provided in the Appendix [J](https://arxiv.org/html/2505.23912v1#A10).

**Table 1: Free-form tagging results using Llama3-8B-Instruct. The top three results outperforming LUQ are highlighted in cyan, with deeper shades indicating better performance. All values are presented as percentages.**
|  | WildHallu | Bios | PopQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ |
| Literature SOTA |  |  |  |  |  |  |  |  |  |
| LUQ | 14.5 | 21.5 | 56.8 | 20.0 | 29.5 | 63.8 | 16.7 | 23.2 | 62.5 |
| Our Methods |  |  |  |  |  |  |  |  |  |
| LoVeC-SFT | 8.9 | 15.1 | 58.8 | 16.6 | 26.1 | 58.9 | 19.4 | 27.8 | 52.6 |
| LoVeC-GRPO | 6.0 | 8.2 | 63.1 | 10.1 | 11.1 | 68.7 | 10.1 | 5.1 | 63.0 |
| LoVeC-DPO | 6.3 | 5.4 | 62.1 | 9.2 | 6.1 | 67.4 | 10.3 | 4.0 | 62.6 |
| LoVeC-ORPO | 7.7 | 7.3 | 56.5 | 10.4 | 7.1 | 60.6 | 14.9 | 12.1 | 45.8 |

**Table 2: Iterative tagging results using Llama3-8B-Instruct. The top three results outperforming LUQ are highlighted in cyan, with deeper shades indicating better performance. All values are presented as percentages.**
|  | WildHallu | Bios | PopQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ |
| Literature SOTA |  |  |  |  |  |  |  |  |  |
| LUQ | 14.5 | 21.5 | 56.8 | 20.0 | 29.5 | 63.8 | 16.7 | 23.2 | 62.5 |
| Baseline Methods |  |  |  |  |  |  |  |  |  |
| Vanilla | 10.8 | 6.0 | 9.1 | 20.9 | 24.1 | 1.2 | 21.7 | 23.7 | 4.9 |
| p(true) | 23.8 | 23.6 | 15.8 | 19.7 | 28.6 | 17.3 | 19.9 | 24.3 | 23.1 |
| Verb-Conf | 20.3 | 22.1 | 13.4 | 21.2 | 25.3 | 10.8 | 18.8 | 22.1 | 18.3 |
| Self-Cons | 16.5 | 24.3 | 47.8 | 20.3 | 26.5 | 58.8 | 17.3 | 21.6 | 56.8 |
| Our Methods |  |  |  |  |  |  |  |  |  |
| LoVeC-SFT | 9.1 | 15.2 | 51.1 | 16.6 | 25.8 | 56.0 | 18.0 | 25.9 | 52.7 |
| LoVeC-GRPO | 5.7 | 2.5 | 57.0 | 8.5 | 4.2 | 64.7 | 11.3 | 6.2 | 62.8 |
| LoVeC-DPO | 6.0 | 5.0 | 60.4 | 9.0 | 7.3 | 65.6 | 9.6 | 1.7 | 63.1 |
| LoVeC-ORPO | 6.8 | 6.9 | 54.9 | 10.7 | 10.1 | 58.5 | 15.7 | 15.8 | 45.0 |

Baselines.
We include the following baselines in our experiments. The selection criteria are as follows: (1) the method should be able to provide confidence scores for each output. Thus we exclude methods that cannot produce a confidence score for each individual response such as Jiang et al. [13] and Kuhn et al. [25]; and (2) it operates at the sentence level, not relying on atomic claim decomposition [15, 16]. Prompt formulations for all baselines are provided in Appendix [J](https://arxiv.org/html/2505.23912v1#A10).

- •
Vanilla: This refers to the original model (e.g., Llama-3-8B-Instruct).
- •
p(true) [56]: We present a sentence to an LLM and ask whether it is factually true or false. The likelihood associated with the "true" label is used as the confidence score. Following [7], we provide additional context to the LLM to address co-reference issues.
- •
Verbalized Confidence (Verb-Conf) [29, 30]: We prompt the LLM to assign a numerical confidence score (ranging from 0 to 10) to a given sentence, reflecting the model’s belief in its factuality. Similar to p(true), we additionally provide the full paragraph as context to the model.
- •
Self-Consistency (Self-Cons) [10]: We generate 10 additional outputs using temperature $T=1$ and compute the agreement between the original output and the sampled outputs. The level of agreement is used as the confidence score.
- •
LUQ [6]: A state-of-the-art (SOTA) uncertainty estimation method specifically designed for long-form QAs. LUQ demonstrates better performance over a range of baselines in short-form uncertainty estimation [14, 25] and is also applied to confidence estimation.

Training Settings.
For the backbone language models, we use Llama-3-8B-Instruct [22] and Gemma-2-9B-It [23]. We first perform one epoch of SFT on $y_{w}$ from the Wildhallu preference dataset for format adherence. For a fair comparison, we subsequently fine-tune each model for one additional epoch using SFT, GRPO, DPO, and ORPO, respectively. More training details are in Appendix [B](https://arxiv.org/html/2505.23912v1#A2).

#### Evaluation Metrics

Both our factuality and confidence scores lie within the range $[0,1]$, making standard binary classification metrics such as AUROC and AUPRC inapplicable. Instead, we adopt the following three metrics:
(1) Brier Score (BS): Originally designed for binary correctness, BS can be extended to continuous correctness labels. It measures the mean squared error between the predicted confidence and the continuous correctness score.
(2) ECE-M [57]: Unlike traditional ECE, ECE-M supports continuous correctness by averaging the calibration error over soft predictions.
(3) Spearman Correlation (SC) [6]: We compute the Spearman rank correlation between predicted confidence scores and factuality scores. SC assesses how well the predicted confidence preserves the ordinal structure of the ground truth.
We use greedy decoding for all methods.

### 5.2 Experimental Results

LoVeC demonstrates substantial improvement on calibration in both freeform and iterative tagging.
As shown in Tables [1](https://arxiv.org/html/2505.23912v1#S5.T1) and  [2](https://arxiv.org/html/2505.23912v1#S5.T2), LoVeC-DPO and LoVeC-GRPO consistently outperform all baselines, including the prior SOTA LUQ, across all evaluation metrics. This trend holds for both Llama-3-8B-Instruct and Gemma-2-9B-It. While SFT alone achieves results comparable to some baselines, applying RL further improves performance, highlighting the necessity of optimizing confidence via RL. LoVeC-ORPO, while trailing LoVeC-DPO and LoVeC-GRPO, still outperforms some baselines. As depicted in Table [7](https://arxiv.org/html/2505.23912v1#A4.T7) and [8](https://arxiv.org/html/2505.23912v1#A4.T8) (Appendix [D](https://arxiv.org/html/2505.23912v1#A4)), by averaging sentence-level confidence and factuality over generated passage, the results exhibit consistent trends in passage-level.

Figure: Figure 3: Running-time Comparison

LoVeC is highly efficient on test-time.
Our method offers the best test-time efficiency. Confidence scores are generated inline with the answer, requiring no additional sampling or decomposition of responses into atomic claims via external API calls. In contrast, existing state-of-the-art sampling-based methods—such as LUQ for long-form generation—incur significant overhead due to repeated sampling and similarity computations. As depicted in Figure [3](https://arxiv.org/html/2505.23912v1#S5.F3) our method completes the inference on Wildhallu test set (792 instances) 20 times faster than existing SOTA LUQ on free-form tagging.

LoVeC generalizes well across domains and short-form QA.
Tables [1](https://arxiv.org/html/2505.23912v1#S5.T1) and [2](https://arxiv.org/html/2505.23912v1#S5.T2) show that LoVeC generalizes effectively to diverse datasets such as Bios and PopQA.
To assess cross-format transfer, we test the model’s ability to adapt to short-form confidence estimation using the TriviaQA dataset [58], a benchmark for short-form QA. As shown in Table [18](https://arxiv.org/html/2505.23912v1#A7.T18) (Appendix [G](https://arxiv.org/html/2505.23912v1#A7)), our RL-trained models achieve competitive ECE and AUROC scores compared to the baselines. Notably, LoVeC approaches the performance of the state-of-the-art RL-based method, RewardingDoubt [40], despite being trained on significantly less and fully out-of-domain data. More details are in Appendix [G](https://arxiv.org/html/2505.23912v1#A7). Overall, the results highlight the robustness and transferability of LoVeC across both domains and task formats.

**Table 3: Generation Statistics for Llama-3 on Wildhallu Dataset.**
| Model | Word Count | Factuality |
| --- | --- | --- |
| Vanilla | 134.29 | 0.72 |
| LoVeC-SFT | 132.31 | 0.73 |
| LoVeC-GRPO | 130.62 | 0.73 |
| LoVeC-DPO | 133.50 | 0.74 |
| LoVeC-ORPO | 136.49 | 0.73 |

LoVeC preserves response length and overall factuality.
In the freeform tagging, our RL-trained models may produce different content compared to the vanilla untuned model. Table [3](https://arxiv.org/html/2505.23912v1#S5.T3) compares the generation lengths and factuality. LoVeC maintains both response length and factual accuracy, confirming that our calibration improvements do not compromise informativeness. Full table for other datasets is in Appendix [D](https://arxiv.org/html/2505.23912v1#A4).

## 6 Analysis

**Table 4: Top 15 tokens for different models under confident and uncertain conditions. Unrenderable characters are shown as [?], and spaces are explicitly displayed as ␣. In Confident, we examine the case that the model is confident about its prediction; Uncertain refers to the opposite case. Models are required to predict the confidence score token. The detailed prompt can be found in Appendix [F](https://arxiv.org/html/2505.23912v1#A6).**
| Model | Top 15 Tokens |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Confident Case: King’s College, Cambridge is a constituent college … and most prestigious universities. <confidence> |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GRPO | 10 | 9 | 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | ␣ | 11 | 90 | 99 |
| DPO | 10 | 9 | 8 | 7 | 11 | 6 | 5 | [?] | 09 | ␣tenth | [?] | 12 | 4 | [?] | ␣ten |
| SFT | 10 | 0 | 1 | 4 | 8 | 2 | 3 | 7 | 11 | 9 | 5 | 6 | 12 | ␣X | X |
| Uncertain Case: MiniGPT4 is a lightweight and efficient variant of … in resource-constrained environments.<br><confidence> |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GRPO | 2 | 3 | 4 | 5 | 1 | 6 | 0 | 7 | 8 | 9 | 10 | ␣ | 30 | 20 | 60 |
| DPO | 2 | 3 | 4 | 5 | 6 | 1 | 7 | 0 | 8 | 9 | 10 | [?] | ␣five | four | ␣four |
| SFT | 0 | 10 | 1 | 4 | 2 | 3 | 8 | 7 | 5 | 6 | 9 | 11 | 12 | 13 | 14 |

RL Methods Comparison.
Among the methods compared, we find that while ORPO occasionally achieves good calibration, it is generally less stable than GRPO and DPO. This instability is likely due to the absence of a reference model for regularization in ORPO. In contrast, both GRPO and DPO consistently deliver strong and reliable performance. GRPO, although more computationally intensive due to its reliance on a reward model, offers additional benefits. It enables explicit modeling of ordinal relationships between confidence scores and improves numerical stability—advantages we further elaborate on below.

Numerical Consistency. We uncover an interesting property by analyzing the top 15 most probable confidence tokens produced by each model. In RL-trained models, particularly GRPO, we observe that the token probabilities are well-ordered, indicating an understanding of the ordinal structure of the confidence scale. As shown in Table [4](https://arxiv.org/html/2505.23912v1#S6.T4) and Figure [5](https://arxiv.org/html/2505.23912v1#A6.F5) (Appendix [F](https://arxiv.org/html/2505.23912v1#A6)), higher confidence values (e.g., $10$, $9$, $8$) are consistently ranked above lower ones when the model is confident in its output. This behavior suggests that GRPO learns a meaningful internal representation of confidence as an ordered quantity. We attribute this to the explicit alignment between predicted confidence and factuality in the GRPO reward function, which reinforces ordinal structure during training. Notably, this pattern also persists in uncertain predictions. As shown in Figure [6](https://arxiv.org/html/2505.23912v1#A6.F6)((Appendix [F](https://arxiv.org/html/2505.23912v1#A6)), when GRPO predicts $2$ as the most likely confidence score, higher confidence tokens appear in ascending order and lower confidence in descending order, forming a well-structured distribution centered around the model’s belief. In contrast, DPO-trained models exhibit some ordering, sometimes disrupted by irrelevant tokens, due to the lack of format supervision and weaker ordinal constraints in its preference modeling. SFT-based models perform the worst in this regard. For example, in the confidence case, though SFT outputs $10$ as the top prediction, the following tokens often lack any meaningful ordering, with values like 0 0 appearing anomalously high. Such lack of order contributes to poorer calibration.

Ablating the Oracle Model. For DPO and ORPO, we initially employ GPT-4o as an oracle model to generate preference pairs based on factuality comparisons. To assess the necessity of this external supervision, we perform an ablation study by replacing GPT-4o with a self-labeling setup. For instance, Llama-3-8B-Instruct generated outputs are fact-checked using a frozen copy of itself. Our GRPO pipeline is oracle-free by design, as generating GPT-4o labels online during training is prohibitively expensive. As shown in Appendix [D.4](https://arxiv.org/html/2505.23912v1#A4.SS4), Table [10](https://arxiv.org/html/2505.23912v1#A4.T10), DPO trained with self-generated labels performs slightly worse than those using GPT-4o, but still outperforms the strongest baseline, LUQ. This suggests that oracle-free training is both effective and more practical. The success of self-labeling highlights the potential for scalability in settings where external oracle models are unavailable.

Different Reward Functions. For GRPO, we additionally investigate the effect of different reward functions for training. Apart from the log-based reward in Equation [6](https://arxiv.org/html/2505.23912v1#S4.E6), we also experiment with linear and quadratic reward variants based on the absolute and squared difference between predicted confidence and correctness. As detailed in Table [11](https://arxiv.org/html/2505.23912v1#A4.T11), while all reward functions encourage alignment between confidence and correctness, we find the log-based reward to be more effective, as it is a proper scoring rule that sharply penalizes overconfident errors and better supports calibration.

## 7 Conclusion

We introduce LoVeC, a reinforcement learning method to improve confidence estimation in long-form factual text generation. Our approach achieves SOTA performance in both confidence calibration and runtime efficiency. Our results also demonstrate that RL enables more consistent and interpretable confidence predictions. Further analysis shows strong generalization and scalability of our model to out-of-domain datasets and short-form confidence estimations. The results highlight the potential of our framework for deployment in risk-sensitive and high-stakes domains, or general LLM use cases, where hallucination detection is crucial for trust and usability.

## References

- Touvron et al. [2023]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas
Scialom.
Llama 2: Open foundation and fine-tuned chat models, 2023.
- Jiang et al. [2023]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.
Mistral 7B, 2023.
- OpenAI [2022]
OpenAI.
Chatgpt blog post.
[https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt), 2022.
Accessed: 2024-09-06.
- Zhang et al. [2023]
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al.
Siren’s song in the ai ocean: a survey on hallucination in large language models.
*ArXiv preprint*, abs/2309.01219, 2023.
URL [https://arxiv.org/abs/2309.01219](https://arxiv.org/abs/2309.01219).
- Huang et al. [2023]
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu.
A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023.
- Zhang et al. [2024a]
Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier.
LUQ: Long-text uncertainty quantification for LLMs.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 5244–5262, Miami, Florida, USA, November 2024a. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.299.
URL [https://aclanthology.org/2024.emnlp-main.299/](https://aclanthology.org/2024.emnlp-main.299/).
- Zhang et al. [2024b]
Caiqi Zhang, Ruihan Yang, Zhisong Zhang, Xinting Huang, Sen Yang, Dong Yu, and Nigel Collier.
Atomic calibration of llms in long-form generations, 2024b.
- Yang et al. [2024]
Ruihan Yang, Caiqi Zhang, Zhisong Zhang, Xinting Huang, Sen Yang, Nigel Collier, Dong Yu, and Deqing Yang.
Logu: Long-form generation with uncertainty expressions, 2024.
URL [https://arxiv.org/abs/2410.14309](https://arxiv.org/abs/2410.14309).
- Yang et al. [2025]
Ruihan Yang, Caiqi Zhang, Zhisong Zhang, Xinting Huang, Dong Yu, Nigel Collier, and Deqing Yang.
Uncle: Uncertainty expressions in long-form generation, 2025.
- Manakul et al. [2023]
Potsawee Manakul, Adian Liusie, and Mark Gales.
SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models.
In Houda Bouamor, Juan Pino, and Kalika Bali, editors, *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 9004–9017, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.557.
URL [https://aclanthology.org/2023.emnlp-main.557/](https://aclanthology.org/2023.emnlp-main.557/).
- Ding et al. [2024]
Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng.
Retrieve only when it needs: Adaptive retrieval augmentation for hallucination mitigation in large language models.
*arXiv preprint arXiv:2402.10612*, 2024.
- Huang et al. [2024a]
Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhingra.
Calibrating long-form generations from large language models, 2024a.
- Jiang et al. [2024]
Mingjian Jiang, Yangjun Ruan, Prasanna Sattigeri, Salim Roukos, and Tatsunori Hashimoto.
Graph-based uncertainty metrics for long-form language model outputs.
*ArXiv preprint*, abs/2410.20783, 2024.
URL [https://arxiv.org/abs/2410.20783](https://arxiv.org/abs/2410.20783).
- Lin et al. [2023]
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun.
Generating with confidence: Uncertainty quantification for black-box large language models, 2023.
- Fadeeva et al. [2024]
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov.
Fact-checking the output of large language models via token-level uncertainty quantification.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, *Findings of the Association for Computational Linguistics: ACL 2024*, pages 9367–9385, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-acl.558.
URL [https://aclanthology.org/2024.findings-acl.558/](https://aclanthology.org/2024.findings-acl.558/).
- Liu et al. [2024]
Xin Liu, Muhammad Khalifa, and Lu Wang.
Litcab: Lightweight language model calibration over short- and long-form responses.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=jH67LHVOIO](https://openreview.net/forum?id=jH67LHVOIO).
- OpenAI [2023]
OpenAI.
Gpt-4 technical report, 2023.
URL [https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774).
- Rafailov et al. [2023a]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, *Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023*, 2023a.
URL [http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html).
- Cao et al. [2024]
Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li.
Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods.
*IEEE Transactions on Neural Networks and Learning Systems*, 2024.
- Lee et al. [2023]
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al.
Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback.
*arXiv preprint arXiv:2309.00267*, 2023.
- Kirk et al. [2023]
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu.
Understanding the effects of rlhf on llm generalisation and diversity.
*arXiv preprint arXiv:2310.06452*, 2023.
- Meta [2024]
Meta.
Llama 3 model card.
2024.
URL [https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md).
- Team et al. [2024]
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al.
Gemma 2: Improving open language models at a practical size.
*arXiv preprint arXiv:2408.00118*, 2024.
- Murray and Chiang [2018]
Kenton Murray and David Chiang.
Correcting length bias in neural machine translation.
In Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor, editors, *Proceedings of the Third Conference on Machine Translation: Research Papers*, pages 212–223, Brussels, Belgium, October 2018. Association for Computational Linguistics.
doi: 10.18653/v1/W18-6322.
URL [https://aclanthology.org/W18-6322/](https://aclanthology.org/W18-6322/).
- Kuhn et al. [2023]
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar.
Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.
In *The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023*. OpenReview.net, 2023.
URL [https://openreview.net/pdf?id=VD-AYtP0dve](https://openreview.net/pdf?id=VD-AYtP0dve).
- Vazhentsev et al. [2023]
Artem Vazhentsev, Akim Tsvigun, Roman Vashurin, Sergey Petrakov, Daniil Vasilev, Maxim Panov, Alexander Panchenko, and Artem Shelmanov.
Efficient out-of-domain detection for sequence to sequence models.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, *Findings of the Association for Computational Linguistics: ACL 2023*, pages 1430–1454, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-acl.93.
URL [https://aclanthology.org/2023.findings-acl.93/](https://aclanthology.org/2023.findings-acl.93/).
- Duan et al. [2023]
Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu.
Shifting attention to relevance: Towards the uncertainty estimation of large language models.
*ArXiv preprint*, abs/2307.01379, 2023.
URL [https://arxiv.org/abs/2307.01379](https://arxiv.org/abs/2307.01379).
- Zhu et al. [2023]
Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao.
On the calibration of large language models and alignment.
In Houda Bouamor, Juan Pino, and Kalika Bali, editors, *Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 9778–9795, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-emnlp.654.
URL [https://aclanthology.org/2023.findings-emnlp.654/](https://aclanthology.org/2023.findings-emnlp.654/).
- Xiong et al. [2024]
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi.
Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=gjeQKFxFpZ](https://openreview.net/forum?id=gjeQKFxFpZ).
- Tian et al. [2023]
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning.
Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.
In Houda Bouamor, Juan Pino, and Kalika Bali, editors, *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 5433–5442, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.330.
URL [https://aclanthology.org/2023.emnlp-main.330/](https://aclanthology.org/2023.emnlp-main.330/).
- Ulmer et al. [2024]
Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Oh.
Calibrating large language models using their generations only.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 15440–15459, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.824.
URL [https://aclanthology.org/2024.acl-long.824/](https://aclanthology.org/2024.acl-long.824/).
- Mohri and Hashimoto [2024]
Christopher Mohri and Tatsunori Hashimoto.
Language models with conformal factuality guarantees.
In *Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=uYISs2tpwP](https://openreview.net/forum?id=uYISs2tpwP).
- Cheng et al. [2024]
Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu.
Can AI assistants know what they don’t know?
In *Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=girxGkdECL](https://openreview.net/forum?id=girxGkdECL).
- Chen et al. [2024]
Lida Chen, Zujie Liang, Xintao Wang, Jiaqing Liang, Yanghua Xiao, Feng Wei, Jinglei Chen, Zhenghong Hao, Bing Han, and Wei Wang.
Teaching large language models to express knowledge boundary from their own signals, 2024.
URL [https://arxiv.org/abs/2406.10881](https://arxiv.org/abs/2406.10881).
- Li et al. [2024]
Jiaqi Li, Yixuan Tang, and Yi Yang.
Know the unknown: An uncertainty-sensitive method for llm instruction tuning, 2024.
URL [https://arxiv.org/abs/2406.10099](https://arxiv.org/abs/2406.10099).
- Lin et al. [2022]
Stephanie Lin, Jacob Hilton, and Owain Evans.
Teaching models to express their uncertainty in words.
*ArXiv preprint*, abs/2205.14334, 2022.
URL [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205.14334).
- Xu et al. [2024]
Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao.
Sayself: Teaching llms to express confidence with self-reflective rationales, 2024.
- Zhang et al. [2024c]
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang.
R-tuning: Instructing large language models to say ‘I don‘t know’.
In Kevin Duh, Helena Gomez, and Steven Bethard, editors, *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pages 7113–7139, Mexico City, Mexico, June 2024c. Association for Computational Linguistics.
doi: 10.18653/v1/2024.naacl-long.394.
URL [https://aclanthology.org/2024.naacl-long.394/](https://aclanthology.org/2024.naacl-long.394/).
- Han et al. [2024]
Haixia Han, Tingyun Li, Shisong Chen, Jie Shi, Chengyu Du, Yanghua Xiao, Jiaqing Liang, and Xin Lin.
Enhancing confidence expression in large language models through learning from past experience, 2024.
URL [https://arxiv.org/abs/2404.10315](https://arxiv.org/abs/2404.10315).
- Stangel et al. [2025]
Paul Stangel, David Bani-Harouni, Chantal Pellegrini, Ege Özsoy, Kamilia Zaripova, Matthias Keicher, and Nassir Navab.
Rewarding doubt: A reinforcement learning approach to confidence calibration of large language models.
*arXiv preprint arXiv:2503.02623*, 2025.
- Band et al. [2024]
Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto.
Linguistic calibration of long-form generations.
In *Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=rJVjQSQ8ye](https://openreview.net/forum?id=rJVjQSQ8ye).
- Ouyang et al. [2022]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
*Advances in neural information processing systems*, 35:27730–27744, 2022.
- Setlur et al. [2025]
Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar.
Scaling test-time compute without verification or rl is suboptimal.
*arXiv preprint arXiv:2502.12118*, 2025.
- Guo et al. [2025]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Zhang et al. [2024d]
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang.
R-tuning: Instructing large language models to say ‘I don’t know’.
In Kevin Duh, Helena Gomez, and Steven Bethard, editors, *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pages 7113–7139, Mexico City, Mexico, 2024d. Association for Computational Linguistics.
URL [https://aclanthology.org/2024.naacl-long.394](https://aclanthology.org/2024.naacl-long.394).
- Shao et al. [2024]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
*URL https://arxiv. org/abs/2402.03300*, 2024.
- Kumar et al. [2024]
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al.
Training language models to self-correct via reinforcement learning.
*arXiv preprint arXiv:2409.12917*, 2024.
- Havrilla et al. [2024]
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu.
Teaching large language models to reason with reinforcement learning.
*arXiv preprint arXiv:2403.04642*, 2024.
- Rafailov et al. [2023b]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in Neural Information Processing Systems*, 36:53728–53741, 2023b.
- Hong et al. [2024]
Jiwoo Hong, Noah Lee, and James Thorne.
Orpo: Monolithic preference optimization without reference model.
*arXiv preprint arXiv:2403.07691*, 2024.
- Schulman et al. [2017]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Christiano et al. [2017]
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.
Deep reinforcement learning from human preferences.
*Advances in neural information processing systems*, 30, 2017.
- Zhao et al. [2024]
Wenting Zhao, Tanya Goyal, Yu Ying Chiu, Liwei Jiang, Benjamin Newman, Abhilasha Ravichander, Khyathi Chandu, Ronan Le Bras, Claire Cardie, Yuntian Deng, and Yejin Choi.
Wildhallucinations: Evaluating long-form factuality in llms with real-world entity queries, 2024.
URL [https://arxiv.org/abs/2407.17468](https://arxiv.org/abs/2407.17468).
- Min et al. [2023]
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.
FActScore: Fine-grained atomic evaluation of factual precision in long form text generation.
In Houda Bouamor, Juan Pino, and Kalika Bali, editors, *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 12076–12100, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.741.
URL [https://aclanthology.org/2023.emnlp-main.741/](https://aclanthology.org/2023.emnlp-main.741/).
- Mallen et al. [2023]
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi.
When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 9802–9822, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.546.
URL [https://aclanthology.org/2023.acl-long.546/](https://aclanthology.org/2023.acl-long.546/).
- Kadavath et al. [2022]
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan.
Language models (mostly) know what they know, 2022.
- Huang et al. [2024b]
Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhingra.
Calibrating long-form generations from large language models.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, *Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 13441–13460, Miami, Florida, USA, November 2024b. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-emnlp.785.
URL [https://aclanthology.org/2024.findings-emnlp.785/](https://aclanthology.org/2024.findings-emnlp.785/).
- Joshi et al. [2017]
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.
TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.
In Regina Barzilay and Min-Yen Kan, editors, *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.
doi: 10.18653/v1/P17-1147.
URL [https://aclanthology.org/P17-1147/](https://aclanthology.org/P17-1147/).
- Hu et al. [2022]
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
In *The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022*. OpenReview.net, 2022.
URL [https://openreview.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9).
- von Werra et al. [2020]
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec.
Trl: Transformer reinforcement learning.
[https://github.com/huggingface/trl](https://github.com/huggingface/trl), 2020.
- Kwon et al. [2023]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the 29th Symposium on Operating Systems Principles*, pages 611–626, 2023.
- Wei et al. [2024]
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le.
Long-form factuality in large language models.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/937ae0e83eb08d2cb8627fe1def8c751-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2024/hash/937ae0e83eb08d2cb8627fe1def8c751-Abstract-Conference.html).

## Appendix A Limitation and Future Work

Our reinforcement learning tuning approach requires access to white-box models, which limits its applicability to black-box settings. Another limitation is our exclusive focus on factuality; this choice is guided by the availability of widely adopted long-form factuality evaluation pipelines in existing research. Future work could explore several directions. First, confidence estimation can be extended to more general long-form generation tasks such as code generation, creative writing, and machine translation. Second, applying our method to high-stakes domains—such as law, healthcare, and finance—represents an important and impactful avenue for future research.

#### Broader Impact

Our work presents potential for enhancing the trustworthiness of large language models in real-world deployments, especially in high-stakes domains such as healthcare, law, and education, by improving sentence-level confidence estimation and reducing hallucinations. The interpretability and efficiency of our method may enable safer AI systems by allowing users to make informed decisions based on model-generated content. However, we recognize that verbalized confidence could be misused—for example, to give unwarranted credibility to inaccurate outputs or manipulate perceived authority. As such, careful deployment and transparency about confidence generation mechanisms are essential to prevent misuse and ensure ethical adoption.

## Appendix B Experiment Details

### B.1 Training Setup

In our experiment we use SFT , GRPO, DPO, and ORPO. We choose them also as an ablation of reward and reference model, with the details in Tabel [5](https://arxiv.org/html/2505.23912v1#A2.T5) below.

**Table 5: Comparison of methods by use of Reward Model and Reference Model**
| Method | Reward Model | Reference Model |
| --- | --- | --- |
| GRPO | Yes | Yes |
| DPO | No | Yes |
| ORPO | No | No |

We design a confidence quantification prompt for instruction-following, which is prepended before each query. However, we observe that the models often fail to generate responses that follow the expected confidence format. Thus, for both of Llama-3-8B-Instruct and Gemma-2-9b-it, we perform 1 epoch of SFT on $(q,y_{w})\sim\mathcal{D}_{\text{pref}}$ for format following on the completion $y_{w}$ only before RL. For GRPO, we use the frozen copy of original model as the reward model for fact-checking. Both DPO and ORPO are using the exact same $\mathcal{D}_{\text{pref}}$.

We use LoRA [59] on q_proj, k_proj, v_proj, o_proj consistently across models and methods to fine-tune $<1\%$ of the model’s parameters. Below are the detailed hyperparameter choices.

### B.2 Infrastructure

We’ve used TRL [60] libraries for training and vLLM [61] libraries for inference. We’ve conducted our experiments on Google Cloud Platform using a2-ultragpu machines with A100 80GB GPUs. We have consumed around 1500 GPU hours for this project. We list the assets we used and their license in Table [6](https://arxiv.org/html/2505.23912v1#A2.T6).

**Table 6: List of external assets used and their licenses.**
| Asset | Category | License |
| --- | --- | --- |
| [TRL v0.15.2](https://github.com/huggingface/trl) | Code | Apache License 2.0 |
| [vLLM v0.7.3](https://github.com/vllm-project/vllm) | Code | Apache License 2.0 |
| [WildHallucinations](https://huggingface.co/datasets/wentingzhao/WildHallucinations) | Dataset | MIT License |
| [Bios](https://github.com/shmsw25/FActScore) | Dataset | MIT License |
| [PopQA](https://github.com/AlexTMallen/adaptive-retrieval) | Dataset | MIT License |
| [TriviaQA](https://github.com/mandarjoshi90/triviaqa) | Dataset | Apache License 2.0 |

### B.3 GRPO Loss Design

Here we provide the full equation of our GRPO loss. For each data point $d=(q,E)$ a dataset $\mathcal{D}=\{d_{1},d_{2},\ldots,d_{N}\}$, we sample a group of output trajectories $\mathbf{y}=\{y_{1},y_{2},\ldots,y_{G}\}$ from the current policy $\pi_{\theta_{\text{old}}}$ and obtain the group reward $\mathbf{r}=\{r_{1},\dots r_{G}\}$. Then we optimize a new policy $\pi_{\theta}$ based on the per-output advantage estimates $\forall y_{j}\in\mathbf{y},\hat{A}_{j}=\frac{r_{j}-\text{mean}(\mathbf{r})}{
\text{std}(\mathbf{r})}$. $\epsilon$ is the clipping factor which helps to stabilize training by preventing excessively large policy updates. $\pi_{ref}$ is the model’s frozen copy for KL-regularization. $t$ denotes the $t^{th}$ token of trajectory $y$.

$$ $\displaystyle\mathcal{L}_{\text{GRPO}}(\theta)=$ $\displaystyle-\mathbb{E}_{q\sim\mathcal{D},\,\{y_{j}\}_{j=1}^{G}\sim\pi_{ \theta_{\text{old}}}(\mathbf{y}\mid q)}\frac{1}{G}\sum_{j=1}^{G}\frac{1}{|y_{j }|}\sum_{t=1}^{|y_{j}|}\Bigg{\{}\min\Bigg{[}\frac{\pi_{\theta}(y_{j,t}\mid q,y _{j,<t})}{\pi_{\theta_{\text{old}}}(y_{j,t}\mid q,y_{j,<t})}\hat{A}_{j,t},$ (11) $\displaystyle\qquad\qquad\text{clip}\left(\frac{\pi_{\theta}(y_{j,t}\mid q,y_{ j,<t})}{\pi_{\theta_{\text{old}}}(y_{j,t}\mid q,y_{j,<t})},1-\epsilon,\ 1+ \epsilon\right)\hat{A}_{j,t}\Bigg{]}-\beta\,\mathbb{D}_{\text{KL}}[\pi_{\theta }\|\pi_{\text{ref}}]\Bigg{\}}$ $$

In our implementation, we applied reward stretching to make sure it is sensitive enough to model’s responses. In order to retain the quality of model’s generation, we additionally added subordinate rewards, $r^{correct}$ represents the total factuality score, judged by reward model. The python implementation of our reward function is below.

[fontsize=, mathescape, linenos]python
def improved_log_reward(confidence: int, correctness: int,
scale=10.0, gamma=1.5, penalty_strength=5.0):
if confidence is None or not (0 <= confidence <= 10):
return -3 * scale # malformed input penalty

# Core log-likelihood reward
p = np.clip(confidence / 10, 1e-6, 1 - 1e-6)
y = correctness / 10
nll = -(y * math.log(p) + (1 - y) * math.log(1 - p))

best_nll = 0
worst_nll = -(math.log(1e-6) + math.log(1 - 1e-6)) / 2

reward = scale * (1 - (nll - best_nll) / (worst_nll - best_nll))

# Stretch reward to amplify good/bad
reward = np.sign(reward) * (abs(reward) ** gamma)

# Correctness bonus (small)
reward += 0.15 * correctness

return float(reward)

## Appendix C Human Annotation: Reliability of GPT-4o Annotations

Although the paradigm of using GPT+Evidence to fact-check has been widely used in previous work [6, 53, 54, 62], we conduct additional human annotation to evaluate the reliability of using GPT-4o as a fact-checker with retrieved evidence. Two annotators with strong English proficiency and a master’s degree in computer science were recruited. They were instructed to fact-check the sentences using the same prompt provided to GPT-4o. A random sample of 50 instances was drawn from the WildHallu dataset, consisting of 312 sentences in total. We use Spearman correlation as the metric for reliability assessment. The inter-annotator agreement is 0.91 between Annotator 1 and Annotator 2. For the comparison between GPT-4o and the human average, we observe a Spearman correlation of 0.88, indicating a very strong alignment between the model and human judgments.

## Appendix D Additional Results on Llama-3-8B-Instruct

### D.1 Reliability Diagrams

Figure [4](https://arxiv.org/html/2505.23912v1#A4.F4) displays reliability diagrams for the SOTA method LUQ, the vanilla model, LoVeC-SFT, and LoVeC-DPO. A reliability curve closer to the perfect calibration line signifies better calibration. We observe that both the vanilla model and LoVeC-SFT exhibit severe overconfidence. In contrast, our LoVeC-DPO method breaks this overconfidence pattern, leading to improved calibration results.

Figure: (a) LUQ
Refer to caption: x3.png

### D.2 Passage-level Results

We provide passage-level results of Llama-3-8B-Instruct. We simply estimate the passage-level performance by calculating the average of sentence-level confidence and factuality. As shown in the tables below, our method, LoVeC, provides better performance than literature SOTA.

**Table 7: Passage-level iterative tagging results using Llama3-8B-Instruct. The top three results outperforming LUQ are highlighted in cyan, with deeper shades indicating better performance. All values are presented as percentages.**
|  | WildHallu | Bios | PopQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ |
| Literatrue SOTA |  |  |  |  |  |  |  |  |  |
| LUQ | 8.0 | 19.1 | 70.5 | 12.4 | 28.1 | 75.3 | 9.8 | 21.9 | 73.1 |
| Baseline Methods |  |  |  |  |  |  |  |  |  |
| Vanilla | 8.4 | 7.1 | 30.0 | 17.4 | 26.0 | 18.5 | 18.9 | 26.0 | 18.3 |
| p(true) | 15.4 | 19.9 | 27.6 | 16.2 | 23.8 | 29.4 | 17.3 | 19.6 | 34.1 |
| Verb-Conf | 17.9 | 20.3 | 23.8 | 17.7 | 20.1 | 22.7 | 16.4 | 16.8 | 25.4 |
| Self-Cons | 12.1 | 17.4 | 59.2 | 18.3 | 21.2 | 64.7 | 14.7 | 17.1 | 61.3 |
| Our Methods |  |  |  |  |  |  |  |  |  |
| LoVeC-SFT | 6.8 | 15.5 | 54.0 | 13.1 | 26.6 | 63.9 | 14.4 | 26.3 | 60.5 |
| LoVeC-GRPO | 3.3 | 2.7 | 72.5 | 5.0 | 5.0 | 77.2 | 7.7 | 7.5 | 75.1 |
| LoVeC-DPO | 3.5 | 5.5 | 73.1 | 5.2 | 7.0 | 78.3 | 6.1 | 5.5 | 74.1 |
| LoVeC-ORPO | 4.1 | 6.1 | 62.2 | 6.5 | 9.2 | 70.4 | 11.4 | 16.7 | 67.7 |

**Table 8: Sentence-level free-form tagging results using Llama3-8B-Instruct. The top three results outperforming LUQ are highlighted in cyan, with deeper shades indicating better performance. All values are presented as percentages.**
|  | WildHallu | Bios | PopQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ |
| Literature SOTA |  |  |  |  |  |  |  |  |  |
| LUQ | 14.5 | 21.5 | 56.8 | 20.0 | 29.5 | 63.8 | 16.7 | 23.2 | 62.5 |
| Our Methods |  |  |  |  |  |  |  |  |  |
| LoVeC-SFT | 6.4 | 15.2 | 60.6 | 12.4 | 25.6 | 66.1 | 15.4 | 27.1 | 59.5 |
| LoVeC-GRPO | 3.8 | 8.0 | 73.0 | 5.8 | 10.8 | 81.5 | 6.4 | 5.3 | 73.6 |
| LoVeC-DPO | 3.6 | 5.6 | 73.1 | 5.3 | 6.9 | 78.1 | 6.5 | 3.9 | 73.2 |
| LoVeC-ORPO | 4.3 | 3.7 | 68.5 | 5.0 | 3.0 | 76.1 | 9.2 | 9.7 | 64.1 |

### D.3 Post-Training Informativeness

To ensure that our fine-tuned models retain the quality of content generation after training, we evaluate their informativeness using the outputs of Llama-3-8B-Instruct models. Specifically, we assess whether models maintain similar levels of word count and factual accuracy post-training. As shown in Table [9](https://arxiv.org/html/2505.23912v1#A4.T9), all fine-tuned variants produce outputs with comparable length and factuality to the base model. This indicates that the models have not exploited the reward function by generating shorter or less informative responses. Manual inspection of model outputs further supports this observation, suggesting that informativeness is preserved across training and that no degenerate reward-hacking behavior occurs.

**Table 9: Llama-3-8B-Instruct: Average Word Count and Factuality Across Datasets**
| Model | Wildhallu | Bios | PopQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Word Count | Factuality | Word Count | Factuality | Word Count | Factuality |
| Vanilla | 134.29 | 0.72 | 136.14 | 0.55 | 135.91 | 0.56 |
| LoVeC-SFT | 132.31 | 0.73 | 135.53 | 0.54 | 131.67 | 0.54 |
| LoVeC-GRPO | 130.62 | 0.73 | 130.83 | 0.54 | 131.22 | 0.54 |
| LoVeC-DPO | 133.50 | 0.74 | 130.86 | 0.54 | 129.67 | 0.55 |
| LoVeC-ORPO | 136.49 | 0.73 | 133.81 | 0.52 | 135.41 | 0.53 |

### D.4 Ablating the Oracle Model

**Table 10: Comparison of WildHallu results for LUQ, SFT, ORPO, DPO, and GRPO across tagging strategies. All values are presented as percentages.**
| Method | Iterative Tagging | Freeform Tagging |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | BS | ECE-M | SC | BS | ECE-M | SC |
| LUQ | 14.5 | 21.5 | 56.8 | 14.5 | 21.5 | 56.8 |
| LoVeC-GRPO | 5.7 | 2.5 | 57.0 | 6.0 | 8.2 | 63.1 |
| Fact-Checking with GPT-4o + Evidence |  |  |  |  |  |  |
| LoVeC-SFT | 9.1 | 15.2 | 51.1 | 8.9 | 15.1 | 58.8 |
| LoVeC-DPO | 6.0 | 5.0 | 60.4 | 6.3 | 5.4 | 62.1 |
| LoVeC-ORPO | 6.8 | 6.9 | 54.9 | 7.7 | 7.3 | 56.5 |
| Fact-Checking with Llama3-8B + Evidence |  |  |  |  |  |  |
| LoVeC-SFT | 8.2 | 9.0 | 49.6 | 8.3 | 9.5 | 58.3 |
| LoVeC-DPO | 7.2 | 9.8 | 58.0 | 7.1 | 7.8 | 60.4 |
| LoVeC-ORPO | 7.5 | 8.1 | 53.2 | 8.0 | 8.1 | 55.1 |

To assess the necessity of using a high-capacity oracle model, we conduct an ablation study by replacing GPT-4o with Llama-3-8B-Instruct for generating preference datasets. Specifically, instead of relying on GPT-4o for fact-checking and labeling preference pairs, we use the training model itself to self-label its outputs prior to DPO training.

As shown in Table [10](https://arxiv.org/html/2505.23912v1#A4.T10), while models trained on self-labeled data perform slightly worse than those using GPT-4o supervision, they still surpass strong baselines. Notably, LoVeC-DPO trained with self-labeling continues to outperform the previous state-of-the-art method, LUQ. This result highlights the practicality and effectiveness of oracle-free training, making the approach more accessible and cost-efficient without significantly compromising performance.

### D.5 The Selection of Reward Function

**Table 11: Comparison of WildHallu results for LUQ, GRPO-log, GRPO-linear, GRPO-quadratic. All values are presented as percentages.**
| Method | Iterative Tagging | Freeform Tagging |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | BS | ECE-M | SC | BS | ECE-M | SC |
| LUQ | 14.5 | 21.5 | 56.8 | 14.5 | 21.5 | 56.8 |
| GRPO-log | 5.7 | 2.5 | 57.0 | 6.0 | 8.2 | 63.1 |
| GRPO-quadratic | 7.0 | 8.7 | 55.1 | 7.3 | 9.3 | 62.3 |
| GRPO-linear | 8.5 | 10.8 | 54.3 | 8.2 | 10.4 | 59.8 |

## Appendix E Gemma-2-9B-It Results

Here we provide results for Gemma-2-9B-It. As described in the tables below, our method shows consistent improvements across models, beating the literature SOTA, LUQ, across datasets.

**Table 12: Sentence-level iterative tagging results using Gemma-2-9B-It. The top three results outperforming LUQ are highlighted in cyan, with deeper shades indicating better performance. All values are presented as percentages.**
|  | WildHallu | Bios | PopQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ |
| Literature SOTA |  |  |  |  |  |  |  |  |  |
| luq | 11.9 | 16.3 | 50.0 | 12.2 | 15.5 | 69.2 | 13.6 | 15.1 | 62.6 |
| Our Methods |  |  |  |  |  |  |  |  |  |
| Vanilla | 22.5 | 26.3 | 28.9 | 24.5 | 28.8 | 35.5 | 24.0 | 27.7 | 23.7 |
| p(true) | 19.3 | 22.8 | 25.4 | 21.0 | 25.0 | 31.0 | 21.5 | 24.5 | 26.0 |
| Verb-Conf | 18.5 | 19.2 | 35.1 | 18.0 | 19.0 | 39.5 | 19.5 | 20.0 | 36.2 |
| Self-Cons | 13.4 | 17.7 | 43.2 | 13.0 | 17.0 | 53.0 | 13.5 | 16.5 | 48.5 |
| Our Methods |  |  |  |  |  |  |  |  |  |
| LoVeC-SFT | 8.0 | 12.2 | 36.1 | 18.8 | 25.1 | 54.9 | 25.8 | 32.6 | 37.1 |
| LoVeC-GRPO | 7.3 | 5.6 | 52.2 | 10.7 | 11.1 | 72.4 | 13.1 | 18.1 | 64.2 |
| LoVeC-DPO | 4.1 | 1.3 | 51.8 | 7.5 | 7.3 | 75.2 | 11.6 | 13.2 | 65.3 |
| LoVeC-ORPO | 5.2 | 7.0 | 46.3 | 10.1 | 11.4 | 67.7 | 13.8 | 15.6 | 56.9 |

**Table 13: Passage-level iterative tagging results using Gemma-2-9B-It. The top three results outperforming LUQ are highlighted in cyan, with deeper shades indicating better performance. All values are presented as percentages.**
|  | WildHallu | Bios | PopQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ |
| Literature SOTA |  |  |  |  |  |  |  |  |  |
| LUQ | 6.3 | 14.1 | 61.2 | 6.8 | 14.2 | 81.6 | 9.2 | 11.8 | 73.8 |
| Baseline Methods |  |  |  |  |  |  |  |  |  |
| Vanilla | 19.3 | 23.1 | 35.5 | 21.6 | 29.4 | 43.1 | 21.4 | 28.1 | 33.6 |
| p(true) | 17.8 | 20.3 | 28.7 | 19.6 | 24.2 | 33.8 | 20.4 | 23.7 | 30.2 |
| Verb-Conf | 16.3 | 17.1 | 34.6 | 17.5 | 17.9 | 35.7 | 18.4 | 18.8 | 32.3 |
| Self-Cons | 12.2 | 15.4 | 48.9 | 12.9 | 16.1 | 58.1 | 13.4 | 15.2 | 54.4 |
| Our Methods |  |  |  |  |  |  |  |  |  |
| LoVeC-SFT | 6.5 | 12.1 | 31.9 | 16.2 | 25.6 | 55.6 | 23.6 | 32.8 | 38.3 |
| LoVeC-GRPO | 2.9 | 3.6 | 64.4 | 3.4 | 5.7 | 82.5 | 8.6 | 7.5 | 74.3 |
| LoVeC-DPO | 2.5 | 2.9 | 65.9 | 4.6 | 7.8 | 84.3 | 9.0 | 13.2 | 75.4 |
| LoVeC-ORPO | 2.5 | 2.6 | 51.5 | 4.7 | 8.2 | 80.0 | 7.6 | 10.2 | 73.0 |

**Table 14: Sentence-level free-form tagging results using Gemma-2-9B-It. The top three results outperforming LUQ are highlighted in cyan, with deeper shades indicating better performance. All values are presented as percentages.**
|  | WildHallu | Bios | PopQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ |
| Literature SOTA |  |  |  |  |  |  |  |  |  |
| LUQ | 11.9 | 16.3 | 50.0 | 12.2 | 15.5 | 69.2 | 13.6 | 15.1 | 62.6 |
| Our Methods |  |  |  |  |  |  |  |  |  |
| LoVeC-SFT | 7.3 | 11.6 | 57.2 | 11.5 | 17.8 | 70.6 | 19.4 | 26.3 | 50.4 |
| LoVeC-GRPO | 4.3 | 4.6 | 56.1 | 8.3 | 3.7 | 72.6 | 8.5 | 8.9 | 63.5 |
| LoVeC-DPO | 4.5 | 2.2 | 55.2 | 6.6 | 4.2 | 70.3 | 9.5 | 8.2 | 66.6 |
| LoVeC-ORPO | 5.7 | 7.1 | 48.4 | 9.5 | 12.9 | 66.4 | 10.9 | 12.9 | 50.2 |

**Table 15: Passage-level free-form tagging results using Gemma-2-9B-It. The top three results outperforming LUQ are highlighted in cyan, with deeper shades indicating better performance. All values are presented as percentages.**
|  | WildHallu | Bios | PopQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ | BS$\downarrow$ | ECE-M$\downarrow$ | SC$\uparrow$ |
| Literature SOTA |  |  |  |  |  |  |  |  |  |
| LUQ | 6.3 | 14.1 | 61.2 | 6.8 | 14.2 | 81.6 | 9.2 | 11.8 | 73.8 |
| Our Methods |  |  |  |  |  |  |  |  |  |
| LoVeC-SFT | 6.0 | 11.7 | 52.3 | 8.3 | 17.6 | 73.9 | 16.0 | 25.7 | 55.8 |
| LoVeC-GRPO | 3.3 | 4.0 | 66.3 | 3.8 | 4.7 | 83.6 | 6.2 | 4.3 | 78.2 |
| LoVeC-DPO | 2.7 | 3.2 | 65.4 | 3.1 | 4.6 | 83.9 | 6.1 | 7.9 | 77.4 |
| LoVeC-ORPO | 2.5 | 2.8 | 59.2 | 4.3 | 9.2 | 78.1 | 5.2 | 8.8 | 64.5 |

## Appendix F Numerical Consistency

We investigate the probability distribution over decoded confidence tokens to assess whether models have learned to internalize the ordinal structure of the confidence scale. Ideally, a well-calibrated model should rank numerical confidence tokens in an order that reflects their semantic meaning—placing higher probabilities on larger values (e.g., $10$ over $9$, $9$ over $8$, etc.) when expressing high certainty.

To probe this behavior, we prompt each model to assign a confidence score to a factual statement it is highly confident about. We then inspect the top 15 tokens with the highest decoding probabilities. As shown below, all models correctly assign the most probable token as $10$, reflecting high confidence. However, the surrounding distributions reveal key differences.

The GRPO-trained model displays a near-perfect ordinal alignment: tokens are ranked in descending order from $10$ down to 0 0, without the presence of extraneous symbols or irrelevant content. This indicates that GRPO not only learns to express high confidence but also internalizes the structure of the confidence scale. In contrast, the DPO model also shows partial ordinal structure, but includes non-numeric or unrelated tokens among its top predictions. We attribute this to DPO’s lack of explicit format control, whereas GRPO incorporates a format penalty during training to discourage malformed outputs.

SFT, although it outputs $10$ as the most likely token, fails to preserve any consistent ordinal pattern in the rest of the distribution—e.g., lower-confidence values like 0 0 or $1$ may appear above intermediate values. This suggests that SFT does not effectively capture the ordinal relationship between confidence scores, which may contribute to its weaker calibration performance.

Table: Table 16: Example of model’s high confidence output next-token-probability probing.

Figure: Figure 5: Token probability distributions when the model is confident. The sentence to tag is: “King’s College, Cambridge is a constituent college of the University of Cambridge, one of the world’s oldest and most prestigious universities.”
Refer to caption: x7.png

More interestingly, such trend holds when the model is uncertain about their output. It demonstrates a desired concave ranking centered at the most probable token. For example in Figure [6](https://arxiv.org/html/2505.23912v1#A6.F6), for GRPO, any confidence score large than it’s most probable token $2$ is in ascending order, any score smaller in in descending order. Again DPO shows similar pattern but with irrelevant tokens, and SFT fails to grasp the order. These findings reinforce the advantage of reinforcement learning in inducing consistent numerical structure and semantic alignment in confidence estimation.

Table: Table 17: Example for model’s low confidence output for next-token-probability probing.

Figure: Figure 6: Token probability distributions when the model is uncertain. The sentence to tag is: “MiniGPT4 is a lightweight and efficient variant of the popular GPT-4 language model, designed to be more accessible and easier to deploy in resource-constrained environments.”
Refer to caption: x8.png

## Appendix G Generalization to Short-form QA

To assess the generalization capability of our method on short-form confidence estimation, we evaluate it on the TriviaQA [58] test set. As illustrated in Table [18](https://arxiv.org/html/2505.23912v1#A7.T18), our results show that the proposed method performs on par with sampling-based self-consistency baselines, and closely approaches the performance of the current RL-based state-of-the-art, RewardingDoubt.

Notably, both ours method and RewardingDoubt use the same base model, Llama-3-8B-Instruct, and a similar LoRA fine-tuning setup. However, our method is trained on the out-of-domain Wildhallu dataset, using only 5.6k examples for a single epoch. In contrast, RewardingDoubt is trained directly on the in-domain TriviaQA dataset, with 174k examples across two epochs. Despite this disparity in domain alignment and data volume, our model achieves a strong approximation to RewardingDoubt’s performance.

These results highlight the robustness and domain-transferability of our approach. We believe this test provides encouraging evidence that our method generalizes well to short-form QA tasks, and has the potential for further gains with in-domain fine-tuning.

**Table 18: AUROC and ECE metrics for various methods on short-form QA dataset: Trivia QA. All values are presented as percentages.**
| Category | Method | AUROC$\uparrow$ | ECE$\downarrow$ |
| --- | --- | --- | --- |
| Baselines | Self-Verb | 50.0 | 69.3 |
| p(true) | 60.1 | 21.1 |  |
| Self-Cons | 73.4 | 12.2 |  |
| Literature SOTA | Rewarding Doubt* | 85.9 | 2.2 |
| Our Methods | LoVeC-SFT | 56.3 | 2.0 |
| LoVeC-GRPO | 69.2 | 6.3 |  |
| LoVeC-DPO | 71.2 | 6.9 |  |
| LoVeC-ORPO | 70.4 | 8.6 |  |

## Appendix H Running Time

We compare the running time of different confidence estimation methods on Wildhallu test set (792 data points) using Gemma2-9B-It and it’s RL fine-tuned model on single A100 80GB. Note that Iterative Tagging time counts the generation time of no-confidence facts from the original model. As depicted in [7](https://arxiv.org/html/2505.23912v1#A8.F7) below, not only do our methods show better calibration, they also runs $10\sim 20\times$ faster than sampling based methods, including the literature SOTA, LUQ.

Figure: Figure 7: Comparison of total processing time (in seconds) for WildHallu test set (792 samples) using Gemma2-9B-It on single A100 80GB.

## Appendix I Case Study

Table: Table 19: Examples generated by LoVeC-DPO from the health domain. The model’s low-confidence sentences (highlighted in red) indeed correspond to hallucinated or factually incorrect statements.

Table: Table 20: Examples generated by LoVeC-DPO from the legal domain. The model’s low-confidence sentences (highlighted in red) indeed correspond to hallucinated or factually incorrect statements.

## Appendix J Prompts

### J.1 Prompt for SFT and RL-based tuning:

### J.2 Prompt for p(true):

### J.3 Prompt for Verbalized Confidence (VC):

### J.4 Prompt for Fact-Checking:

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - In Table 2 how does the literature sota LUQ perform at par/worse than vanilla baselines?
soundness: 3
strengths: - Simple and effective approach: LoveC applies a straightforward idea by training the model to verbalize numeric confidence inline, achieving strong calibration 

- Strong empirical performance: Both LoveC GRPO and DPO consistently outperform prior methods across multiple datasets and metrics, showing that the approach is robust and generalizable.

- Efficient and practical: The method produces confidence scores quickly during generation, making it efficient 

- Clear and well-written paper: The writing is clear and well-structured and well explained


Overall, this is a promising paper and I'm happy to increase scores if my concerns are adressed
summary: The paper introduces LoveC, a method to make LLMs verbalize their confidence (0–10) sentence-by-sentence during long-form generation, trained with reinforcement learning so that the stated confidence aligns with factual correctness. It targets two common gaps: (i) prior long-form confidence methods are mostly post-hoc and expensive (multi-sample self-consistency, extra claim-extractor models), and (ii) most verbalized confidence work is short-form. LoveC outputs statements and confidence in one decoding pass and is evaluated in two settings: Free-form Tagging (model generates answer + confidence) and Iterative Tagging (given a fixed answer, the model tags each sentence’s confidence). The authors experiment with two variants: LoveC GRPO (on-policy RL) and LoveC DPO (off-policy RL). Experiments on WildHallu, Bios, and PopQA show strong calibration improvements.
weaknesses: - The paper would benefit from additional ablations exploring alternative reward formulations or scaling choices to show how each component contributes to performance.

- It would be interesting to see whether combining LoveC with self-consistency or other sampling-based calibration methods could further improve confidence estimation.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 4
questions: - Currently, the reward is computed by aggregating all confidence scores in the generation. And all confidence tokens receive the same reward. For instance, the confidence score of a sentence might perfectly match the ground truth; however can still receive a negative or a positive reward due to other sentences. Would a more granular reward assignment approach improve the calibration level? I think exploring this direction both improves the contribution/originality of the paper and makes it more interesting.
- During iterative tagging evaluation, the fixed text is generated by the confidence estimaton model, right?
soundness: 3
strengths: - The paper is well-organized, easy to follow, and read
- Uncertainty estimation for long-form generation is an important topic to explore.
- The proposed method is efficient, and this aspect is supported by experimental evidence as well.
summary: This paper proposes LoVeC (Long-form Verbalized Confidence). It is a reinforcement learning framework that trains language models to generate numerical confidence scores alongside long-form factual text. Using on-policy (GRPO) and off-policy (DPO) variants, the model learns to align its self-reported confidence with factual correctness, which avoids costly post-hoc sampling. The authors also introduce two evaluation settings: 1. free-form tagging (confidence generated inline) and 2. iterative tagging (confidence assigned to fixed text). The proposed method is faster than existing works.
weaknesses: - I find the main contribution of the proposal limited. Using existing RL algorithms to train models to generate confidence scores per sentence lacks originality. This is a combination of sentence-level uncertainty estimation and RL fine-tuning for verbalized confidence, which both exist in the literature. While the method is well-executed and may yield practical benefits, it does not introduce a fundamentally new algorithmic component or theoretical insight beyond this combination.
- The only supervised method in the benchmark is the proposed method, which gives a significant advantage. It has been previously shown that fine-tuning the model for p(true) and verbalized confidence improves performance. I think fine-tuned versions of these methods are essential. 
- Does the proposed method yield calibrated confidence scores when we do sampling? Analysis of this is important since temperature sampling is a common practice.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Could one calibrate, e.g. NLL of segments of interest directly with this approach instead of producing verabalized confidence?
2. How does the reward function deal with invalid confidence assignment (e.g. in Tab.3 11,12,... appear)? 
3. How does ablation in paragraph at line 452 relate to the known effect of judge LMs preference for own outputs?
4. Would the method be good at assessing prompt perturbation? I.e. the predicted confidences should be lower for randomly perturbed prompts.
soundness: 2
strengths: The novel setting provides fine grained verbalized confidence estimates and may even be useful in situations where only the output of the model is available without any additional information. RL use for calibration is an interesting approach and the optimization objectives seem somewhat innovative. The authors perform many ablations to determine the best way to fine tune model to produce verbalized confidence scores.
summary: This work proposes a relatively novel setting, in which the model must directly decode its sentence-wise confidence in long-form QA in a verbalized fashion.
The method, LoVeC, uses either DPO or GRPO to perform online or offline fine tuning of the models to assess own uncertainty.
The authors devise a specialised reward that optimizes its calibration through RL. 
This resulted in improved calibration on selected benchmarks.
weaknesses: 0. Typos (not affecting my assessment, just heads up):
    1. Line 461: double full stop.
1. Evaluation:
    1. The evaluation with fact extraction and fact checking pipeline is complex. Only QA setting is considered, but possibly the paper would benefit from evaluation on synthetic problems or somewhat more verifiable domains (see e.g. [4]). 
    2. (major) Several generally accepted strong uncertainty estimation methods (i.e. Perplexity or GNLL of a segment [1][2] or intenal states based methods, i.e. [3]) have not been considered. These methods can be easily applied in this setting, since log probabilities are usually available. 
    3. Somehow "Vanilla" prompting approach is consistently better than Baseline Methods and LUQ on Windhallu. From the description Vanilla and Verb-Conf should be roughly identical. Are the same models used everywhere throughout evaluation?
    4. To follow up the previous point, it appears that the model for LoVeC might be fine tuned directly on WildHallu. Would this be unfair to other methods?

### References:
1. Aichberger, L., Schweighofer, K. & Hochreiter, S. Rethinking Uncertainty Estimation in Natural Language Generation. Preprint at https://doi.org/10.48550/arXiv.2412.15176 (2024).
2. Fadeeva, E. et al. LM-Polygraph: Uncertainty Estimation for Language Models. Preprint at https://doi.org/10.48550/arXiv.2311.07383 (2023).
3. Kossen, J. et al. Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs. Preprint at https://doi.org/10.48550/arXiv.2406.15927 (2024).
4. Ielanskyi, M., Schweighofer, K., Aichberger, L. & Hochreiter, S. Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation. Preprint at https://doi.org/10.48550/arXiv.2510.02279 (2025).

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: My questions mainly concern Weakness 2, specifically the intermediate policy behaviors during online RL training. Providing additional analysis in this area could help address some of my concerns.
soundness: 2
strengths: 1. This paper introduces an RL-based method for generating and calibrating confidence scores in long-form factual text using LLMs.  Extensive experiments demonstrate the effectiveness of the proposed approach across both free-form and iterative tagging settings.

2. The authors thoroughly analyze the proposed method, as well as its baselines, and perform ablation studies on each component, providing a clearer understanding of the model’s design and effectiveness.
summary: This paper proposes LoVeC, an RL–based framework that enables LLMs to generate confidence scores alongside long-form factual statements in one-time generation. It introduces two new evaluation settings, free-form tagging and iterative tagging, to assess confidence calibration in long-form text. Experiments show that LoVeC generalizes robustly across domains and achieves better calibration than traditional self-consistency methods as well as other baselines.
weaknesses: 1. The presentation of the paper needs improvement. For example, in line 258, the authors mention using additional subordinate rewards (e.g., informativeness and format rewards) in Appendix B, yet the appendix does not provide clear details about them. While $r^{\text{correct}}$ is defined as the total factuality score judged by the reward model, the specific details and prompts of these subordinate rewards, which are crucial for the success of the RL algorithm, remain unclear.

2. The authors do not provide intermediate validation or analysis of model behavior during RL training, such as how confidence assignments evolve over time. Since the reward formulation in Eq. (6) appears potentially unstable for online RL training, including intermediate evaluations, such as tracking reward trends or showcasing intermediate model outputs for correctness and confidence, would strengthen the paper’s persuasiveness.