Learning to Reason for Hallucination Span Detection
Authors: Hsuan Su
Ting-Yao Hu
Hema Swetha Koppula
Kundan Krishna
Hadi Pouransari
Cheng-Yu Hsieh
Cem Koc
Joseph Yitan Cheng
Oncel Tuzel
Raviteja Vemulapalli
Venue: ICLR 2026 Poster (2026)
Keywords: Hallucination Detection
Reasoning
Reinforcement Learning
Large Language Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce RL4HS, a span-level RL method for hallucination detection. With CAPO to balance precision–recall, it outperforms supervised and pretrained reasoning models on RAGTruth.

Abstract: Large language models (LLMs) often generate hallucinations---unsupported content that undermines reliability. While most prior works frame hallucination detection as a binary task, many real-world applications require identifying hallucinated spans, which is a multi-step decision making process.
This naturally raises the question of whether explicit reasoning can help the complex task of detecting hallucination spans. 
To answer this question, we first evaluate pretrained models with and without Chain-of-Thought (CoT) reasoning, and show that  CoT reasoning has the potential to generate at least one correct answer when sampled multiple times. Motivated by this, we propose RL4HS, a reinforcement learning framework that incentivizes reasoning with a span-level reward function. 
RL4HS builds on Group Relative Policy Optimization and introduces Class-Aware Policy Optimization to mitigate reward imbalance issue. Experiments on the RAGTruth benchmark (summarization, question answering, data-to-text) show that RL4HS surpasses pretrained reasoning models and supervised fine-tuning, demonstrating the necessity of reinforcement learning with span-level rewards for detecting hallucination spans.

Published as a conference paper at ICLR 2026
LEARNING TO REASON FOR HALLUCINATION SPAN DETECTION
Hsuan S
Ting- Yao Hu
Hema Swetha Koppula
Kundan Krishna Joseph Yitan Cheng
Hadi Pouransari
Cheng- Yu Hsieh
Cem Koc
Oncel Tuzel
Raviteja Vemulapalli
National Taiwan University
Apple
ABSTRACT
Large language models (LLMs) often generate ballucinations Iolluciiatioossuuuuupprrre uusuppporedd con- that undermines reliability. While most prior works frame hallucination de- tection as a binary task, many real-world applications require identifying halluci- nated spans, which a multi-step decision making process. This naturally raises the question of whether explicit reasoning can help the complex task of detecting hallucination spans. To answer this question, we first evaluate pretrained mod- els with and without Chain-of-Thoune Thoouhh (CoT) reasoning, and show that CoT rea- soning has the potential to generate at least one correct answer when sampled multiple times. Motivated by this, propose RL4HS, reinforcement learning framework that incentivizes reasoning with a span-level reward function. RLAHS builds Group Relative Policy Optimization and introduces Class-Aware Policy Optimization to mitigate reward imbalance issue. Experiments on the RAGTruth benchmark (summarization, question answering, data-to-text) show that RL4HS surpasses pretrained reasoning models and supervised line-tuning, demonstrating the necessity of reinforcement learning with span-level rewards for detecting hal- lucination spans.
INTRODUCTION
Over the past few years, Large Language Models (LLMs) have demonstrated remarkable capabili- ties across wide range of natural language processing tasks (Xie al. 2023 Zhang et al., 2023
Gao et al. 2024 OpenAl et al., (202)). However, they are still prone to generating hallucinations content that not supported by the input context the underlying knowledge sources (Zhu al.) 2024 Kalai et al. 2025; Huang et al. 2025). Hallucinations pose critical risks in downstream ap- plications such as summarization and long-form question answering, where reliability and factual consistency with respect to the input context are paramount. Hence, the ability to detect hallucina- tions crucial for successful real-world deployment of LLMS.
Most existing research works focus binary hallucination detection problem, where the goal is to determine if the model output contains hallucinations or not (Yang et al.) 2024a (b) Tang et al. a [024]]
Ravi al. 2024; Ji et al. 2024 Chuang et al.) 2024). While useful, this formulation is limited: in many real-world applications, one often needs to know which specific spans in the model output are
hallucinated in order to assess the reliability of the generated content. This motivates the problem of
hallucination span detection, where the goal is to precisely locate unsupported content in the model
output (Wu et al., 2023 Ogasa & Arase 2025
Recently, reasoning the process of systematically arriving at conclusions by generating and utiliz- ing intermediate steps-has has been shown to significantly enhance the capabilities of LLMMs in solving complex tasks such as mathematics (Shao et al.] 2024) Yu et al. 2025) and coding (Liu & Zhang 2025| Chen et al. 2025). Hallucination span detection also complex multi-step decision making process as it requires carefully analyzing the model output to extract all the stated facts and verifying
"Work done during intersship Apple.

---

Published as a conference paper at ICLR 2026
whether each of these facts is fully supported by the input context, and could benefit significantly from a learned reasoning process.
Some existing hallucination detection works (Luo et al. 2023 Eliav et al. 2025 explored Chain-of- Thought (CoT) prompting, and showed that simple CoT can lead to considerable improvements in binary hallucination detection performance providing motivating evidence explore reasoning for hallucination detection. However, these works do not focus on the fine-grained hallucination span detection problem and they do not explore training a reasoning model for hallucination detection. In this work, we focus on concretely answering the following two research questions: (i) Is learned reasoning process helpful for hallucination span detection? How learn an effective reasoning process for this task? (ii) Is it necessary to learn reasoning process specifically for ballucination span detection or do existing general-domain reasoning models suffice for this specific task?
To answer the first question, we train CoT sassoning-basel hallucination span detection model using Reinforcement Learning (RL). Specifically, we train the model on a dataset labeled with hal- lucination spans using Group Relative Policy Optimization (GRPO; [Shao et al. (2024)) with a re- ward function based on the target span-Fl metric. To the best of our knowledge, this the first work training a reasoning- -baeed hallucination span detection model using RL. The resulting model signif- icantly outperforms a non-reasoning model trained for span detection using Supervised Finetuning (SFT) the same training dataset. This clearly shows that the reasoning process learned using RL is highly beneficial for detecting hallucination spans.
While the reward based span-F1 score is effective, notice that its asymmetric nature over- incentivizes non-hallucination predictions due to the normalization used GRPO advantage cal- culation. To address this issue, we propose a modified version of GRPO, which we refer to as class-aware policy optimization, by introducing a scaling factor for the advantages computed for non-hal balluulucination samples. By using value smaller than one for this scaling factor, we are able to achieve a better balance between hallucination and non- non-hallucination classes leading to an overall higher span-Fl score.
To answer the second question, we evaluate several recent reasoning models that have been trained with data from various domains such as mathematics, coding, tool-calling, etc. Our evaluation results show that, despite being much larger in size, state-of-the-art reasoning models perform sig- nificantly worse than a 7B reasoning model trained specifically for hallucination span detection.
Major contributions: (i) We train a hallucination span detection model using reinforcement learn- ing with span-level reward, and show that the resulting reasoning process improves the hallucina- tion span detection performance by a significant margin when compared non-reasoning model trained with the same dataset. (ii) We show that existing reasoning models perform significantly worse when compared using RL with span-F1 reward. (iii) We identify an issue with span-Fl reward that leads to over- emphasis non-hallucination predictions in the context of GRPO, and propose class-aware policy optimization to address this issue.
reasoning model specifically trained for hallucination span detection
HALLUCINATION SPAN DETECTION
2.1 TASK
This paper focuses on the task of hallucination span detection in the context of Conditional Natu- ral Language Generation (CNLG) tasks such as summarization and long-form question answering. Given the input context and the generated response y (y1 32...gr) consisting of T characters, the goal to identify all the hallucinated spans, which are text segments in y that are not supported by Each hallucinated span s is represented using its start and end positions in y.
2.2 MODEL
Existing works on hallucination span detection train either a decoder- based generative model that directly outputs hallucinated content list of text segments (Wu et al. (023) or encoder-based discriminative model that performs token-level binary classification (Ogasa & Arase (025). While generative models are natural fit for exploring CoT reasoning, it is unclear how reasoning can
2

---

Published as a conference paper at ICLR 2026
CoT
w CoT
Summarization
Answerinnn
Data-to-Text
Qwen2.5 5.78
Spun--11100
Span-k 100K
9.5
Spam-P
94
10
65
118
16
190
20
49
Number of predictions K
Number ofreddcctioss K
Number of predictions K Dutu-to-Text
Summarization
Question Amwering
Qwen3- 88
1p
Spin-F1 18K
8.3
Span-P1
0z
04
03
Number of predictions K
Number of prediictions K
Number of predictions
Figure Span-F1 @K for different number of predictions K. Using CoT reasoning provides significant boost as K increases clearly demonstrating the potential of CoT reasoning.
be incorporated into token-level binary classifiers. Hence, in this work, we follow the generative modeling approach of Wu et al. (222)) and train LLM directly output list of hallucinated text segments. For each predicted text segment, we get the corresponding span start and end index in by searching for matching content.
2.3 EVALUATION METRIC
For comparing model predictions with groundtruth, use the dataset-level span-F1 metric defined
in Wu et al. 2023). Given the groundtruth spans S = {sm = [imm Jmm] ]] and the predicted spans
{sa=
[n. jn 1 the span-F1 metric is computed using
F1 2 Precision Recall
PP n G PP|
PPggl |g]
Precision
Recall
(1)
Precision + Recall
M
where U 8m and P = U
8n- Here, denotes set union, n denotes set intersection, |-|
m=1 denotes set cardinality, and [i denotes the set of integers from to j.
n=1
RL4HS: REINFORCEMENT LEARNING FOR HALLUCINATION SPAN DETECTION
3.1 MOTIVATING RL WITH DIVERSE CoT REASONING
central question in this study is whether explicit reasoning beneficial for identifying hallu- cination spans. As preliminary experiment, evaluated Qwen2. 5-7B TTeam 2024 and Qwen3-8BB 2 Yang et al., 2025) models with and without CoT reasoning on data from three CNLG tasks, namely summarization. question answering and data-to-text using the RAGTruth dataset (Wul et al. 2023). In CoT reasoning mode, the model encouraged to first reason about the factual con- sistency between the input context and the generated output, and then predict hallucinated spans. In the non-reasoning mode, the prompt given to the model instructs it to directly prediction hallucina- tion spans without generating any intermediate content. For each input, the model is run K times
We use the instruct version.
We use the reasoning mode and non-reasoning mode with non-COT prompt elaborated in Qwen3.
3

---

Published as a conference paper at ICLR 2026
Non hal Ialicination
Hallacination
Summarization
Question Answering
Data-to-Text
Densiyy
Density
Donsity
1.5
7B
63
Advantage
Advantage
Advantage
Figure 2: Advantage distribution by model predictions. Advantage distributions across tasks on Qwen2.5-7BB 1enn2.5-7B-Insruuue pretrained model. Non-hallucination predictions (red) receive higher advan- tages than hallucination predictions (blue), revealing class imbalance issue.
and the best prediction is selected based span-F1. We repeat this experiment for different values of and show the corresponding Span-F1 @K results in Figure !
At K = CoT reasoning provides no gains for Qwen2.5-22 and limited gains for Owen3-88 However, as increases, the gap in terms of Span-F1 @II@K, K increases significantly demonstrating the potential of CoT reasoning generate at least one accurate prediction when sampled multiple times. These results provide clear motivation to use reinforcement learning for bringing the reasoning ca- pacity of LLMs related to hallucination span detection the forefront.
We also conducted this experiment with Qwen2. 5-14B and Qwen3-144 4B models and observed a similar behavior. See Appendix for details.
3.2 RL WITH GRPO
As our reinforcement learning framework, we employ Group Relative Policy Optimization (GRPO)
Shao et al. (2024). Unlike Proximal Policy Optimization (PPO) SShhulann al.] (2017), GRPO eliminates the explicit value function and instead computes baselines from relative group scores. The learning objective is defined as:
LaRFO By,,y [min (ro(T)A(T), - +) A(T)))
where 7 denotes a trajectory sampled from the current policy æe, and re(r) = Fu(r)) is the prob- ability ratio between the updated and reference policies at each step. Instead of relying on a critic network as in PPO, GRPO defines the advantage purely from group-based rewards
re(T)
R2 - 1--mean((1)) ((R A(r)
(3)
std std(((a)eettt)
In this formulation, the baseline is determined by the average performance of the group, normalized by its standard deviation, making GRPO particularly suited for scenarios where relative ranking within group is more informative than absolute value estimates.
3.2.1 VERIFIABLE SPAN-F1 REWARD
To apply GRPO for hallucination span detection, we directly use the target span-F1 metric to define the reward. Let be the predicted hallucination spans and S be the ground-truth spans. Then, the reward is defined as
if $ = Ø and S = 8
T'pan
(oonn-F S). otherwise.
This formulation naturally handles both hallucination and non-haln oothaallucinatonn cases. If no hallucina- tions exist and none are predicted, the model receives maximum reward (r (Tspan = 1). In other cases, the reward reflects the quality of overlap between predicted and groundtruth spans.
4

---

Published as a conference paper at ICLR 2026
3.3 REWARD IMBALANCE ACROSS CLASSES
Although GRPO normalizes advantages within groups, we find that the prediction type strongly biases the ad- vantage values. As shown in Figure predictions of non-hallucination consistently receive higher advantages than predictions of hallucination. Figure shows the av- erage advantage values by prediction type confirming that predicting non-hallucination is systematically rewarded more, independent of correctness.
Moan A.brantags
Hallacitation
Summmarizaiion
Quediio Asswering
Data-to-Tstt
This stems from an inherent asymmetry in the reward function Jspen- In the non-halluee on-hallucination class, a model only needs predict an empty span list obtain high reward. In the hallucination class, the model must pre- cisely localize and output the correct spans. This is a harder target, and small errors cause steep drops in the Fl-based reward. As result, GRPO tends to over- incentivize non con-hallucination predictions, leading to bi- ased behaviors with high precision but suppressed recall.
Figure 3: Expected values advan- tage given to Qwen2.5-7B-Instruel pretrained model predictions based on the prediction type. Values shown separately for the three task- based splits of the RAGTruth dataset.
3.4 CLASS- POLICY OPTIMIZATION
may seem like a natural fix to the reward asymmetry issue is to use a smaller reward value for the case $ = S Ø. However, the standardization step used in GRPO will eliminate the effect of such scaling. Hence, to address this imbalance issue, introduce Class-Aware Policy Optimization (CAPO), which uses scaling factor to scale the advantage values computed for samples that belong the non-hallucination class.
A(7)(nh) = a. R, mean ( (Raie)))))
std (Ri (R.))))))
This formulation balances the contributions both classes, mitigating reward sparsity in non- hallucination examples and preventing dominance by non-hallucination examples. We 0.5 in our experiments. This value has been chosen based on the performance of trained model on validation set.
EXPERIMENTAL SETUP
We design our experiments to answer the following research questions, which structure the results and discussion (Section 5): Q1: What the effectiveness of RL4HS?: Q2: Does CAPO alleviate reward hacking and achieve better precision-recall balance?; Q3: Is in-Domain reasoning necessary for hallucination span detection?; Q4: Can simply scaling rewards solve reward hacking?; Q5: What does RL4HS Iearn?
Dataset. We conduct experiments on the RAGTruth benchmark Wu et al. (2023) as the statistics described in Table 6 which provides hallucination span annotations across three generation tasks: Summarization, Question Answering (QA), and Data-to- Text. Each task contains paired source doc- uments, model-generated responses, and human- labeled hallucination spans. This makes RAGTruth one of the few datasets suitable for training and evaluating hallucination detection at the span level rather than only binary classification.
and models as base L.MSs. For comparison, we additionally evalu- ate: Pretrained reasoning models: Qwen3-8B, Qwen3-144 14, and QwQ-32B. Proprietary reasoning models: GPT-5, o3, GPT-4o-mini and m-n-5mm We use the default decoding strategy elaborated in the pre-trained models and use top-p 0.95 (Holtzman et al. 2020), top-k 20 (Holtzman et al. 2020), temperature 0.6 for fine-tuned model generation.
Models. Our experiments primarily use the 5-een255-1BBT1
5

---

Published as a conference paper at ICLR 2026
Baselines.
We compare RL4HS against the following approaches:
Supervised Fine-Tuning (SFT) (Wu et al. 2023 :: trained with cross-entropy on hallucination
span annotations.
RL4HS-GRPO: our RL4HS approach but trained with GRPO instead of CAPO.
Multi- View Attention Ogasa & Arase 2025): token-level detector using features aggregated
from multiple attention heads and attention diversity views; evaluated attention distributions across summarization and data-to-text tasks.
RESULTS & DISCUSSION
Table I: Span-level hallucination detection results on RAGTruth. We report Fl, precision, and recall across summarization, question answering, and data-to-text. Best scores are in bold. means the results taken from Ogasa & Arase
2025
Model
F1
Summarization
Precision Rocall
Fi
Question Answering
Precision
Recall
Fl
Data-to-Text Precision
Rocall
F1
Avg. Precision
Rocall
Proprictary
GP 1-4--mnni w/ CoT
38.4
43.4
34.4
27.3
35.7
23.0
33.7
34.2
33.2
33.1
37.1
30.2
UPT---mini CcT
43.9
33.0
65,5
47.2
37.9
62.7
42.5
29.8
74.7
44.5
33.6
67.6
03 w/ Co'
48.5
40.7
60.1
49.9
43.4
38.9
53.2
45.6
70.0
31.2
43.2
63.0
Non Ron--Resnnnng models
Qwen2. In8-srrrutt
14:7
19.2
10.2
16.0
Qwen2.5- 3-7B--nnstrue w/ CoT
21.0
27.4
17.1
14.5
18.8
11.7
13.0
32.5
8.2
16.2
26.2
12.3
Owen2.5 I4B-Instuct wio CoT
31.5
36.2
27.8
50.7
55.8
29.0
22.8
39.8
29.4
33.8
Qwen2.5 2.5- 148-Insttuct
18.6
27.3
Reasoning models
QwQ-32B
19.4
12.0
12.9
48.1
7.5
7.6
15.3
53.3
Qwen3-8B
34.7
42.2
29.5
30.5
32.0
29.1
20.3
45.2
13.1
28.5
39.8
23.9
Owen3-14B
35.8
36.9
34.9
30.6
30.7
30.6
34,8
40.9
30.4
33.7
36.2
32.0
Finetuned models
SFT-7B
44.1
38.2
51.3
51.3
51.4
54.8
58.8
51.5
50.1
54.1
47.0
SPT-14B
52.7
57.6
48.7
53.9
53.1
54.8
59.6
61.6
57.8
55.4
57.4
53.8
Attenion-BB
Ours: RLAHS
RLAHS RLAHS-GRPO-TD GRPO-7B
51,2
68.7
40.9
55.0
59.6
52.1
56.3
66.5
48.8
54.2
64.9
47.3
RLAHS-77
57.1
RL4HS-144 -4B
57.6
64.2
52,3
54.8
52.5
37.3
62.6
67.2
58.7
58.3
61.3
56.1
5.1 Q1: WHAT IS THE EFFECTIVENESS OF RL4HS?
Table span-level hallucination detection results on RAGTruth across summarization, ques- tion answering, and data-to-text. We compare pretrained prompting baselines with models fine- tuned under our RLAHS framework.
Pretrained instruction-tuned models. Qwen2.5-7B/14R, --stssrttt//-- with or without CoT, perform poorly (F1 below 30), indicating that prompting alone is insufficient for accurate span localization.
Pretrained reasoning models. Models designed for reasoning (QwQ-32B, Qwen3-8B, Qwen3- 14B) transfer some reasoning ability to hallucination detection. For example, Qwen3-14B improves summarization Fl to 35.8 compared to 32.9 for Qwen2.5- Dwwen2.5114B-Instruu However, these models still trail fine-tuned approaches, showing that general reasoning ability alone is insufficient for span-level detection.
Finetuned baselines. scale. Multi View Attention (Ogasa & Arase 2025 further pushes the 7B model to 49.1 F1, though still behind larger SFT models.
Supervised fine-tuning (SFT) provides strong gains, reaching 55.4 Fl at 14B
RL4HS RL4HS consistently outperforms all baselines, including proprietary GPT-4o/5-mini. GPT-5, and o3. RL4HS-7B outperforms SFT on all three tasks (avg. 55.9 v.s 50.1). At 14B, RL4HS-14B achieves 57.6 summarization, 54.8 QA, and 62.6 Data-to-Text Text,, surpassing Qwen3 and the strongest GPT-5 and o3 models. This establishes RL4HS demonstrating that rein- forcement learning with span-level rewards effectively aligns reasoning with hallucination detection.

---

Published as a conference paper at ICLR 2026
Span-FI
Opan-Precision
Span- Recn-Recall
Precision
Rocall 2
Training Step
Trining Step
Training Step
Figure 4: Training dynamics of GRPO (red) and CAPO (blue) on Qwen2.5-7B-Instee model. While GRPO exhibits high precision but declining recall due to reward hacking, CAPO stabilizes recall without sacrificing precision, yielding consistently higher span F1. Shaded regions denote standard deviations across runs.
5.2 Q2: DOES CAPO ALLEVIATE REWARD HACKING AND ACHIEVE BETTER PRECISION RECCLL BALANCE?
A key limitation observed with GRPO is that models often exploit the reward design by default- ing to predicting no hallucination spans, which yields high precision but severely hurts recall. This behavior reflects a form of reward hacking, where the model learns shortcuts that maximize rewards without genuinely improving hallucination detection. As shown in our advantage distribution anal- ysis (Figure 2}. predictions of non-ha hallucinationn systematically receive higher advantages, biasing the policy toward conservative behavior.
Figure 4 compares training dynamics of GRPO and our proposed CAPO across span F1, precision, and recall. We make two observations: (1) GRPO favors precision over recall. As training pro- gresses, GRPO maintains relatively high precision but recall gradually drops, showing the model's tendency to avoid making positive span predictions.;
(2) CAPO balances precision and recall. By re-weighting class-specific advantages, CAPO stabilizes recall while preserving strong precision, resulting in clear improvement in span Fl throughout training.
These results confirm that CAPO directly addresses the imbalance highlighted our advantage distribution analysis. By correcting for class class-dependent reward sparsity, CAPO mitigates reward hacking and achieves a better precision-recall trade-off, consistently yielding higher span F1 com- pared vanilla GRPO.
5.3
Q3: Is IN-DOMAIN REASONING NECESSARY FOR HALLUCINATION SPAN DETECTION?
To assess whether hallucination span detection requires in-domain reasoning rather than generic reasoning ability. we conduct leave-one-out training with RI RLSHS RLLAHS-OOD-700 holding out one task at a time and evaluating on the unseen task. Figure 5 shows results compared against reasoning- focused models (QwQ. Qwen3) and large-scale GPT-series basclines.
Ancwrring
Anp
Spandi
Covert oft
10 Pevert An aLdis-in a (haay OPFS-p (PP- CTT
Cyp
To Couest atl (Pett to
Chans kg aLais LDD Rt ORun) OPPs Sommn soO (PF SOT aLanom 18 (hem
tweets A. Cevt coil
(rend 40 at aadisann-- D-- (hany oPfir OPE for atanee B8 (hann
TB Gromd eef Oat in Comnat
aLdis motd- TB (rury) OPXS OFT CoT
Urend A4
RLaIs (160
Cpet
grra
Figure 5: Out-of-domain evaluation on RAGTruth. Span-Fl scores Ragtruth dataset. Our RLAHS-OOD-7B model performs competitively with larger reasoning models, showing the benefit of sean-level reward fine-tuning. We use Instruct models for Qwen2.5 models.
7

---

Published as a conference paper at ICLR 2026
General- pureroe- reasoning models such Qwen3 and QwQ transfer some reasoning ability but their Span-Fl scores often remain below 40, showing that generic reasoning is insufficient for fine- grained hallucination detection. RL4HS-OOD-PH in contrast, achieves consistently stronger results across all held-out tasks, approaching the in-domain "task" topline RL44S-7B. Moreover, despite being much smaller, RLAHS-OOD-7B performs better than GPT-4-mini and remains competitive with GPT-5-mini, and even GPT-5, underscoring the efficiency of span-level reward fine-tuning. These findings highlight that in-domain "reasoning" learned with span-level rewards is essential for robust hallucination detection.
5.4
Q4: CAN SIMPLY SCALING REWARDS SOLVE REWARD HACKING?
One concern with GRPO that its standard- ization of group rewards diminishes the intrin- sic difficulty difference between hallucination and non-hallucination cases, often biasing the model toward predicting non- con-hallucination To address this, we explored variant of Dr. (Liu et al. 2025), which removes standardiza- tion and instead scales the reward for success- fully predict non-hallucination by factor ? Table reports results under different values. While Dr.GRPO influences the precision-recall tradeoff (e.g., higher " increases recall at the cost of precision), overall performance is infe- rior to standard GRPO and RL4HS. This sug- gests that the normalization step in GRPO is crucial, and simple reward rescaling cannot ef- fectively address reward hacking in our task.
Table 2: Comparison of GRPO, CAPO, and DrGRPO variants with RL4HS. CAPO im- proves F1 by addressing reward imbalance, while GRPR with different 7 values shows varying precision-recall trade-offs but does surpass
CAPO.
Avg. Precision
Method
F1
Recall
GRPO
54.2
64.9
47.3
CAPO
55.9
62.9
51.2
Dr.GRPO w/ -=0.1
52.5
53.6
52.3
Dr.GRPO w/ 005.
54.7
62.2
49.4
Dr.GRPO w/ 1=1.0
53.1
64.1
45.8
5.5
Q5: WHAT DOES RL4HS LEARN? A CASE STUDY
To better understand the reasoning behaviors learned by RL4HS, we examine qualitative outputs on the RAGTruth dataset (Table 3) The example highlights discrepancy regarding whether the restaurant provides catering services. Pretrained model. Before training, the pretrained model fails identify the inconsistency. Although it checks structured business hours and customer reviews, it overlooks the fact that the structured data contains no attribute related to catering services. As result, the model produces no hallucination spans. RL4HS. In contrast, RL4HS correctly flags the catering services claim as hallucination. Its reasoning process closely mirrors the numan-designed heuristic pipeline:
Step 1: Identify explicit claims in the article (e.g., "provides catering services").
Step 2: Cross-check these claims against structured business data (which does not list catering services attribute).
Step 3: Conclude that the claim is inconsistent and mark as hallucinated.
This case demonstrates that RLAHS goes beyond surface-level reasoning traces. Instead of produc- ing generic or irrelevant explanations, the model performs systematic consistency checks that align with heuristic rules used in prior hallucination detection pipelines. This suggests that the reasoning behavior learned under span-level rewards is genuine, faithful, and semantically grounded.
RELATED WORKS
Hallucination Detection. Hallucination detection research has evolved from binary classification to fine-grained span detection. Early work focused binary
whethherr text contains
hallucinations (Manakul et al. 2023 Luo et al., 2023 Tang et al. 2024p. However these approach failed to localize where the hallucination. Yang et al. (2024b); Scirè et al. (2024) proposed cade pipeline that leverage atomic-fact generation, natural language inference to detection halluci- nation. But the pipeline is hard optimize. Recent methods target span-level detection. introduced
8

---

Published as a conference paper at ICLR 2026
Table 3: Case study comparing pretrained and RL4HS models on detecting hallucinations.
Review data
"name"; "Benchmark Eatery'. 'address": '1201 Stare St" "city": "Santa Barbara', 'state': 'CA, 'cotegories': "American (Traditional), American
(New), Breakfost Brunch, Resturants, Seafood, Vegetarian, Nightlife, Event Planning & Services, Bars Venaes
Spaces", 'bours': "Monday':
"0:0 0:0", "Tuesday:: "11:30-20:0 20:0", Woodnedayy; 11:30-20:00 "Thunsayy: : 10:30020:00,, Friday;; 11::0016600" 'Saturday' : 11::0011::0" "Sunday" : gangee: True. 'sreett Troe. 'aaisdaedd Fubse, "lo': Trae valet' Fulse, RestaranssRsvvvaiins : Fabse "OutdorrSeniing" : True, "WiFF; "hee'. Cestauramt-TakkoOu True. ResaannmtGooooFurCem
"I 11:30-20x0", "itribbtes': : "BusinssPPrkingg
True, "Music': False, 'Ambience" "couristy' "divey' False. "intimate" False, "trendy" False. "upscale": False, "classy" True, "casual: Tree "business stars' 4.0, 'reveww info": l'review stars' 40, reeiew date:: "2022-01-00 21:02:49" 'review text": "Nice litll place downtowe Sonta Barham where we stepped for late lumciiearry dinnrr on our way back hore to San Diego We loved our flathreads. had been craving clam chostler and ordered hure.
False, "hipster": False, "remantic"
It was a littl thick for my liking, hut tisted good. Nice, friendly upscale creaal place on Sizte Street.. "roview stars: 1.0, 'review datee: "2021 3021-12-20
22:29:32", "review text": "I had high hopes for this place, but fell flat hig time. I ondered the BL AT with grilled chicken and a side salad. specifieally asked for would not trust this place to mindful that. The grilled chicken was not scasoned
and they put
both the sandwich and the salad. Not
deal.
pick all off. And food overall was just exteemely bland. My sandwich
have allergy.
was stacked so high that the avocado all fell out as I was cating it The side salad is lettuce, cucumber, red onion, and comato. My bf get burger with
onion rings and he did not enjory wither part of the meal The onion rings are boer battened and basically all batter. The burger was also unscasomed and and mok for his soda be refilled and were never offered not fresh becouse my meal y have any allergens had stamach pains for would' ve given that reaction. Lactly, cleaning spma
not enjoyable. The service is ok. The staff is nice enough but my bi did have to refill
both left the
with
sche. pensenally
antowaads and I
only assume it's because the they used on the tables just smelled funky. Idkk how to explain it but we actually moved an outside table because the inside one smelled so musty and mildowy.patio is nice, but not worth coming back for. Additionally, the menu too scattered and overwhelmiag hewheiiingg I think they should stick to one type food and work on increasing the quality rather than having many different
options that aren't cohusive." 'review slars:; 5.0. 'review cale': 2021--1-22 1-29 20:45 43', rrevew text: 'We have been here few during trial the past couple weeks and
keep ondering the
burrito but overyone scems enjoy their food. The one piece of feedback would Spanish soup seems change from not too spicy to spacy.1
deoeennn-gwerrteeddeee article given review data
Benchmurk Eatery is an American-style le restaurant located in downtown Santa
Barhara, Californin. The restautant offers variety of cuisines, including traditional Americus, seafood, vegetarian, and event planning services. They also provide outdoor seating, free Wi-Fi, takeout options, and catering services. According to customee reviews, the restaurant has a four star tating, with mixed Some customers have priied the restaurant's fluthreads and clum chowdes, while others have criticiied restaurant for poor flavor. blandness, and inconsistest ingredients, Additionally, some customers have expressed concerns about restaurant' handling of allergies and the cleanliness of the establishment. Overall, Benchmark Eatery appears to be popular choice for dining in dowatown Santa Barbara, with range of menu optoos and a cosy atmosphere.
opinions about the quality of the food and
Before 8L441S-78 (Pratrained model)
an Step I: Analyze the structared data to understand the restaurant's ' categories and services. restinmant casegorized alfering American (Traditiona) American
well Breakfast Brunch. alse provides seafood, vegetarian options, event planning services, ouidoar seating. flee Wi-Fi, takeout options, catoring services. et Step 2: Compure information article with the stuctured data. The article correcly mestions that Benchmark Eatery is an American-sryle restaurant located in downtown Santa Barbarn. It also correctly states that they offer a variety of cuisines including traditional American and scafood. The article accurately notes that they provide outdeer seating and free Wi-Fi, st Stop 3: Check for any discrepuncies between the review information provided in sencttred data and meniioned in the article. The review information includes both positive feedhack about flathreads and clam chowder from article accurately captures these mixed opinices. de Step Verify if there are any incomisseencees regarding
well negative feedback
about poor flavor. blandness from others. food hindling practices or cleunliness. The structured data does not explicilly mention concerns about handling allergiss or cleanliness issues; however. one customer review did express conser about incomsistenn ingredients used in their meal wh which could imply some issues with food quality or prepuration consistency. The article mentions concems about handling allergies but does spocilically siste cleanliness issues bused on this data aloee. #4 Step S: rating according
Confirn rating based
The business
reviews mentioned both
Rasad
this analysis
("hallucination list": U}}
RLA
Step 1: Check for Catering Services The article states "They also provide ouidoor seating. free Wi-Fi, takoout options, and catering servioes." " However, per datts, colering services seems inconsistent. It scems to say "catering services" which sooms inconsistent, It appears to be an extor; suspecting poiential candidates typo misinerrpeetttion. et Step 2: |Vaaliate Food Quality and Services. The text says that acoording to castomer's opinion's the restaurent has four-star matnn with mixad food quality and service as discussed. #* Step 3: Validate Food Qualitys 's states that "Some enstomers have praised the restaurant's 's Baibreeds and clam chowder, while others have unticized the restaurant for poor flavor, blandnces, and incomistent ingredients. This part is conformed by our data, et Step 4: Evaluate Customer's s Concems.. It says customers expressed concerns about food quality again as well as cleanliness estabiismment
purt sooms valid per data. Step 5: Check for Food Variety deseribes variety of caisines traditional American (both traditional new), breakfast brunch, seafood, vegetarian food again as said in test. Step 6 Evaluste Overall Assessmeni Resisummn's 's Attractionn The article suggests overall populaitt in dining with mang of menu options and aory stmosphere per our data, Now we suspect posestial condidates such is catering services or other sispects based on apparent erron. it says eatering services it seams suspicious. The candidate suspects are often candidates typographical emors or misinterpretation: eee Compaled Results: ( ("hallucination list": ("entering services"]]
RAGTruth (Wu et al. 2023) with human-annotated spans across three generation tasks. Ogasa & Arase 2025) aggregated multi-head attention features for token-level detection. However, these attention- methods lack explicit reasoning mechanisms.
Reasoning Enhancement in NLP. Group Relative Policy Optimization (GRPO), originally de- veloped to improve mathematical reasoning by comparing groups of outputs rather than relying on a separate value model. GRPO has since been extended and adapted to a variety tasks such coding (Liu & Zhang 2025 Chen et al. 2025), planning (Hao et al. 2023), tool-calling (Feng et al.; 2025a Shang al.) 22255) More recently, researchers has show that GRPO can also be applied to enhance reasoning in traditional NLP tasks such as NLI (Shao et al. 2024), intent classification (Feng et al. 20256), and safety alignment Li et al. (2025). Showing the effectiveness of GRPO with L.MMM
9

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
We introduced RL4HS, a reinforcement learning framework that uses span-level rewards to align LLM reasoning with hallucination detection. While CoT offers limited single-sample gains, RL4HS distills its multi-sample advantages into stronger predictions. With CAPO to address reward im- balance, RL4HS outperforms pretrained reasoning models and SFT RAGTruth, and produces faithful, heuristic-like reasoning traces that improve both accuracy and robustness.
REFERENCES
Yongchao Chen, Yueying Liu, Junwei Zhou, Yilun Hao, Jingquan Wang, Yang Zhang, and Chuchu Fan. II-coeeeinterpreter::: Training Ilms to reason with code via supervised and reinforcement learning, 2025. /ttptttts///at:ttt /ttpa/twww..i..aaaa..oooo.ooo.o/ http://aa/2222211112 21668
Yung-Sung Chuang, Linlu Qiu, Cheng-Yo Yu Hsich, Ranjay Krishna, Yoon Kim, and James R. Glass. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1419-1436, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10. 20555//1122224.mmpmmaaaa emm1p- ma in.i 84/
URL httpt:ttp///cc //ccl antho logy. org/2024,
Ron Eliav, Arie Cattan, Eran Hirsch, Shahaf Bassan, Elias Stengel-Eskin, Mohit Bansal, and Ido Dagan. Clatter: Comprehensive entailment reasoning for hallucination detection, 2025. URL
attps://arxxe wttp://wwgbbbb......ba//// 05243
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms,
2025a. UtLhttps::/ www.iv.ooror abs/ 2504. 11536
Zihao Feng, Xiaoxue Wang, Ziwei Bai, Donghang Su, Bowen Wu, Qun Yu, and Baoxun Wang. Im- proving generalization in intent detection: Grpo with reward-based curriculum sampling, 2025b.
URL (RLLhhtss://axxtt www. 2tt.b///5544559999
Haoyu Gao, Ting-En Lin, Hangyu Li, Min Yang, Yuchuan Wu, Wentao Ma, Fei Huang, and Yong- bin Li. Self-explanation prompting improves dialogue understanding in large language mod- els. In Nicoletta Calzolari, Min-Yen& Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 14567- 14578, Torino, Italia, May 2024. ELRA and ICCL. URL http:://aclanthhlll.ee org/ 2024. 1eec-rrecmaal -aa.ee-mmiin..... 1269/
Shibo Hao, Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model, 2023. URL https: //axiv..
srg/abs/2305 14992
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration. In International Conference on Learning Representations, 2020. URL https: //openreview
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2) 1-55, January 2025. ISSN 1558-2868. doi: 10.1145/3703155. URL
Ziwci Ji, Delong Chen, Etsuko Ishii, Samuel Cahyawijaya, Yejin Bang, Bryan Wilie, and Pascale Fung. Llm internal states reveal hallucination risk faced with query, 2024. URL https:
// warxiv.org/aba..... http://abb2207...222222
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. Why language models hallucinate, 2025. URL httpps:/aaatt wttp:////ww.////////////// 220../00/0.9...
10

---

Published as a conference paper at ICLR 2026
Xuying Li, Zhuo Li, Yuji Kosuga, and Victor Bian. Optimizing safe and aligned language gen- eration: A multi-objective grpo approach, 2025. URL https://aaxiv org/ abs 2503. 21819
Jiawei Liu and Lingming Zhang. Code-rl: Reproducing r1 for code with reliable rewards. 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang. Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective, 2025. Uttpttss:
org/aba/2503.c 20783
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. Chatgpt as a factual inconsistency evaluator for text summarization, 2023. URL httthtttt//aaat http://aaxxxvvorr/aa///3333 15621
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black-box hallu- cination detection for generative large language models. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Lunguage Processing. pp. 9004-9017, Singapore, December 2023, Association for Computational Linguis- tics. doi: 10. 111186655/112222.mmmm.... main. URL /tL./ttpss///ccaatooooooo.. org.202230002223333333
emmlpp main. 557/
Yuya Ogasa and Yuki Arase Hallucinated span detection with multi-view attention features, 2025. URL htttts:ssss/aii....tt/
/504. 04335
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Sernadett-Shaping Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho. Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning. Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Hcaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirehner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Fukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco. Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo. Hyconwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross. Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky,
11

---

Published as a conference paper at ICLR 2026
Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang. Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https://arxiv otg/abs////ooooooooo 220...2233... 220.//3333.7744
Selvan Sunitha Ravi, Bartosz Mielczarek, Anand Kannappan, Douwe Kiela, and Rebecca Qian. Lynx: An open source ballucination evaluation model, 2024. URL https::/axxiv..ott
abs/ /207 08488
Elisei Rykov, Kseniia Petrushina, Maksim Savkin, Valerii Olisov, Artem Vazhentsev, Kseniia Titova, Alexander Panchenko, Vasily Konovalov, and Julia Belikova. When models lie, we learn: Multi- lingual span-level hallucination detection with PsiloQA. Christos Christodoollpoollss, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 11663-11682 Suzhou, China, November 2025. Association for Computational Linguistics.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL httpp:://a.xi.....e//// abtp///as//////////// 063477
Alessandro Scirè, Karim Ghonim, and Roberto Navigli. FENICE: Factuality evaluation of sum- marization based on natural language inference and claim extraction. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACL 2024, pp. 14148- 14161, Bangkok, Thailand, August 2024. Association for Computational URL
Linguistics. doi: 10. 10.18653/w1/20222 htt.://////22.....nnnnnalllllll 2024. fi nd acs 8417
Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, Xudong Zhou Bowen Zhang, Ying Xin, Ziming Miao, Scarlett Li, Fan Yang, and Mao Yang. aggar:-agent: Agentic reasoning technical report, 2025. URL
abs/ 0008/2508.20 20722
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. /Rtttt//////xxxi. org.abs/2//..2...
03300
Liyan Tang, Philippe Laban, and Greg Durrett. MiniCheck: Efficient fact-checking of LLMs on grounding documents. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Pro- ceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 8818-8847, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.18653/v 810/5553//1122244mmmmmmmmmmii emnlp-main in. 4997]
URL htthhttss://cclaan http:///llaatttlll.orrrr////////
Qwen Team. Qwen2.5: A party of foundation models, September 2024. Utthhtts:://qwenl
github. wttp:/ic//////ooo (wall/.ggwwen..mmm qwen2.
Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Cheng Niu, Randy Zhong. Juntong Song, and Tong Zhang. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmenned language models, 2023.
Tingyu Xie, Qi Li, Jian Zhang. Yan Zhang, Zuozhu Liu, and Hongwei Wang. Empirical study of zero-shot NER with chatGPT. In The 2023 Conference on Empirical Methods in Natural Lan- guage Processing, 2023. URL htt.tttss:// /ttp:////eeeeeww...omm wet.eee...eeeeeeeeeeee htt.//ormmmmmo /id=Wssiiisiiis ImmU
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu,
12

---

Published as a conference paper at ICLR 2026
Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin. Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mci Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: //tp://aaivvviv/iaa org/ abs/2505... 09388
Jiuding Yang, Hui Liu, Weidong Guo, Zhuwei Rao, Yu Xu, and Di Niu. Reassess summary factual inconsistency detection with large language model. In Sha Li, Manling Li, Michael JQ Zhang, Eunsol Choi, Mor Geva, Peter Hase. and Heng Ji (eds.). Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pp. 27-31, Bangkok, Thailand, August 2024a. Association for Computational L Linguistics. doi: 10. 16533v1/2024.kmm/ 1.3. URL https: /lccss://aclantiin wwwphhhooloooogg////// know 1.3/
Joonho Yang, Seunghyun Yoon, ByeongJeong Kim, and Hwanhee Lee. FI77: Factual inconsis- tency detection by zoom-in summary and zoom-out document. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Meth- ods Natural Language Processing, pp. 30-45, Miami, Florida, USA, November 2024b. As- sociation for Computational Linguisties. doi: 10. 1806://///02222mmlmmmm.o /aclaatthlogy.orqq
URL https:
cannpp-mann.3
Qiying Yu, Zheng Zhang. Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guang- ming Sheng, Yuxuan Tong. Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingx- uan Wang. Dapo: An open-source 11m reinforcement learning system at scale, 2025. URL https::/aaxxt http://axxiv.oooa/////////////// wttg/as///500000 14476
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing, Sentiment analysis in the of large language models: A reality check, 2023. URL btttps://axxvv.org/// 2305 15005
Zhiying Zhu, Yiming Yang, and Zhiqing Sun. Halueval-wild: Evaluating hallucinations of language models in the wild, 2024. URL https: www.xxii.og/aas//./oo 2000g//bb//43.......
2403, 04307
A APPENDIX
A. USE OF LARGE LANGUAGE MODELS (LLMS)
In preparing this paper, we made limited use of large language model (LLM) general-purpose writing assistant. Specifically, the LLM was employed to refine the clarity, grammar, and flow the manuscript text. The LLM did not contribute to the research idcation, methodology, experimental design, analysis, or interpretation of results. All scientific content, claims, and conclusions are solely the responsibility of the authors.
13

---

Published as a conference paper at ICLR 2026
A.2 PROMPT
COT for Summarization
"Below is the original document:"
[reference]
"Below is summary of the document:"
[response]
"Your task is to determine whether the summary contains hallucinations." "First, provide reasoning with the following format:"
## Step 1: < your first reasoning step ## Step < your next reasoning step >
...dadd as many steps needed) Then, compile the labeled hallucinated spans into a JSON dict, with a key hallucination list and its value is a list of hallucinated spans. If there are potential hallucinations, the output should be in the following JSON format: hallucination list: [hallucination spanl, hallucination span2, ...]. Otherwise, leave the value as an empty list follows: hallucination list: (]-
COT for Question Answering
"Below is a question:"
{question)
"Below are the related passages:"
(reference)
"Below is an answer:"
{response]
"Your task is to determine whether the answer contains hallucinations." "First, provide rea- soning with the following format:"
## Step 1: < your first reasoning step
## Step < your next reasoning step >
...dadd many steps needed) Then, compile the labeled hallucinated spans into a JSON dict, with a key hallucination list and its value is a list of hallucinated spans. If there are potential hallucinations, the output should be in the following JSON format: hallucination list: [hallucination 1,an hallucination span2, ...]. Otherwise, leave the value as an empty list follows: hallucination list: [].
COT for Data-to-text
"Below is structured data in JSON format:" [reference]
Below is an overview article written in accordance with the structured data:" {response}
"Your task is determine whether the article contains hallucinations." "First, provide rea- soning with the following format:"
## Step 1: your first reasoning step
## Step 2: < your next reasoning step
...(add as many steps needed) Then, compile the labeled hallucinated spans into a JSON dict, with key hallucination list and its value is list of hallucinated spans. If there are potential hallucinations, the output should be the following JSON format: hallucination list: [hallucination 1, hallucination span2, ...]. Otherwise, leave the value as empty list follows: hallucination list: [].
14

---

Published as a conference paper at ICLR 2026
w/o COT for Summarization
"Below is the original document:"
{reference}
"Below is summary of the document:" {response]
"Your task is to determine whether the summary contains hallucinations." Then, compile the labeled hallucinated spans into JSON dict, with a key hallucination list and its value is list of hallucinated spans. If there are potential hallucinations,, the output should be in the following JSON format: hallucination list: [hallucination spanl, hallucination span2.... ...]. Otherwise, leave the value empty list as follows: hallucination list: U.
w/o COT for Question Answering
"Below is question:"
{question] }
"Below are the related passages:"
{reference]
"Below is an answer:"
(response}
"Your task is to determine whether the answer contains hallucinations." Then, compile the labeled hallucinated spans into JSON diet, with key hallucination list and its value is a list of hallucinated spans. If there are potential hallucinations, the output should be in the following JSON format: hallucination list: [hallucination spanl, hallucination span2.... ...]. Otherwise, leave the value as an empty list as follows: hallucination list: [].
w/o COT for Data-to-text
"Below is structured data in JSON format:"
{reference]
Below is an overview article written in accordance with the structured data:" {response]
"Your task is to determine whether the article contains hallucinations."
Then, compile the labeled hallucinated spans into ISON dict, with key hallucination list and its value is a list of hallucinated spans. If there are potential hallucinations, the output should be in the following JSON format: hallucination list: [hallucination spanl, hallucination span2.... ...]. Otherwise, leave the value as an empty list as follows: hallucination list: [].
A.3 TRAINING DETAILS
Table 4: Training details for SFT and RI.
Method
Size
Learning Rate
Batch Size
7B
le-6
64
SFT
14B
le-6
64
7B
le-6
RL
14B
5e-7
64
We trained our models using 8 H100 GPUs. The learning rate and batch size configurations provided in Table g For reinforcement learning training, we set the group size to 16 and used rollout generation with temperature 1.0, top-p 1.0, and top-k -1. Following Yu et al. (2025)
15

---

Published as a conference paper at ICLR 2026
also adopted clipping threshold of clip_high 0.28. Due to the lack of the reasoning data, fine-tuned instruct model with RL directly instead of doing SFT first.
For GPT-series models, used top-p 0.95 and temperature 0.7 to generate response during inference. All the trained models were trained with epochs and selected the checkpoints with the best performance self-splitted validation set. In our training, resolved the data class imbalance by upweighting hallucination class to have equal amount of data.
A.4 STANDARD DEVIATION
Table Span-level hallucination detection results (STD) on RAGTruth.
Summarization Precision
Question Answering Precision
Data2Text Precision
Model
F1
Recall
Rocall
F1
Recall
Proprictary models
GPT-4o-mini w/ COT
1.5
1.9
1.5
0.6
0.7
0.8
1.9
2.1
1.9
GPT-S-mini w/ COT
1.0
0.9
1,4
0.5
0.4
0.9
1.5
1.1
2.5
GPT-5 w/ COT
0.5
0.4
1.2
0.4
0.4
1.1
0.9
0.6
1.7
03 w/ COT
0.8
1.4
0.5
0.5
0.8
2.1
1.5
Non-Reasoning models
Joven2.5-7B-Instie
w/o COT
1.7
1.4
1.3
0.9
1.1
1.3
Qwen2.5-P7 Boven2.57BB-Be B-B-nnsruuuttt
2.0
2.7
1.8
1.3
0.9
1.0
2.6
0.7
w/
Owen2.5-14B-Inst w/o COT
1.5
0.7
0.9
0.6
0.6
0.7
Owenn.5-144BIInseie w/ COT
2.3
2.2
1.8
3.0
1.4
1.3
1.9
Reasoning models
QwQ-3BB
1.9
3.8
1.3
1.4
4.0
0.9
1.0
2.6
0.6
Qwen3-8B
2.1
2.2
1.8
2.6
1.4
2.7
Qwen3-14h
1.8
1.8
2.1
1.1
1,2
Finetuned models
SFT-7B
1.4
2.8
0.6
0.3
1.6
1.2
U.4
2.4
2.4
SFT-14B
3.2
0.5
0.7
1.0
0.5
0.6
1.8
0.5
Multi View Attention-7B
N/A
N/A
N/A
NIA
N/A
N/A
Ours: RLAHS
RLAHS-GRPO-TB
1.0
2.2
1.3
2.4
7.5
4.5
0.5
0.5
0.4
RL4HS-7B
0.5
0.5
0.4
4.4
4.8
7.3
0.3
1.5
1.6
RL4HS- 14B
1.1
1.0
1.9
1.0
1.9
0.7
1.7
A.5 DATASET STATISTIC
Table 6: Dataset statistics for RAGTruth. Numbers indicate the number of hallucination examples, with the number of non hallucination examples shown parentheses.
Summarization
Question Answering
Data-to-Text
Train
1209 (2646)
1277 (2732)
3048 (1347)
Val
271 (629)
269 (614)
624 (276)
Test
204 (696)
160 (715)
579 (321)
A.6 SPAN-F1@K
More results on Span-F1 @ for Qwen2.5 (Figure and Qwen3 (Figure models.
16

---

Published as a conference paper at ICLR 2026
wisCOT
con
Summarizution
Question Answering
Data-to-Test
08
Qwen2.5.7 B
Spon-F 160K
9.5
28
45
1.
23
69
Number of prodictions K
Number of predictioons K
Number of predictions K. Data-to-Text
Summarization
Question Answering
Qwen2.55 Dwenn2.5.145 55.44
Span-F1 L@K
@KK
9.
84
49
Number of predictions K
Numbet of predictionns
K
Number of predictions K.
Figure 6: Hallucination span detection with and without CoT reasoning. Results are shown for summarization, question answering, and data-to-text tasks the RAGTruth benchmark.
W/CCOT COT
Summarization
Question Answering
Datn-to-Text
Qwenn 8B
Span-F L.CK
Spun P.
9.5
6.4
28
48
110
190
23
49
Numher of predictions K
Number ofpredictions K
Number of predictions K
Question Answering
Qwen3-14p 143 Span-F160K
Span-F1og Lg
FAGK
0.5
0m
28 40
90 69
100
20
49
68
Number of Prodictions K
Numbet of predictions K
Number of predictions
Figure 7: Hallucination span detection with and without CoT reasoning. Results are shown for summarization, question answering, and data-to-text tasks the RAGTruth benchmark.
17

---

Published as a conference paper at ICLR 2026
A.7 MORE RL4HS REASONING EXAMPLES
We provide three additional examples of RL4HS reasoning paths in Table - and 9 The third example illustrates a common mistake of RL4HS. Specifically, the RL4HS model becomes confused while converting between 24hr clock to 12hr clock formats.
Table 7: RL4HS reasoning example 1 (QA)
Quextion
difierences and similarities between red and white blood cells
Retrieved pavsager
passage Shares Saves, Save. Blood is up three parts: red pells, white cells and platelers Red blood colls responsible for the oxygen delivery from the lungs to the tissuc in the body. White blood cells are a part of the immune system. These components circalate in the fluids in the blood vassels, which is pamped by the heari luncs & Saves. Save. Blood is made up of three parts: rod cells, white sells and platelets. Red blood colls are responsible for the oxygen delivery from the lungs to the tissue in the hodyy White blood colls are part of the immune system.
found blood. passage 2: Both red blood sells and The red blood cells and the White Blood Cells can me found the circulatory system. 4.lt both have from one part to other part our body cell membrane and cytoplasm hite blood cells are part of the Immunne System and help the body combat discase, viruses, etc. Red blood eells are part of the Circulttoyy System and curry things such
blood cells
bload
white blood eells
carried
gen throughouu the body. are formed elements
passuge 3: Best Answer: Similarites: RBCs,
from pluripotent stem in bone marrow est Answer: Similaritie: RBCs, WBCs are formed clements of blood. RBCs, WBCs are formed from pluripotent stem cells in red hone marrow.
RBCs, WBCs
Machine-g
maaechin--enerateie generated
given question and retrieved passagos
Based the given passages, here are the differences similaniies between red and blood cells:
Differences:
Function: Red bload cells are responsible for delivering exygen from the lungs to the body's rissues, while white bloed cells are part of the immune sysiem help combat discases inficctions, Structure: Red blood calls
unique biconcave stape allows to masimiize their surface area for gas exchunge, wherass white blood cells have a irrepular, amochu-like shape. Similarities:
Formation: Both red and white blood cells are formed from pluripoient stem colls in the red bone marrew Presence in circulatory system: Both red and can be carried by blood from one white blood cells have cell membrane and cyuplasm. Therefore,
and while blood oells are found in the circulatory
the body another. Cellular components:: Both red and white blood cells have some similarittiss in terms their formation and presence in the circulttory system, they difler in their function and structure.
BLAHS roasoning path and output
at Step The statement "Dffeereness" with Function: Red bland cells responsible for delivering oxyeen from the lungs the body's tissues, Passage seems accurate from provided content about their roles directly. As Step 2: The statement Strncture: Red clll have unique biconcave disk shape that allows them to maximize their surface area for gas exchange, wherens white bload cells have irregular, amocha-like shope.' appears without strong hasis from any given related passages (Passgell Passuge2 Passgee). None these suippets confirm details shout structure shapes dircclly. at Step The statement 'Similatties" with content shout fomution hased off passage3 seem accurate. Details providdd from snippet directly. sit Step 4: The stalement Tresence in circulatory system:' along with 'Cellalar componants:" socm based off passage2 accurately.
while blood cells are part the immune system and help combat diseases and infections.
from Passage
{( (("alltxination lisi": 1na Strecture: Red blood cells have unique biconcave disk shape that allows them to maximize their surface for exchunge, whereas white bloed cells have inzeular, amocba- like stape.")]
Table 8: RL4HS reasoning example 2 (QA)
Question
what is the proper way to fertiliee your grass
Retrievedl passoges
passage I: Feriilize
right time year. onder your fottilieer work you'll time that you fertilize beginning of growing season. This is different for different types of grasses. Fertilize warm season grasses when the grass begins to tum green at the beginning of spring.
passuge 2: Picking Out Fertilizer. I. Know your grass. The type of grass you have will desermine what type of fertilizer you need often you
and how
fertilize. 2. Do pll soil lest. measures the alkalinit or acidity soil, and helps determine what pil your fertilizer should have. 3. Mensure your lown.
passuge 3: Pour in the rest of the femilizer. Apply the rest of fertilizer in pespendicular pattern. Step on the turf and if the leaf bladd bounces back up, your
good shape. loys flat, you' over-fertlizedd
Machine-gunerated aoveer given question and retrieved passages
Hased on the provided passages, the proper way to lettilize your grass is as Follows:
proper
Time your fertilication at the beginning of the growing season, which varies for different types of grasses. Warm grasses should be ferillzed cool scasom grasses should
(Passage Chuose lettiliece that appropriate for your specific type of grass and perform a pll soil test to determine the correct pil level for your fertiliner. (Passage 2) Apply the remaining fertilieer in perpendicular puttern, and chock if the leaf blade bounces back up after stopping on the lurf to ensure that your lawn is good hased given passages. The passages do not provide
when grass tams green
carly spring,
shape. If the leaf blade lies flat, may have over ferillized (Passage 3) Unable informaiion on the amount fertilizer to use, how often fertilize, any specific guidelines for organic slow release fertilizers.
RL4HS reasoning path and output
The statement" '1. Time your fertilization at the beginning of the growing semson, which varies for different rypes of grasses. Warm senson grasses should fertilized when the grass green in the early spring. answer soams to include about cool spason grasses part about heing leriliied in fall based from content given from Passage I directly about warm season hut lacks information about cool scasons details strictly from provided snigpets. de Step 2: The test about picking out ferillizer and related content accuratic based off Passag directly. AA Step 3: About applying fertilizer and its mothod alongside checking turf's condition is based striclly Passage3 honce appears accurate.
At Step I:
cool season grasses should he ferilizeed the fall. (Passage 1y from
Step Other details mentioned amount, frequency or any specilic guidelines organic or slow-releuse fertilizers are noted as not having enough info. {("hall (("Iallucination list": [" ('whiko cool scason grassss should be fortilized in the fall""])
18

---

Published as a conference paper at ICLR 2026
Table 9: RL4HS reasoning example 3 (Data2Text)
Review Dola
'uame': 'Comedy Hidkaway", , 'aukhess': 'sily': "Santa Barbani. 'state'; 'CA' 'catlogories' "Performnng
Entertainment Calerers,
Party & Event Phaning. Com- cdy Clabs, Venses & Event Spaces, Event Planning Services, Nightlife", 'hours:: "Tuesday': : "18:0-1:0" 1:01 'Friday" '18:0-23:0', 'Satudayy" : '13:0-23:0", "attributes" 'BusinesPPariing garage: False, 'strect': False, 'validated"; Fulse. "lot' True. 'valet: Fabse Restaranseeerrniier None. OOtdoorSearingg: Fulse. 'WiF'; 'no". RestaannttTaalOer Nove. RRstaama::ooooooCruuee
True. "Music'
Ambience": 'ouristy": False, 'hipster:: False, bornaniic': False, 'divey:: False, 'intimate' Trae. 'trendy': False, 'upccale' False, "classy" False, "assual:: Trme. 'busi ness stars' 4.5, 'reieww inda": : ('review stans' 5.0, review date:: 22019-05-11 II 20:55:41", 're- view text" "Great local club to rising talent. A hidden gem. You need to experience this before it's too late (plans are to move to a bigger location in DTSB).. 'eevew stars' 're- view date:: "2018-11-12 20:05:5W. 'reviww text': "Went Satunday
great. Drinks cheap, plenty laughs a super incognito location. All the show was blast", 'review stars": 'review date': "2018. 10:30 02:26:26 'veviww text"; "The place is fun the comies are good food is good scating could be better but for free it's awesome"
Lauric Kilmartin
to he had next the nirport makes it
Machine-generuten article given review data
Comedy Hidcoway local business located Santa Barbura
variety services including performing arts, arts entertainment caterers, party event planning, comedl clubs. venues event spaces, event planning services, and nightlife. The business operates on Tuesdays from 6:00 PM to 1:00 AM, Fridays from 6:00 PM to 11:00 PM, and Satundass from 1:00 PM to 11:00 PM.
Acconding customer reviews, Comedy Hideoway highly recommended for its great local club atmosphere and rising talent performances. described as a hidden gem that provides a unique experieace. Some customers mentioned that vense plans to move to a bigger location in DTSB, indicating porential growth and expansion.
The business has rating of 4.5 stars, reflecting the positive expetiences of its customers. Reviewers uppreciated the affordable drinks, plenty of laughs, and the location's proximity Overall, Comedy Hideoway
airport. which adds charm. However, an enjoyable and experience with good food, talented comics, and casual ambiance. resommended destination for those looking to have a great time and enjoy live comedy shows in Santa Barbara.
mentioned that scating, could improved.
RLAHS roasoning path and oulput
## Step The overview antill states "business local business located Santa Barbura, CA' the structured about Comelly Hidesway
appears to have city as Sania Barbura and state as CA. Tacrefore, this statement scems accurate based off given structured data, At Step 2: The everviow articli states The business operales on Tuesdays from 6:00 PM to 1:00 AM. Fridays from 6:00 PM to 11:00 PM, and Saturdays from 1:00 PM 11:00 PM. Based off stractured data for hours shows Tuesday from 18:001:0 1:0" finterpreted
Tuesday from 6PM IAM), Friday from 6PM -PM equivalenily written commoo context would he till late night around midnight but given directly its till mentioned time Le 9pm), and Saturday 'rom" '1 -23 "iinerpretedd as Satunday from noen (or pu) onz o clock ail late night around eleven pm) Therefore, for Friday time meniooned directll is till late night but based off provided content
"18:0-23:0" (interpacted Friday
6PM 23(23) which equivalent saying Friday
here up until midnightt-til late night), for sake of accumey with stmictured provided we can confimm Tuesday Satudayy times accurate while for Friday noted up until late evening or more specificcllly up until mentioned time 1..9pm but article mentions up unil at least midnight so might nox strictly align with exact details provided. ty Step 3:The review snippet about restumrant includes positive aspects about food quality.taennt etc alongside attributes about restaurant heing casual which aligns well with structured details. Step4:The details provided regarding business characteristics aligned based off structure content. "hallucination list": ("Frdays from 6:00 PM to 11:00 PM"
noted up until mentioned value le 9pm though
A.8 BINARY IALLUCINATION DETECTION
We compare SFT and the proposed RLAHS on both binary and span detection tasks, and report the results in Table TO While both SFT and RL4HS perform equally well in terms of binary detec- tion, RL4HS clearly outperforms SFT in terms of fine-grained span detection, demonstrating the importance of RL for span detection task.
Table 10: Binary and span level hallucination detection results
Model
Binary Fl
Span F1
Model
Binary Fl
Span Fl
Qwen2.5-7B-CO.
56.2
16.2
Qwen2.5-14B-COT
62.0
27.3
Qwen2.5-7B-SFT
77.1
50.1
Qwen2.5-14B-SFT
77.9
55.4
Dwen2.5-7B-RL44H
77.1
55.9
78.0
58.3
A.9 RL4HS WITH LIMITED TRAINING DATA
We trained 2.ven2.5-IIstruuc and Qwen2.5- Innttsstttttttt-1455 models using one-third of the RagTruth training set. The results in Table show that RLAHS clearly outperforms SFT in low-data settings.
Table 11: RL4HS with limited training data: Both SFT and RL4HS models are trained with 1/3 Rag Truth training data.
Model
Summarization
Question Answering
Data2Text
Avg.
Qwen2.5-7B. Bren2.5-7B-RLAABE
44.4
48.6
50.5
48.8
51.1
52.4
57.1
54.2
Qwen2.5-14B-SFF
51.8
51.7
55.7
53.4
Owen2.5-14B-RLAHS
57.3
55.0
59.2
57.3
19

---

Published as a conference paper at ICLR 2026
A. EXPERIMENTS ON PSILOQA DATASET
To evaluate the generalizability of RL4HS, we train Qwen2.5--BBIInstruet using both SFT and RL4HS on the PsiloQA dataset (Rykov et al. 2025 While the SFT model achieves Span-F1 score of 73.0, the RLAHS trained model reaches 75.5. These results further demonstrate the superi- ority of RL4HS over SFT.
20


Reviews:
[{"review_id": "dHhso5bGZb", "reviewer": "Reviewer_NqfQ", "summary": "This paper propose RL4HS, a reinforcement learning framework that designed for hallucination span detection. The authors first conduct experiments comparing the pass@k performance, demonstrating that incorporating reasoning can be beneficial for span detection. Then, they train the model using GRPO with a span-level reward function. To address reward hacking issues caused by the imbalanced advantages between hallucination and non-hallucination cases, this work further introduce Class-Aware Policy Optimization (CAPO), which adjusts the advantages for non-hallucination predictions.", "questions": "See above section", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a systematic scaling analysis, highlighting the potential of reasoning in improving hallucination span detection. These findings and insights could be valuable for guiding future research.\n\n2. The study clearly identifies and analyzes the reward hacking issue caused by the imbalanced reward designs, and the proposed CAPO method offers a effective solutions.", "weaknesses": "1. Experiments are conducted exclusively on RAGTruth. It is unclear whether the proposed method generalizes to other hallucination datasets. There are other hallucination detection benchmarks with span-level annotations, such as FAVA, that could be included for a more comprehensive evaluation.\n\n2. Although the authors compare to several reasoning and proprietary models, hallucination-specific baselines is limited.\n\n3. The paper does not report case-level (or binary-level) results, so it remains unclear whether the span-level reward leads to consistent gains in overall performance.\n\n4. The SFT baseline seems suboptimal relative to the original RAGTruth paper, possibly due to inappropriate learning rates (1e-6 in appendix) or hyperparameters, which may underestimate supervised performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6RIlSQpUrK", "reviewer": "Reviewer_mScM", "summary": "The authors reframe hallucination detection as a reinforcement learning problem instead of a simple classification one. The authors first show that using Chain-of-Thought (CoT) reasoning before predicting hallucinated spans makes the model’s predictions more diverse — so when decoding multiple times (say, K attempts), at least one prediction tends to be correct. Building on this, they design RL4HS, which fine-tunes a model using Group Relative Policy Optimization (GRPO) with a span-level F1 reward to explicitly encourage reasoning that improves hallucination span localization. However, they find that if the input has no hallucination, directly giving reward = 1.0 biases the model toward always predicting “no hallucination.” To fix this, they propose Class-Aware Policy Optimization (CAPO), which scales the non-hallucination advantages (by 0.5) to balance rewards between classes. They also confirm that simple fixes like Dr.GRPO cannot solve this imbalance. Experiments on RAGTruth (covering summarization, QA, and data-to-text) show that RL4HS significantly outperforms supervised fine-tuning and existing reasoning models, proving that reinforcement learning with span-level rewards and in-domain reasoning is essential for robust hallucination detection", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper offers a novel reformulation of hallucination span detection as a reinforcement learning problem, which is conceptually original and well-motivated.\n- It carefully designs a span-level reward and introduces class-aware scaling to prevent reward hacking, showing thoughtful methodological innovation.\n- The motivation analysis with Span-F1@K clearly illustrates the benefit of reasoning diversity and provides strong empirical grounding.\n- Experimental results demonstrate substantial improvements over both supervised and reasoning baselines, highlighting the method’s effectiveness and significance.\n- The paper is clearly written and logically structured, making complex ideas easy to follow.", "weaknesses": "- The approach mainly focuses on the RAGTruth benchmark; it remains unclear how well it generalizes to other OOD data. But indeed the author shows good transferability among different three subsets under RAGTruth, using and holdout setting.\n- It could be beneficial to analyze and categorize what kinds of strategy the RL4HS model uses for producing more accurate hallucination span detection, by some human evaluation on the CoT paths.\n- Here are only one qualitative study examples shown in the paper. It would be good if the authors can provide more in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oOo1LG0lKG", "reviewer": "Reviewer_gYuN", "summary": "The paper proposes RL4HS, a reinforcement learning framework for hallucination span detection in large language models. It builds upon Group Relative Policy Optimization (GRPO) and introduces Class-Aware Policy Optimization (CAPO) to handle class imbalance in rewards. The authors claim that RL4HS enables better reasoning for hallucination localization and outperforms supervised fine-tuning and prior reasoning-based baselines on the RAGTruth dataset. The study attempts to connect reasoning, reinforcement learning, and span-level detection but provides limited novelty beyond adapting existing RL techniques.", "questions": "What specific novelty does RL4HS offer over GRPO beyond the scaling factor (CAPO)? Why is this sufficient for publication in a top-tier conference?\n\nHow is “reasoning” objectively measured or verified in this work? Are the CoT traces evaluated for correctness or interpretability?\n\nGiven that results rely solely on RAGTruth, can the authors demonstrate performance on unseen domains or datasets?\n\nHow are baseline models such as GPT-5 or Qwen3 controlled for fair comparison—are they fine-tuned under identical conditions?\n\nThe reported F1 improvements seem small; can the authors provide statistical significance tests or ablations to confirm that these are not due to random variation?\n\nCould similar gains be achieved simply with improved supervised training or calibration, without the complexity of reinforcement learning?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The topic—fine-grained hallucination detection—is relevant and timely, given the increasing importance of factual reliability in LLMs. The paper is well-organized, and the experimental setup is systematically described. The inclusion of span-level reward signals and the effort to analyze precision–recall imbalance show some awareness of practical issues in training reasoning-based detection models. The authors also provide qualitative examples to illustrate how reasoning might enhance model behavior.", "weaknesses": "Despite a clear structure, the work suffers from conceptual and methodological shallowness. The proposed RL4HS framework merely repackages existing GRPO methodology with a minor weighting adjustment, which can hardly be considered a significant algorithmic contribution. The claim that RL improves reasoning for hallucination span detection is weakly justified—there is no convincing evidence that the “reasoning” is genuinely learned rather than memorized through reward shaping. Experiments are conducted on a single dataset (RAGTruth), which raises concerns about generalizability. Furthermore, the comparison with GPT and Qwen models seems superficial and lacks control over model size, data exposure, and inference strategies. Many of the reported gains are marginal and could be attributed to overfitting or differences in fine-tuning procedures rather than the proposed RL method. The discussion of “in-domain reasoning” is vague and not theoretically supported. Overall, the paper feels more like an engineering report than a principled research contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oNtwVgO1iQ", "reviewer": "Reviewer_iny4", "summary": "The paper addresses hallucination span detection in large language models (LLMs), moving beyond binary detection to identify unsupported spans within generated text. The authors propose RL4HS, a reinforcement learning framework that leverages Chain-of-Thought (CoT) reasoning and introduces Class-Aware Policy Optimization (CAPO) to mitigate reward imbalance. Experiments on the RAGTruth benchmark (summarization, QA, data-to-text) show RL4HS outperforms both supervised fine-tuning and existing reasoning models, demonstrating the necessity of span-level rewards and in-domain reasoning for accurate hallucination detection.", "questions": "- Generalization: How does RL4HS perform on hallucination detection in domains outside RAGTruth (e.g., medical, legal, conversational AI)?\n- Annotation Efficiency: Can RL4HS be adapted to settings with limited or noisy span-level annotations? Is weak supervision feasible?\n- Deployment: What are the computational and practical challenges for deploying RL4HS in real-world LLM pipelines?\n- Failure Modes: What types of hallucinations or contexts remain challenging for RL4HS? Any observed systematic errors?\n- Comparison to Post-hoc Methods: How does RL4HS compare to post-hoc hallucination correction or filtering approaches in terms of accuracy and efficiency?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Novelty: First to train a reasoning-based hallucination span detector using RL with span-level rewards, addressing a gap in prior work focused on binary detection.\n- Technical Contribution: CAPO effectively balances precision and recall, overcoming reward hacking issues found in standard GRPO.\n- Empirical Rigor: Extensive experiments on RAGTruth across multiple tasks, withcomparisons to strong baselines (SFT, multi-view attention, proprietary reasoning models).\n- Insightful Analysis: Ablation studies and case analysis convincingly show the benefits of in-domain reasoning and span-level reward optimization.", "weaknesses": "- Generality: Evaluation is limited to RAGTruth and a few CNLG tasks; broader applicability to other domains or real-world LLM outputs is not demonstrated.\n- Model Scale: While RL4HS outperforms larger models, results for very large-scale proprietary models (e.g., GPT-5) are not fully explored.\n- Complexity: The RL training setup (GRPO, CAPO) adds complexity and may be challenging to reproduce or deploy in production settings.\n- Limited Error Analysis: The paper could benefit from deeper qualitative analysis of failure cases and limitations of RL4HS, especially in ambiguous or noisy contexts.\n- Data Requirements: Reliance on span-level annotated data (RAGTruth) may limit adoption, as such datasets are rare.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)