VeriTrail: Closed-Domain Hallucination Detection with Traceability
Authors: Dasha Metropolitansky
Jonathan Larson
Venue: ICLR 2026 Poster (2026)
Keywords: hallucination detection
faithfulness
fact-checking
traceability
provenance
error localization
Primary area: interpretability and explainable AI
TL;DR: We introduce VeriTrail, the first closed-domain hallucination detection method designed to provide traceability for processes with any number of generative steps, and demonstrate that it outperforms baseline methods in hallucination detection.

Abstract: Even when instructed to adhere to source material, language models often generate unsubstantiated content – a phenomenon known as “closed-domain hallucination.” This risk is amplified in processes with multiple generative steps (MGS), compared to processes with a single generative step (SGS). However, due to the greater complexity of MGS processes, we argue that detecting hallucinations in their final outputs is necessary but not sufficient: it is equally important to trace where hallucinated content was likely introduced and how faithful content may have been derived from the source material through intermediate outputs. To address this need, we present VeriTrail, the first closed-domain hallucination detection method designed to provide traceability for both MGS and SGS processes. We also introduce the first datasets to include all intermediate outputs as well as human annotations of final outputs’ faithfulness for their respective MGS processes. We demonstrate that VeriTrail outperforms baseline methods on both datasets.

Published as a conference paper at ICLR 2026
VERITRAIL: CLOSED-DOMAIN HALLUCINATION DETECTION WITH TRACEABILITY
NALLUIIAATOOO
Dasha Metropolitansky, Jonathan Larson Microsoft Research
(dasham, jolarso, @mi crosoft.ccmmmm
ABSTRACT
Even when instructed to adhere to source material, language models often generate unsubstantiated content a phenomenon known "closed-domain hallucination.." This risk is amplified in processes with multiple generative steps (MGS), compared processes with a single generative step (SGS). However, due to the greater complexity of MGS processes, we argue that detecting hallucinations in their final outputs is necessary but not sufficient: it is equally important trace where hallucinated content was likely introduced and how faithful content may have been derived from the source material through intermediate outputs. address this need, we present VeriTrail, the first closed-domain hallucination detection method designed to provide traceability for both MGS and SGS processes. We also introduce the first datasets to include all intermediate outputs well human annotations of final outputs' faithfulness for their respective MGS processes. We demonstrate that VeriTrail outperforms baseline methods both datasets.
INTRODUCTION
Language models (LMs) are widely used to generate content based on source text. However, even when instructed to adhere to the source, LMs are known to produce unsupported content phe- nomenon known as "closed-domain hallucination" (OpenAl al., 2024). Detecting hallucination important in many settings (e.g.. a doctor seeking guidance from medical literature, lawyer summa- rizing precedent cases, customer service agent answering questions based on policy documents). Closed-domain hallucination detection is also known "faithfulness evaluation" use both terms (as well as "hallucination detection" " for brevity) interchangeably throughout this paper.
Processes that use LMs to generate content based source material can be divided into two categories: processes with a single generative step (SGS) and processes with multiple generative steps (MGS). In SGS processes. the LM produces only final output, without generating any intermediate outputs. One limitation of SGS processes is that they may be less reliable for long documents or large collections of documents: if the source material exceeds the model's context window, truncation or retrieval is required, increasing the risk of information loss.
Given the limitations of SGS processes, applications where intermediate outputs generated by the LM are used as inputs to subsequent steps. In MGS processes, since the source material (or the task) can be split into smaller parts, truncation or retrieval may not be necessary, reducing the risk of information loss. However, MGS processes are more susceptible to hallucination, as each step presents an additional opportunity for errors arise and propagate. Therefore, as the use of MGS processes accelerates, effective hallucination detection methods become increasingly important.
LMs increasingly rely on MGS processes,
We argue, however, that detection alone is not enough. In many settings, we need to understand how the output may have been derived from the source (provenance) and where errors may have been introduced (error localization). Provenance helps users verify and trust the output, while emoor localization is critical for addressing hallucinations and understanding which parts the process most error-prone. We refer to provenance and error localization collectively traceability. The transparency enabled by traceability especially important for MGS processes due to their complexity.

---

Published as a conference paper at ICLR 2026
Prior works frame faithfulness evaluation assessing whether given LM output is supported by the source material and do not distinguish between intermediate and final outputs (see $ for examples). For SGS processes, this simplification is reasonable: since there is only one generative step, evaluating its output against the source text is sufficient both to assess faithfulness and to provide traceability. In MGS processes, however, while can still evaluate the final output against the source text to determine whether hallucination occurred, we cannot achieve traceability without utilizing the intermediate outputs.
A simplistic approach to traceability is to check the final output against each individual intermediate output. However, this approach can be prohibitively expensive when there are many intermediate outputs (e.g., one of the processes studied in this paper produced over 100,000 intermediate ouiputs). The simplistic approach also fails when the final output is based on a combination of intermediate outputs. For instance, suppose the final output states: " "Company X acquired two startups in 2020 as part of its expansion into healthcare." " One intermediate output mentions single acquisition in 2020; second references another acquisition in 2020; and a third describes Company X's s overall acquisition strategy in 2020 as focused on expansion into healthcare. Taken individually, none of these intermediate outputs supports the claim but together, they do. These limitations highlight the need for an approach to traceability that goes beyond evaluating outputs in isolation.
This paper makes the following contributions:
1. We propose conceptual framework that provides a unified representation of generative processes for the purpose of faithfulness evaluation.
2. We introduce VeriTrail, the first closed-domain hallucination detection method to provide traceability for MGS and SGS processes. We also demonstrate that VeriTrail outperforms baseline methods in hallucination detection while remaining cost-effective.
3. We construct FABLES+ and DiverseSumm+, the first datasets to include all intermediate outputs well as human annotations of final outputs' faithfulness for their respective MGS processes
CONCEPTUAL FRAMEWORK
We define a non-generative step as an operation that outputs only unmodified text spans from the input (c.g.. noun phrase extraction using spaCy; Honnibal (poniil et al. 2020). In contrast, generative step may modify the input text or introduce new information. Steps involving LMs are typically generative, although exceptions exist (e.g., constrained decoding; [Geng et al. 2024).
generative process is a sequence of steps that produces a final output from a set of source documents D and includes at least one generative step. At each step, the input consists of text spans from D and/or outputs from earlier steps.
Generative processes be categorized containing either single generative step (SGS) multiple generative steps (MGS). An example of an SGS process is SetrievallAAumentee Generation (RAG) where non-geneee non-generative retrieval system selects subset of the source material to provide as input to an LM (Lewis et al. 2020). Although RAG involves two steps, it is an SGS process because only one step generative. We provide examples of MGS processes in Appendix A| including two processes used in our experiments: hierarchical summarization (Wu et al. 202 Chang et al.. 2023)
and GraphRAG (Edge et al.) 2025).
We model generative process as a directed acyelic graph (DAG) G (V, F).. where each node e E V represents text span, either originating from D or produced by a step. Each directed edge (u, v) E E indicates that node was included an input to the step that produced node
For any node v, we define its source nodes as src(v) = {u E V (u, v) E E} - the set of nodes used as input to produce v. Root nodes Vn C V have no incoming edges (i.e., V E Vo, sre(e) = 0) and correspond to text spans from D. The terminal node 8" E V has no outgoing edges and represents the final output. We refer to any node that is neither root node nor the terminal node as an intermediate node.
The datasets will be released at httttssp////aat http:///aaaaaamerrrrrrraat dattaset.
2

---

Published as a conference paper at ICLR 2026
We define function stage: - N that assigns stage to each node, such that for each edge (u, v) E, stage(v) 2 stage(u). Conceptually, the stage reflects a node's ' position in the generative process. Root nodes assigned the minimal stage, and the terminal node is assigned the maximal stage. For intermediate nodes, some processes (e.g., GraphRAGi have clearly defined stage assignments due to distinct step types, while others (e.g., hierarchical summarization) do not have single "correct" assignment. [Appendix B.I.I and Appendix B.2. describe the stage assignment procedures used in our experiments.
Faithfulness evaluation aims to determine whether the terminal node v* is supported by the root nodes that have a path to v². Intuitively, these root nodes represent the subset of the source material that could have contributed to the final output. To enable fine-grained evaluation, we follow prior work (e.g., Min et al., 2023, Hu al., (025) decomposing the final output into set of factual where each claim is self-contained, verifiable statement. Each claim
claims C (
e E C' is assigned one of the following verdicts:
1. Fully Supported:
The source text strongly implies the entire claim. A careful reader would naturally infer the claim without relying on assumptions external knowledge.
2. Not Fully Supported: At least one part of the claim is not strongly implied by the source text. This may occur because the source text contradicts the claim, strongly implies it is false, only weakly implies it, or does not address it at all.
3.
Inconclusive: The source text ambiguous or conflicting such that both "Fully Supported" and "Not Fully Supported" verdicts are possible, with neither clearly favored.
3 VERITRAIL
In this section, we describe VeriTrail, our new closed-domain hallucination detection method.
3.1 HALLUCINATION DETECTION PROCESS
VeriTrrail has three main inputs:
1. A DAG representing completed generative process (as described in 822 where each node is assigned a unique ID;
2. A hyperparameter q, which specifies the number of consecutive "Not Fully Supported" verdicts that will trigger termination of the hallucination detection process; and
3. A set of factual claims C, extracted from the terminal node v* of the DAG.
In our experiments, we obtain by applying Claimify (Metropolitansky & Larson 2025). a claim extraction method, v². VeriTrail evaluates each claim C' independently from the other claims.
The subsections below describe the main steps in VeriTrail's 's hallucination detection process, which assigns verdict to each claim. An example is shown in Figure I, and the full procedure is provided in Algorithm Appendix C.T All prompts are in Appendix C.2
3.1.1 SUB-CLAIM DECOMPOSITION
claim may contain multiple distinct parts, which refer as "sub-claims." For example, the claim "Company X acquired two startups in 2020 as part of its expansion into healthcare" can be split into: (1) Company X acquired two startups in 2020, and (2) these acquisitions were part of its expansion into healthcare. Identifying sub-claims is important because, as noted in S2 all parts of a claim must be supported in order to justify a "Fully Supported" verdict.
To identify sub-claims for the claim c, VeriTrail applies Claimify's 's Decomposition module, which attempts to rewrite the input text as a set of simpler, independently verifiable statements. If multiple sub-claims are returned, they are added to a queue for further decomposition. If only a single sub- claim returned, it treated as final and not processed further. To prevent infinite loops, VeriTrail skips previously processed sub-claims and enforces maximum number of decomposition attempts Sub-claims are retained as context for the following steps but are not verified directly.
We this maximum 20 in our experiments.
3

---

Published as a conference paper at ICLR 2026
3.1.2 EVIDENCE SELECTION
Next, VeriTrail identifies sre(u") - the set of nodes used as imput to produce the terminal node. Each source node is segmented into sentences using NLTK's sentence tokenizer (Bird & Loper 2004). and each sentence is programmatically assigned unique ID. An LM is then prompted to select all sentences that strongly imply the truth or falsehood of c or any of its sub-claims (see the prompt in Appendix C.2. I for details).
If the sentences do not fit within a single prompt, they are split across multiple prompts, which are processed parallel to minimize latency. By default, each prompt includes as many sentences as can fit within the LM's context window, although this limit is configurable. See SeeeAppp E.2 for an ablation study on the effect of the input size limit.
The LM returns the IDs of selected sentences and summary of their combined content. If returned ID does not match programmatically assigned ID, it is discarded; otherwise, mapped to its corresponding sentence. This approach guarantees that the sentences included in the evidence trail are not hallucinated. Additionally, identifying specific sentences that support or refute the claim is arguably more informative than classifying entire nodes relevant not. It also simplifies human verification, since reviewing a selection of relevant sentences requires significantly less effort than reading full passages.
3.1.3 VERDICT GENERATION
If no sentences selected during Evidence Selection, the claim e assigned a "Not Fully Supported" verdict. Otherwise, an LM is prompted to assign one of three verdicts ("Fully Supported," "Not Fully Supported," or "Inconclusive") based on the Evidence Selection results (see [Appendix C.2.2).
Sentences selected during Evidence Selection are not included directly in the Verdict Generation prompt, they may be ambiguous when used out of context. For example, if the claim is "John Smith was the CEO of Company X," and a selected sentence is "He served as its CEO from 2006-2010," it unclear whether "He" refers to John Smith and "its" Company X. To avoid such ambiguities, VeriTrail determines the input for the Verdict Generation prompt based on the nodes from which the sentences were selected: if the node is a root node, its full content is included; otherwise, the summary generated during Evidence Selection is used.
As in Evidence Selection, an input size limit can be specified for the Verdiet Generation prompt. However, unlike in Evidence Selection, Verdict Generation requires all inputs (i.e., the evidence) to fit within a single prompt. Therefore, if the input size limit is exceeded, VeriTrail reruns Evidence Selection this time on the previously selected evidence until either (a) the evidence fits within the limit, or (b) a maximum number of reruns is reached (where this maximum is configurable). condition (b) occurs, then the largest subset of evidence that fits within the limit is used.
3.1.4 CANDIDATE NODE SELECTION AND TERMINATION
Recall from $ 3.1.2 that VeriTrrall selects the source nodes of the terminal node, sre(u"), as input for the initial round of Evidence Selection. Once Evidence Selection and Verdict Generation based on sre(v*) are complete, VeriTrail selects new set of nodes to use as input for the next round of Evidence Selection. For ease of reference, we call this set the "candidate nodes." " The candidate nodes are selected as follows:
If the latest verdict was "Fully Supported" or "Inconclusive": include the source nodes of all nodes from which sentences were selected during the latest round of Evidence Selection.
If the latest verdict was "Not Fully Supported"; include the source nodes of all nodes verified in the latest iteration - not just those that yielded evidence 5
'In the "Not Fully Supported" case, the primary motivation for verifying the source nodes of all previously verified nodes, regardless of whether they yielded evidence, is to reduce the risk of false positives (i.e., incorrect "Not Fully Supported" verdicts). False positives can caused by Evidence Selection failing to select evidence from nodes that support the claim. If only the source nodes of evidence-yielee yielding nodes are considered for further verification. nodes overlooked by Evidence Selection will be excluded from future iterations, perpetuating the error. By including the source nodes of all previously verified nodes, TeiiTrail increases the likelihood of recovering missed supporting evidence for the claim.
4

---

Published as a conference paper at ICLR 2026
Include any root nodes from which evidence selected in previous iterations. These nodes are not reprocessed during Evidence Selection but are used in Verdict Generation +
To avoid redundant processing, exclude any candidate nodes that were verified in earlier iterations, except for the root nodes described above.
The hallucination detection process terminates if any of the following conditions is met:
1. The candidate nodes consist only of previously verified root nodes from which evidence was selected;
2. There are candidate nodes, meaning that the root nodes never reached that none the root nodes yielded evidence; or
3. The number of consecutive "Not Fully Supported" verdicts has reached q.
Under condition 1, the latest verdict is deemed final. Under conditions 2 and 3, the final verdict is to "Not Fully Supported." If none of these conditions is met, the steps described in $3.1.2 $3.1. $.3.3
and 3.1.4 are repeated using the newly identified candidate nodes as input for the next round of Evidence Selection.
Generative Process
VeriTrail's Hallucination Detection Process
Claim from
11 Jane loves graphs and logic
11
Final summary
Sub-cleims Jane loves graphe; Jane loves logic
Stage Si Summariss of chapter sum maries
10
Iteration Check
9 and
10
6
Stage Chapter summaries
Evidence:
Sentence "Chapters 3 3ndd 4ocus Jane, an onthusiast of grapps and logic."
Verdict:
Fully Supported
2
3
Book chapters
Iteration 2: Check
and
Evidence:
N/A
Sentense 16: "Jane tald John that she is pesaionats about abautgraphs and loglo."
Verdict:
Fully Supported
Iteration 3: Check
Evidencs: Sentonce 81: "ovooo logic, Janc laughed norvoualy, 'Just kidslnng - my true passion graphs."
Final Verdict: Not Fully Supported
Figure 1: Left: Hierarchical summarization as a DAG. Circles represent nodes; arrows represent edges. Right: VeriTrail's hallucination detection process. The evidence trail includes sentence 2 from node 10, sentence from node 8, and sentence 81 from node 4. Evidence summaries are not shown.
3.2 TRACEABILITY
For each claim extracted from the terminal node v", VeriTrail returns: (1) final verdict, along with the LM's reasoning; (2) all interim verdicts; and (3) evidence trail composed of (a) selected sentences with their corresponding node IDs and (b) generated summaries from all iterations of Evidence Selection.
Collectively, these outputs provide traceability as follows:
'For example, recall the claim "Company X acquired two startups in 2020 part of its expansion into healthcare." " Suppose that in the latest iteration, the claim deemed "Fully Supported" based (1) intermediate node stating that Company acquired two startups 2020, and (2) a root node stating that Company X's acquisition strategy focused on healthcare expansion. In the next iteration, we need verify the source nodes of the intermediate node. Assume the source nodes contain the same information the intermediate node. If we exclude the already-verified root node from Verdict Generation, the information about healthcare expansion would be lost, and the claim would be incorrectly labeled "Not Fully Supported."

---

Published as a conference paper at ICLR 2026
Provenance. For claims whose final verdict is "Fully Supported" or "Inconclusive," the evidence trail documents path through the intermediate nodes to the root nodes.
Error Localization. For claims deemed "Not Fully Supported," the interim and final verdicts are used to identify error stage(s) the stage(s) where the unsupported content was likely introduced.
To find the error stages, VeriTrail identifies iteration n, the last iteration where the claim received an in- terim "Fully Supported" verdict prior to the final "Not Fully Supported" verdict. Let Ve(n) denote the set of nodes from which at least one sentence was selected during the Evidence Selection step in itera- tion The error stages then defined the stages of the nodes from which sentences selected during Evidence Selection For example, in Figure I, the last iteration that received "Fully Supported" verdict before the final "Not Fully Supported" verdict was Iteration 2. In Iteration 2, evidence was selected only from node 8, which belongs to stage 2. Therefore, the error stage stage 2
iteration n, excluding root node: (stago(u)
E (n), & Vo)-
There are three scenarios where a claim never receives a "Fully Supported" verdict. First, the first iterations returned "Not Fully Supported," causing verification terminate. In this case, the error stage is stago("") (i.e., the unsupported content was introduced in the final output). Second, the interim verdicts were a mix of "Inconclusive" and "Not Fully Supported" verdicts. Third, all interim verdicts were "Inconclusive." In the second and third scenarios, an error stage cannot be identified.
4 EXPERIMENTS
4.1 DATA
Since detecting hallucinations in MGS processes is increasingly important yet underexplored, we focused on evaluating VeriTraills performance across diverse MGS processes. With regard to the source material, we targeted long documents and large document collections (i.e., >10KK tokens), where hallucination detection is especially challenging and MGS processes tend to be most valuable.
While many existing datasets provide human annotations of faithfulness for LM-generated outputs, few are based on MGS processes, and we did not find any that include all intermediate outputs. To address this gap, constructed two new datasets by augmenting prior work:
1. FABLES+ is based on FABLES (Kim et al. 2024). dataset of book summaries generated via hierarchical summarization. Since the original dataset did not preserve all intermediate outputs required construct the DAG input for VeriTrail, we re-genegee re-generated summaries for 22 books with an average length of 1 118K tokens. We extracted 734 claims from the final summaries using Claimify. 48% of the extracted claims restated information that had already been deemed faithful in the original dataset. As result, we labeled these claims as faithful without further annotation. We manually annotated the remaining claims.
2. DiverseSumm+ based on DiverseSumm (Huang et al. 2024), dataset of news stories (e.g., the Rursia-Ukraine conffict), each linked to 9-10 articles and set of questions answered by multiple articles. We retained 148 stories, linked to 1,479 articles with a cumulative length of 1.19M tokens. From these stories, generated answers using GraphRAG over the full article set. We then extracted 560 claims from the answers using Claimify. To label the claims' faithfulness, we recruited four annotators through Upwork Annotators were given access to the 9-10 articles associated with each claim, while one of the authors served fifth annotator with access to the full article set. For 87% of claims, the associated 9-10 articles were sufficient to determine faithfulness; 13% required consulting additional articles.
sampled 20 questions and
Full details for both datasets are provided in Appendix
$ As defined in the root nodes are the ground truth, so they cannot be 'Ver'Trail may return multiple error stages. For example, in GraphRAG, the source nodes for certain stage 4 nodes may include both stage 2 and stage 3 nodes (see Appendix B.2. for stage definitions). If Ve(n) includes nodes from both stages, VeriTrail lists both potential error stages since it is not possible to attribute the hallucination just onc.
of errors.

---

Published as a conference paper at ICLR 2026
4.2 BASELINE METHODS
We compared VeriTrail against three types of methods commonly used to evaluate faithfulness and capable of processing large source texts:
1. Natural Language Inference (NL.). IIJI methods assess whether claim is entailed by a source document. We selected three methods that represent different strategies for handling large source documents:
(a) INFUSE (Zhang et al. 2024a) splits the document into sentences and ranks them based on bi-directional entailment (i.e., the probability that the sentence entails the claim or vice versa). It constructs an evidence set by adding top-ranked sentences, stopping when the predicted probability that the claim is neutral with respect to the evidence set begins to increase. The entailment probability is computed over the final evidence set.
(b) AlignScore (Zha et al. 2023) splits the document into small chunks (-350 tokens) and computes the probability that each chunk entails the claim. We aggregated these probabilliiees using the mean of the top-k values, testing & [1]
(c) Llama-3.1- -lama-3.1-Bespoke- Pespokk-MiniCheede Score, but uses much larger chunks (-32K tokens). For each claim, we computed the mean of the top-k entailment probabilities, testing k E [1, 5) for FABLES+ and k E [1, 15] for DiverseSumm+.
(Bespoke Labs 2024) operates similarly to Align-
To convert entailment probabilities into binary labels, we tested thresholds т E (0.5, 0.6, 0.7, 0.8, 0.9}: claim was labeled "Fully Supported" if the entailment proba- bility was at least T, and "Not Fully Supported" otherwise. See Appendix Foro details.
Finally, note that Zha al. (203) Bespokk Labs (2024) reported that AlignScore and 1. -Bespooe-MinnCheee and Claude-3.5-Soneee respectively) on standard factual consistency benchmarks (e.g.. SummEval, QAGS, and LLM--Agrefact), making them particularly strong baselines.
outperformed much larger models (e.g., GPT-4
2. Detrieval-Augmenterr Generation (RAG). We reused the document chunks created during dataset construction (see Appendix B. and Appendix B.2.I). Claims and chunks were embedded using OpenAl's text eemeeddbddd Bolliin--3 3-g-3-large model. For each claim, we retrieved the top-k most similar chunks, which were passed to an LM for verdict generation.
For retrieval, we used Faiss' k- nenearest neighbors search (Douze et al. 2025) with L2 distance, testing k: E (1,3, 5, 10, 15, 20, 25, 30).
3. Direct Verification Using I Long-Contert LMs. We provided the source document(s) full book for FABLES+ and all 1,479 articles for DiverseSumm+ direelly to an I.M for verdict generation. We tested two models with long context windows: Gemini 1.5 Pro (2M tokens) and GPT-4.1 Mini (-IM tokens). Since the DiverseSummt articles exceeded GPT-4.1 Mini's context window, each time evaluated claim, we randomly shuffled the articles and selected the largest subset that fit (typically- -80%).
4.3 RESULTS
We evaluated VeriTrail and baseline methods in two settings: hard prediction, where methods output a single label per claim ("Fully Supported" or "Not Fully Supported" and soft prediction, where methods produce continuous score representing the probability that claim is "Fully Supported." Soft prediction results are reported in Appendix E.3
We report results for AlignScore, INFUSE, Llama-3.1- Desspoke-MiniCheck... and RAG using the best-performing hyperparameter configuration for each dataset; results for all configurations are provided in Appendix E.4 For RAG and Direct Verification, we used VeriTrail's Verdict Generation prompt to ensure that any performance differences are not attributable to prompting. We also tested an alternative prompt from the FABLES paper, including for VeriTrail. As shown in Appendiix E.5 the alternative prompt yielded worse results for all methods except GPT-4.1 Mini on FABLES+
"Claims assigned an "Inconclusive" verdict by any method (4% for FABLES+ and 8% for DiverseSumm+) were excluded from the hard prediction evaluation. We also tested treating "Inconclusive" verdicts either always correct always incorrect and observed only marginal differences the reported metrics.
7

---

Published as conference paper at ICLR 2026
Table shows hard prediction results for FABLES+ and DiverseSumm+t respectively. Due to class imbalance (see Appendix B), we used macro Fi and balanced accuracy as our primary met- rics. For VeriTrail, report results for q 1 and q 3: results for additional q values analyzed in Appendix E.6 All Veri'rral and RAG results in Table 1 were produced using OpenAl's gpt-40-2024-08- model; results for other models are included in Appendix E.7
On our primary metrics, VeriTrail outperformed all baseline methods for both datasets and all models tested, except for mistralllargee 2411, where VeriTrail had the highest balanced accuracy but not the highest macro Fj. VeriTrrail also outperformed all baselines in the soft prediction setting (Appendix E.3).
Hard prediction results (%) for the FABLES+ (F) and DiverseSumm+ (D) datasets. We report macro F1. balanced accuracy (Bal. Acc.), and class-specific precision and recall for fully supported (FS) and not fully supported (NFS) claims. For RAG, AlignScore, INFUSE, and Llama- 3.l-Bespoke-MimiCheee
Table 1:
we report the best-performing configuration by macro F1: RAG uses k = 15 for both datasets; AlignScore uses 1, T 0.6 for FABLES+ and & = 1, = 0.9 for DiverseSumm+; INFUSE uses = 0.5 for both datasets; uses k 1, t 0.5 for FABLES+ and
(denoted "Bespoke-Me Manoook--MiiChhe---BB--
Llama-3. 1.taaa...-Besppe (Bespoke-MiniCheee DiverseSumm+. Bolded values represent the highest score in each column.
= 0.5 for
Macro F1
Bal. Acc.
Precision
Recall
Precision
Recall NPS
Method
D
D
VeriTrail (g
74.0
76.6
83.0
97.5
95.8
76.2
44.1
55.1
86.5
89.8
VeriTrail (g 3)
84.5
79.5
83.6
76.3
95.4
87.1
96.4
96.7
75.6
84.5
70.8
55.9
RAG
69.6
75.1
76.5
74.0
94.6
86.7
83.3
90.5
39.6
66.4
69.8
57.5
7B
62.2
72.1
69.0
69.4
92.6
83.8
77.7
95.4
29.9
75.3
60.4
43.3
Gemini 1.5 Pro
61.1
49.8
60.8
57.6
89.3
82.2
90.3
45.1
33.7
29.4
31.2
70.1
GPT-4.1 Mini
60.7
62.9
61.5
80.3
93.8
60.7
17.7
29.1
AlignScore
59.6
60.4
67.5
62.7
92.4
82.8
73.6
70.3
26.8
37.6
61.5
55.1
INFUSE
20.0 40.5
59.5
50.1
92.9
100.0
36.8
0.30
17.0
24.6
82.3
100.0
4.4 ANALYSIS
To supplement our main findings, we include the following analyses in the Appendix:
Appendix D
We analyze VeriTrail's computational cost. We show that, despite sig- nificantly larger verification burden, VeriTrail outperformed the baseline methods while remaining cost-effective.
Appendix E.
We present the results of an ablation study to identify which aspects of VeriTrail's s design contributed most to its performance gains.
Appendix G
We analyze error cases
VeriTrail's limitations.
Appendix H
We examine the distribution of error stages identified by VeriTrail to understand where hallucinations tend to arise in the processes we studied.
RELATED WORK
Open-Domain vs. Closed-Domain Hallucination. Hallucination can arise in either closed-domain or an open-domain setting. In closed-domant setting, model is given source material and is expected to remain faithful to it. In an open-domain setting, no source material provided, so the model relies its parametric knowledge when generating output. VeriTrail is designed specifically for the closed-domain setting.
Reference-Free Reference-Basee Detection. Hallucination detection methods can be categorized as reference-fin free or reference-basee Reference-free methods estimate the model's confidence in its output without consulting any reference text. Although traditionally used in open-domain settings, recent works have applied reference-fie methods to closed-domain hallucination as well (e.g..
Chuang al. 2024
Sun et al. 2025 Ridder & Schilling. 2025). Because reference-free methods
8

---

Published as a conference paper at ICLR 2026
rely model internals (e.g., attention maps), they may not be usable for claims extracted from Already-generatee outputs, and they are often incompatible with closed-source models. In contrast, reference-biee baseee--bbdddee methods compare the model's output directly to reference text, they are model- agnostic and can be applied to already-generann
outputs. In the open-domain setting, reference- based methods must retrieve reliable external evidence serve the reference text Be.g--Chenn et al.) 2024 Wei et al., 2024). In the closed-domain setting, the reference text is simply the source material that the model was given. wide range of closed-domain, reference-based methods have been proposed,
including fine-tuning models for question generation and answering (Durmus et al. 2020 Wang et al.
2020 Scialom et al., 202T) and using LMs as verifiers (Liu et al. 2023 Es et al. 2024 Saad-Fall Falcon
et al. 2024). VeriTrail is also reference-based method.
Evaluation of Intermediate Outputs.
To the best of our knowledge, no existing closed-domain accounts for the full structure of
hallucination detection method - reference-free or reference- the generative process. Instead, prior methods evaluate LM outputs in isolation, which is insufficient to achieve traceability for MGS processes. A related area of work that involves the analysis of intermediate outputs is the evaluation of LM-generated reasoning chains (i.c., sequences of steps leading to final answer). However, existing methods (e.g.. Hao et al. 2024] Paul et al. 2024) focus on relatively simple chains (e.g., those that fit within a single prompt) and may not generalize to the longer, more complex processes addressed in this paper.
Generative Processes as DAGs, Several prior works have modeled generative processes as DAGs.
In LangGraph (LangChain 2025 nodes represent generative steps and edges denote execution order: an edge from u to U means was executed after u, but not necessarily that u's output was used by e. Heterogeneous Swarms (Feng et 2025) adopts a similar structure, but requires that u's output be used as input to v. In MacNet (Qian et al. 2025), both nodes and edges represent generative steps: an edge from u to U means u produced an output, another step provided feedback, and v refined the output based on the feedback. In VeriTraillss 's representation of generative processes, nodes represent text spans, not steps, and edges capture input-output relationships. For example, LM summarizes five document chunks in single step, VeriTrail represents the process as six nodes one for the summary, and one for each input chunk - rather than a single node.
CONCLUSION
In this paper, we address two increasingly important challenges: effective hallucination detection and traceability for processes with multiple generative steps. We introduce VeriTrail, the first closed- domain hallucination detection method that not only evaluates the output's ' faithfulness to the source material, but also provides provenance and eтrоr localization. For outputs deemed faithful, VeriTrail constructs an evidence trail that traces the path to the source material through intermediate outputs. For unfaithful outputs, it identifies the stages where hallucination likely occurred. As result, VeriTrail provides transparency and actionable insights into generative processes.
To evaluate VeriTrrail's performance and support future work traceability, we created FABLES+ and DiverseSumm+, the first datasets to include all intermediate outputs of processes with multiple generative steps as well as human-annotated faithfulness verdicts. Across both datasets, which span diverse generative processes and source document types, VeriTrail outperformed strong baselines in hallucination detection while remaining cost-effective.
REPRODUCIBILITY
The paper includes all necessary information to re-implement VeriTrail, including detailed descrip- tions of the method and hyperparameters (63) complete algorithm specification (Appendix C.I). and all prompts (Appendix C.2). We also specify all settings for the baseline methods evaluated in our experiments (S 4.2, Appendix F) to support reproducibility. Finally, we plan to release the FABLES+ and DiverseSumm+ datasets at [nttps::.............. : //aka. ms /wws/viiitaaii datasets
9

---

Published as a conference paper at ICLR 2026
REFERENCES
Nishant Balepur, Alexa Siu, Nedim Lipka, Franck Dernoncourt, Tong Sun, Jordan Lee Boyd-Graber, and Puneet Mathur. MoDS: Moderating a mixture of document speakers to summarize debatable queries in document collections. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 465-491, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176- 189-6.
URL /attps:/aacaane 2ranthooloo..orr/22222.. 2025. maaci-long 207
Bespoke Labs. Bespoke-minicheck-77. 2024.
URL httpss//hugginnfaacecco
Minicheck -B
Steven Bird and Edward Loper. NLTK:
The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pp. 214-217, Barcelona, Spain, July 2004, Associ- /tt//clannnt
ation for Computational Linguistics. URL htt.lpsss//cllanhhooooo..oooo.. wtwp:000-0...//0030000000000
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong. Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo. Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li. Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma. Wenchang Ning. Linke Ouyang, Jiantao Qiu, Yuan Qu. Fukai Shang. Yunfan Shao, Demin Song. Zifan Song. Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang. Guoteng Wang. Jiaqi Wang. Jiayu Wang, Rui Wang. Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jía Yu, Jing Yu, Yuhang Zang, Chuyu Zhang. Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang. Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report. 2024.
Santiago Castro. Fast Krippendorff: Fast computation of Krippendorff's alpha agreement measure. https::/gittub..co com fing ude wwdelarrfast...o wrtpsttt-ripppedpddoroo 2017.
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit lyyer. Booookscore: A systematic exploration of book-length summarization in the era of llms. arXiv preprint arXiv:2310.0077. 2023.
Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. Complex claim verification with evidence retrieved in the wild. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 3569 Mexico City, Mexico, June 2024. Association for Computational Linguis- tics. doi: 1 1/t///222222222221/1/2 196. URL https: httpss//aaaaathooogooooo wrw.ooggg///o 2024 dlacclllone 1el.oog..15.. 1967|
Yung-Sung Chuang, Linlu Qiu, Cheng-Yr Yu Hsieh, Ranjay Krishna, Yoon Kim, and James R. Glass. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1419-1436, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 18533/v 00/55//112222444 122p://.22....mmmmam...... URI. httphtttts//ccaaaa http://cclanthhhooo woy.aaaooooooyoooooooooo ogy.or//2220 emn p-maa-mm--mmm 847
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy. Beerreemmmnuul Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library, 2025. URL https
//arxiv. //arxiv.org/eeeeee abs / 2401 08281
Esin Durmus, He He, and Mona Diab. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5055-5070, Online, July 2020. Association for Computational
10

---

Published as a conference paper at ICLR 2026
Linguistics. doi: 10.18653/w1/222..ad main.454. URL wtl..ttss//ac.aattooooor../ acl aaaiminnm 4547
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach to query-focuse summarization, 2025. URL httph:pss:/ http//axxvvv.rr.
2404 16130
Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. RAGAs: Automated evaluation of retrieval augmented generation. In Nikolaos Aletras and Orphee De Clereq (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp. 150-158, Julians, Malta, March 2024. Association for Computational Linguistics.. doi: 10.
URL
Shangbin Feng, Zifeng Wang, Palash Goyal. Yike Wang, Weijia Shi, Huang Xia, Hamid Palangi, Luke Zettlemoyer, Yulia T'vetkov, Chen-Yu Lee, and Tomas Pfister. Heterogeneous swarms: Jointly optimizing model roles and weights for multi-llm systems, 2025. URL UtR.httpss//arxi
org/abs 22tp. 04510
Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Grammaa-coostrained decoding for structured nlp tasks without finetuning, 2024. URL http:::/aaa.....gg/////o
13971
Max Glockner, Ieva Staliünaité, James Thore, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. AmbiFC: Fact-checking ambiguous claims with evidence. Transactions of the Association for Computational Linguistics, 12:1-18, 2024. doi: 1622///a__aa0006600 URL
ottps///eeaantooooooooo////222
Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 59532-59569. Curran Asso- ec http:/apprr_fllepppppeeeep
ciates, Inc., 2024. URLLhcte::/ppooceedsss..rre 2024 f 1e/ http:///wwooee.......a.aaaaaaaa/ Soreraaaeeeeeaaaaaeeeaooeeeeeeee Conference..cceeeeoooeeoeo
Shibo Hao, Yi Gu, Haotian Luo. Tianyang Liu, Xiyan Shao, Xinyuan Wang. Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, and Zhiting Hu. Lim reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models, 2024. URL https
7/arxiv. www.vv.org/as... wtt.:orr//bbaa/0/0m 2522244..222111
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python, 2020.
Qisheng Hu, Quanyu Long, and Wenya Wang. Decomposition dilemmas: Does claim decomposition boost or burden fact-checking performance? In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings the 2025 Conference of the Nations Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 6313- 6336, A Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-1894 doi: 10.18653/v1/200 1/2025 1aco22.naaclloae naacl-long URL httttts///claatthlllee
Americas Chapter of the Association for
320/
Huang, Philippe Laban, Alexander Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. Embrace divergence for richer insights: A multi-document summarization benchmark and a case study on summarizing diverse information from news articles. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 570-593, Mexico City, Mexico, June 2024. Association for Computational I Linguistics. doi: 10.18653/v 1/2024.aacc) -ong..2...332 URL https: //pps://cc anwaahaannnttooooo www.thoooo...oooooo wot.hhooo...rr//222222 naacl llng. 32/
11

---

Published as a conference paper at ICLR 2026
EunJeong Hwang, Yichao Zhou, James Bradley Wendt, Beliz Gunel, Nguyen Vo, Jing Xie, and Sandeep Tata. Enhancing incremental summarization with structured representations. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 3830-3842, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10. 10.188/31112222
http:///222222nnseeeeee..e Ainding--emnl..200 URL https://
aclanthology wtt. findings- emnlp. 220/
Dulhan Jayalath, James Bradley Wendt, Nicholas Monath, Sandeep Tata, and Beliz Gunel. Prism: Efficient long-range reasoning with short-context Ilms, 2025. URL https: httpss://arxiii org.v.org. abs/2412.1891.
Nan-Jiang Jiang, Chenhao Tan, and Marie-Cathering de Marneffe. Understanding and predicting
human label variation in natural language inference through explanation, 2023. URL https
//arxiv. 2tl.aab//223.... 12443
Greg Kamradt. Needle in haystack pressure testing Ilms. Lnw.:eetteeeeaaeeeaaaaaaatttttta InABaystack] 2023.
https://github. com/
gkamradt / /LLMTest SitimraadttLMtTet_eeeedLe. NeedWleest_eeeeleeeeee
Yekyung Kim, Yapei Chang. Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo,
Tanya Goyal, and Mohit lyyer. Fables: Evaluating faithfulness and content selection in book-length summarization, 2024. URL.bttee http://tsss//axxi...oooot wwwpaawwwww.g...////ooooaoaaaaaa 2404 01261
K. Krippendorff. Content Analysis: An Introduction to Its Methodology. SAGE Publica- tions, 2013, ISBN 9781412983150.
8 S_ygFXnGqj0C
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu
Soricut. Albert: A lite bert for self-supervised learning of language representations. In International
Conference on Learning Representations, 2020. URL
id=Hlee7AEtv3.
LangChain. LangGraph: A graphical interface for langchain. github. ic/ anggraphh/ 2025.
ittps::/langchainn
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- augmented generation for knowledge-intensee inteesive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
J. Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 1755)114 1991. doi: 10. 10.1 109/18.111..
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511 1-11112200 1-2222, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10. 1 11553/1112223.....o www.eeennammnn.. 153. URL https: wtt.aass//ccaatiololoooooo wwwp/a/oooaacccoc.coooooooooooaa wwt.oogy.o.g/2233. emm ma in. 153/
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining
approach, 2019. URL ht /ttp/aaaiii...a///iv/eee/ea.aela wwtp:/wwwww..///////...oo. 11692
Dasha Metropolitansky and Jonathan Larson. Towards effective extraction and evaluation of factual claims, 2025. URL httpthtsss///iii.trttttteeeee wttp:///r/agaa......1.1/
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076-12100, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/ /ttp//222.22mmmmaiiia 741. URL bttpa::/accattthoooog org/2023 lemn p-ma i n. 7417]
12

---

Published as a conference paper at ICLR 2026
Yixin Nie, Xiang Zhou, and Mohit Bansal. What can learn from collective human opinions on natural language inference data? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 9131 1-143, Online, November 2020. Association for Computational Linguistics. doi: 11118/6/11112222m///// 2020. ewm.p-mmpppeee main. 7347
htt.ttss:: //cc httpa///ooo.aa.ooooooooooo.ooo.. .rg/
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine. Gabriel Bernadett-Shappe Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang. Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Contijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga. Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Eukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Fukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger. Vishal Kuo, Michael Lampe, Ikai Lan. Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan. Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Fillipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich. Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak. Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https: /atpvv//axivvvv/a wwg/abs////..o..o.o. 2303 08774
Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. Making reasoning matter: Measuring and improving faithfulness of chain-off taain-of-thougee thought reasoning. In Yaser Al- Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 15512-15000 2-032 Miami, Florida, USA, November 2024. Association for Computational
13

---

Published as a conference paper at ICLR 2026
Linguistics. doi: 10.18633/112044.nndnggg.. orq/2024. www.diinnnngneemllpp.....---
URL https://aclanthol.o.
Chen Qian, Zihao Xie, YiFei Wang. Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large language model- based multi-agent collaboration, 2025. htR.:ttts:2... /tttpp:://rxivv awwaa.iivvvoooo wwwpaaawww.../b////////./// 07155
Fabian Ridder and Malte Schilling. The hallurag dataset: claee--oinn hallucinations in rag applications using Ilm's internal states, 2025. 17056
Jon Saad-Falcon Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: An automated evaluation framework for retrieval- -atgmeeel- generation systems. In Kevin Duh. Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 338 -344, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v 11t5://1/222..mamm m/2024.naacl-oogg2200 URL battps://aclaattoloogg.. org/ 2024. naacl long.on.0.
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. Raptor: Recursive abstractive processing for tree-organized retrieval, 2024. URL https://
arxiv.org/ abs 1104401.80599
Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! robust fact verification with contrastive evidence. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur; Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 624-643, Online, June 2021. Association for Computational Linguistics. doi: 20.18653/v1/0222 www.l-aaaaaai main. URL https
//aclanthology. org...rr//20 2021 maacl-mailn m...maannnn...... 52/
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. QuestEval: Summarization asks for fact-based evaluation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference
Empirical Methods in Natural Language Processing, pp. 6594-6604, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 00111863/1112221.mmlll maind.29.... URL 2021. www.p-main.n...e.ee.aamm.mmmmawm 529/
Charles Spearman. The proof and measurement of association between two things., 1961.
Zhongxiang Sun, Xiaoxue Zang, Kai Zheng, Yang Song, Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, and Han Li. Redeep: Detecting hallucination in retrieval-augmmened generation via mechanistic interpretability, 2025. URL htt.:tpp://aaxxi..rrr/////////
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning, Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433-5442, Singapore, December 2023. Association for Computational org/
Linguistics. doi: 10. 2023. www.penwwmaa.... maim-minn... 3307
URL https: httpp:ss//aaaaatthhoooc
Vincent A Traag, Ludo Waltman, and Nees Jan Van Eck. From louvain to leiden: guaranteeing well-connected communities. Scientific reports, 91):1-12,,00 2019.
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. Online, July 2020. Association for Computational Linguistics. doi: 10.18653/vI/ 2020.acl- main.450. htt.tttss::/
antho ooyy wgw.oogg/.00..ooo ac mamn.nn..... 4507
14

---

Published as a conference paper at ICLR 2026
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang. Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. httltttss///xxii...aa
abs 2203. 2ttpo//223..11111
Jerry Wei, Chengrun Yang. Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng. Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. Long-form factuality in large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9798331314385.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/N88-11000 URL htt.://ppp../...aaaaooooooooooop
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback, 2021. URL https://
www.v.orrg/aooo aataabss1100 /0..o////21112222 10862
Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. AlignScore: Evaluating factual consis- tency with a unified alignment function. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 11328-1 1132--1138,, Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: 10. 11111/////222......
URL https
7/aclanse aallaanhhoogy... org
6347
Huajian Zhang, Yumo Xu, and Laura Perez-Beltrachini. based faithfulness evaluation for diverse summarisation tasks. In Yvette Graham and Matthew Purver (eds.). Proceedings of the 1 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1701-1722, St. Julian's s, Malta, March 2024a. Association for Computational Linguistics. URL https: awl.aaaaaantooo....co........lla org/ 2024. eac44eacl-oo long. 102/
Fine- -oe-ggraied natural language inference
Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ö. An k. Chain of agents: Large language models collaborating long-context tasks. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 132208-132237, Curran Associates, Inc., wwwpsffllesss.eeeee 2024/ pdf
2024b. URL httpppps//pppcceeinni...eeeeeeee / wwwp::iiiii....iiiii..........
File/ F112267264442222000000eee8822222
Jun Zhao, Can Zu, Xu Hao, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. LONGAGENT: Achieving question answering for 128k-token-long documents through multi-agent collaboration. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp- 16310-16324 Mi- ami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024. 0024.emnlp-main99 912. URL httpt:tt:///// http:///caaaaahtoaa www.tthhhoooy...o/////go emn p-main 9127
15

---

Published as a conference paper at ICLR 2026
A EXAMPLES OF PROCESSES WITH MULTIPLE GENERATIVE STEPS
We used the following processes with multiple generative steps (MGS) in our experiments:
1. Hierarchical summarization (Wu et al. 2021; (Cangg et al. 2023). Source documents are split into chunks. A language model (LM) summarizes each chunk individually, then resulting summaries are repeatedly grouped and summarized until final output is produced.
2. GraphRAG (Edge et al., 2025). Source documents are split into chunks. For each chunk, an LM extracts entities and relationships. along with short descriptions. If an entity or relationship was extracted from multiple chunks, an LM summarizes the descriptions. knowledge graph is constructed from the final set of entities and relationships, then community detection algorithm, such Leiden clustering (Traag et al., 2019). groups entities into communities. For each community, an LM produces a "community report" that summarizes the entities and relationships. To answer user's question, an LM generates "map-level answers" based groups of community reports, then synthesizes them into final answer.
Additional examples of MGS processes include:
Incremental Summarization. Source documents are split into chunks, which are processed sequentially. An LM summarizes the first chunk; for each subsequent chunk, it updates the latest summary based on the current chunk (Chang et al. 2023 Hwang al. 2024 Jayalath
et al. 2025).
Indexing-Basel Methods. Many MGS processes include an indexing phase that is distinct from the querying phase. During indexing, the source material is converted into structured representation, such as graph or tree. Indexing is typically performed only once (unless the source material changes), while the querying phase occurs for each new query issued by the user. GraphRAG one such method that uses knowledge graph as its index structure. Additional examples include RAPTOR (tree-based; Sarthi et al., 2024) and HippoRAG (graph-based; Gutiérrez et al. 2024).
Multi-Agent Systems. LM-based "agents," each responsible for a specific document or sub-task, process the source material in parallel or sequentially. A final agent synthesizes their outputs into complete response. Examples include Chain of Agents (Zhang et al. 20246). LONGAGENT (Zhao al. 2024). and Mixture of Document Speakers (Balepur et al. 2025
B DATASET DETAILS
This section provides additional details about FABLES+ and DiverseSumm+, the datasets we con-
structed by augmenting the FABLES (Kim et al. 2024 and DiverseSumm (Huang et al., 2024) datasets introduced in $4.1
We performed claim extraction for both datasets using Claimify (Metropolitansky & Larson 2025 with - -08-06 and default hyperparameter settings. We manually removed redundant claims. The average number claims per book FABLES+ was 33, and the average per question in DiverseSumm+ was 28.
B. FABLES+
B.1.1 OVERVIEW
The original FABLES dataset contains summaries of 26 books published from 2023 to 2024. The summaries were decomposed into claims, and each claim was labeled faithful or unfaithful by crowdworkers who had read the corresponding book. For FABLES+, we used 22 books for which Kim et al.) provided access the texts. The retained books' token counts ranged from 49,156 to 242,683, with an average of 118099.. Token counts were calculated using OpenAl's tiktoken library with the el 00k base encoding.
16

---

Published as a conference paper at ICLR 2026
To re-generate the summaries, used the text chunks and hierarchical summarization implementa- tion from the original paper. Our only modification was storing all intermediate outputs, which we later used to construct the DAG imputs for VeriTrail. We also obtained the original summaries of the text chunks from Kim et al, and re-generated only the higher-level summaries. We generated half of the summaries using OpenAl's get-3. 5 5-turbo model and the other half using gpt-4-06 Both models were also used in the original study, and we applied the same summarization parameters: chunk_sian size 2, 048 and max mor__cnteet 8, 192.
When constructing the DAG for each book, we assigned stages to nodes as follows:
Stage root nodes (i.e., book chunks);
Stage nodes with at least one source node stage I (i.e., summaries book chunks);
Stage 3 = nodes with at least one source node in stage 2, etc.
All DAGs had four stages: the root nodes, two intermediate stages, and the terminal node. The number of nodes in stages I and 2 ranged from 26 to 141, with an average of 67. The number of nodes in stage 3 ranged from
6, with an average of 2.7. The total number of nodes across all stages ranged from 29 to 148, with average of 70.7.
To reuse as many labels as possible from the original dataset, we prompted an LM check whether each extracted claim was entailed by claim labeled as faithful in the original dataset or by any evidence or comments from the annotators, which we assumed to be faithful. We used opp-44--224-- 20----022-- 08-06 with the prompt in Anppeedii B.1.2. All matches were manually reviewed, and only clear cases were retained. For example, our extracted claim "Altha accused of the murder John Milburn" was deemed entailed by the following claim from the original dataset: "The villagers and prosecutor believe Altha Weyward used her powers to make John Milburn's 's cows trample him to death, despite lack of evidence." Since the claim from the original dataset was labeled as faithful in the original study, we classified our extracted claim as "Fully Supported."
Ultimately. 14% of claims in our dataset were labeled "Not Fully Supported," and the remaining 86% were labeled "Fully Supported." For summaries generated by GPT-3.5 Turbo, 12% claims were labeled "Not Fully Supported" (88% "Fully Supported"); for GPT-4, 14% of claims were labeled "Not Fully Supported" (86% "Fully Supported").
B.1.2 CLAIM MATCHING PROMPT
Claim Matching System Prompt
You are an expert in Natural Language Inference. You will be given a premise and hypothesis. Your task is to answer the follewing: Given the premise, is the ENTIRE hypothesis
NECESSARILY TRUE? In other words, would it be correct to say that if the premise is true, then the ENTIRE hypothesis MUST be true? If at least ene component of the hypothesis is NOT necessarily true based the premise, then the hypothesis is NOT necessarily true.
Note the following rules:
You will NOT make any assumptions or speculations.
You will NOT use any external information.
any weak implications the basis for the hypothesis being necessarily true. Only strong implications are al lowed (note that this is a weaker standard than requiring explicit statements).
You will NOT
If the hypothesis consists of multiple components, ALL components must be necessarily true given the premise in order for the hypothesis to be necessarily true. For example, if the hypothesis "John works at Mary's s favorite restaurant" and the premise is "John works at restaurant, then there is no evidence that the restaurant he works at is Mary's favorite restaurant.
17

---

Published as a conference paper at ICLR 2026
Claim Matching System Prompt (Continued)
You may also be given evidence and/or reasoning that helps explain the premise. It is EXTRAMELY important that you take it into account when answering the question.
First, print the full premise and hypothesis. Then, identify all components of the hypothesis.
Then, walk through your zeasoning step-by-step, Remember: ALL components of the hypothesis must be necessarlly true for the hypothesis to be necessarily true and weak implications, assumptions, and speculations are NOT allowed,
Lastly, print "Given the premise:", folloxed by one of the folloxing: "The entire hypothesis is NECESSARILY TRUE" or "The hypothesis is NOT NECESSARILY TRUE".
Claim Matching User Prompt FABLES+
Premise:
Hore is some information about the novel :book)": (premise) (evidence) (reasoning)
Hypothesis:
(hypothesis)
Claim Matching User Prompt DiverseSumm+
Premise:
Here is an answer to the question "[question)"; (premise) [evidence) (reasoning)
Hypothesis:
(hypothesis)
B.2
DIVERSESUMM+
B.2.1 QUESTION SELECTION AND ANSWER GENERATION
The original DiverseSumm dataset contains 245 news stories. For each story, Huang et al. formulated questions answered by multiple articles with varied perspectives. As a result, DiverseSumm is more realistic and challenging benchmark for hallucination detection than other datasets where questions have only a single answer from single source. Moreover, since each question was linked a single story, and the stories covered distinct topics, we expected the search space for faithfulness evaluation to be reasonably constrained an important consideration for annotation reliability. To reduce article overlap across stories, excluded stories with least one article that paired with multiple stories.
We then filtered out questions that (a) asked for opinions (c.g.. "How might the SEC's issuance of the Wells Notice impact Coinbase's 's business processes?") or (b) could not be understood without additional context (e.g.. "What assistance has been provided to the affected communities?" ?"). We sampled 20 of the remaining questions, each from different story. We aimed select diverse set of questions covering range of reasoning types. For example:
Focused, factual: "What is Genesis' strategy with the Electrified GV70?"
Broad, analytical: "What are the long-term implications of the interest rates hike for the economy?"
"How will the IRS's approach to NFTs compare to its approach other cryptocurrencies?"
Multi-hop:
"What are the concerns surrounding user data and TikIok, and what legal measures have been put in place to address them?" ?
Compound:
18

---

Published as a conference paper at ICLR 2026
To generate answers to selected questions using GraphRAG, we split the articles into 600-token chunks with 100-token overlap, resulting in 3,199 chunks. Token counts were calculated using OpenAl's tiktoken library with the o200k base encoding. For the indexing phase of GraphRAG, we used OpenAl's gpt-4o 1e:o--024-0555 querying phase, used GraphR g clabal search method with gpt -08-666-----
and text -exteee ng-ana- ada-002 models. For the
To construct DAG for each question, we assigned stages as follows:
Stage article chunks (i.e., the root nodes);
Stage 2 entities and relationships;
Stage 3 summarized entities and relationships;
Stage 4 = community reports;
Stage 5 map-level answers; and
Stage 6 final answer.
The same indexing phase outputs (stages 1-4) were used for all questions. Only nodes in stages 5 and 6, produced during the querying phase, varied by question. The node counts for stages 1-4 were 43,123 and relationships = 52,340); stage 3= == 11.974 (entities = 5,584 and relationships = 11.974); stage 4 = 3,650. For stage 5, the number of nodes ranged from 27 170, with an average of 79. The total number of nodes across questions ranged from 114,316 to 114,459, with an average of 1 114,368.
as follows: stage 3,199; stage 2 95,465 (entities
B.2.2 ANNOTATION PROCEDURE
To label the extracted claims, we recruited annotators through Upwork. As a screening test, candidates were asked to label 42 claims associated with one of the questions. We reviewed their labels and compared them to our own. Six candidates completed the test, and four whose results met our quality standards selected continue. All selected annotators were fluent English speakers, based in the United States, and had a 100% success rate for prior jobs. Three had completed a bachelor's degree. They were compensated at rate of $15-20 per hour.
To minimize annotator fatigue, we divided the remaining samples into two batches, which were annotated sequentially over five days. Each annotator labeled all claims in both batches. The first batch included 6 questions, and the second included 13. We provided detailed feedback to the annotators after both the screening test and the first batch. To avoid overloading the annotators, they were shown only the 9-10 articles associated with each claim. Independently, one of the authors annotated all claims with access to the full set of articles. As noted in 4.T only 13% of claims required the full set of articles, confirming our hypothesis that the constrained set would be sufficient to verify most claims.
Appendix B.2.3 contains the annotation instructions provided to the annotators. For the annotation interface, we created a separate Excel file for each question and annotator. The files contained one row per claim and columns for assigning label, indicating uncertainty, quoting supporting refuting evidence, and providing comments.
The label options used in the annotation study were more granular than those used by VeriTrail. Our goal was to encourage annotators to be as precise as possible about the relationship between each claim and the articles. Specifically:
The labels "At Least One Part is Refuted" and "Insufficient Evidence (None of the Above)" were both mapped to VeriTrail's ss "Not Fully Supported" label, but the former required annotators to provide evidence that refuted the claim, while the latter indicated lack of both supporting and refuting evidence.
The "All Parts are Supported" label was mapped to VeriTrail's "Fully Supported" label.
The "Conflicting Evidence with No Clear Resolution" label was mapped to "Inconclusive."
* All "I Don't Understand the Claim" labels were excluded from our analysis
*Conflicting Evidence with No Clear Resolution" and "I Don't Understand the Claim" labels were rare: they each represented only 1.6% of all labels.
19

---

Published as a conference paper at ICLR 2026
For 81% of claims, the majority label was used as the final label. For the remaining claims, the author's ' label was used, for one of three reasons: (a) the claim required evidence beyond the 9-10 articles provided the Upwork annotators; (b) there the annotators had incorrectly applied the instructions or missed relevant evidence. The final label distribution for the 560 claims was 74% "Fully Supported" and 26% "Not Fully Supported."
majority label; or (c) we determined that
We assessed alignment between our labels and annotations from the original DiverseSumm dataset. In addition creating questions for each news story. |Huang et al. used the associated articles generate answers, which were validated by human annotators. They also generated summaries of the articles, and annotators labeled the faithfulness of each summary sentence. We repeated the entailment- based matching procedure described in [Appendix B.1 to check whether any of our extracted claims were entailed by either a validated answer or a summary sentence that was deemed faithful. We identified matches for 79 claims (14%), of which 78 were labeled "Fully Supported" in our dataset, indicating near-pe near-perfect alignment.
Across the five annotators in study, Krippendorff's alpha (Krippendorff 2013 Castro 2017) was 0.49 For 56% claims where no annotators reported uncertainty. alpha was 0.68. For claims where at least one annotator reported uncertainty, alpha dropped to 0.38. These results are comparable to those reported by AmbiFC (Glockner et al. 2024), which we view as the most methodologically similar study: in the condition where at least five annotators labeled claims' entailment with respect to passage, alpha was 0.55 for the high certainty subset and 0..1 for the low certainty subset (where annotators indicated uncertainty, in study). Notably, their passages while our annotators worked with much longer texts. The fact that our agreement scores were higher despite the increased task difficulty encouraging.
capped at 20 sentences,
We offer two key takeaways based on the inter-annotator agreement results. First, claims where annotators expressed uncertainty had lower levels of agreement, suggesting that asking annotators to indicate uncertainty may help flag challenging or potentially ambiguous cases. Second. after reviewing the claims where annotators disagreed on the correct label, we found that some cases could be attributed to annotation errors, while others reflected reasonable differences in how the claim and/or the evidence were interpreted. This finding suggests that in future studies, might be helpful to complement single-label annotations with probability distributions reflecting annotator agreement (e.g., as explored Nie et al. 2020 and Jiang et al. 2023).
B.2.3 ANNOTATION INSTRUCTIONS
Annotation Instructions
Thank you so much for participating in this study! We greatly appreciate your time and effort. Please read the following instructions carefully and let know if you have any questions.
# Overview
You will be provided with one more foiders. Each folder will have code like "Q0", "Q1", "Q2", etc. Each folder corresponds to distinct question or topic area.
Kach felder will contain two files:
1. claims.xls. factual "claim, such "As of 2021, India overtook China the country with the largest population." docc docx A Word document containing the content of 2. articles. approximately 10 news articles.
spreadsheet where each corresponds to single
For each folder, your task to fact-check every claim in the claims .xlsx file using only the information found in the corresponding articles.dod docx file from the sare folder.
Foor the 13% of claims that required the full article set, the author's labels were omitted from the agreement calculation, since they cannot be fairly compared to the labels from annotators who only had access to the constrained article set.
20

---

Published as a conference paper at ICLR 2026
Annotation Instructions (Continued)
You will record your results directly the claims xisx file within each folder. When you' ze tinished, return the same folders you received, with the claims. .xixx file completed in each one. For example, if you were given 10 folders, you should submit those same folders, each containing completed claims xlax file.
# Columns to complete
In the claims.xlss xlsx file, you' 11 need to complete four columns for each claim: Column (Label), Column D (Uncertain), Column (Evidence), and Column (Comments).
### Column C: Label
Each cell in Column should have drop-down menu with the fol following five options:
1. A11 parts are supported
2. At least one part is refuted
3. Conflicting evidence with no clear resolution
Insufficient evidence (none the above)
I don't understand the clain.
Here are some key definitions you'll 11 need in order to correctly apply the labels:
1. A claim is "supported" by the articles if the articles either explicitly state or strongly imply that the claim true. simple test here is whether you'd d feel confortable saying: "According to the news articles, <insert claim>."
A claim is "refuted" by the articles if the claim directly contradi or strongly impl to be false by the articles.
simple test here is whether you'd d feel confortable saying: "Based on the news articles, it's not true that <insert claim>" "Based the articles, it's very unlikely to be true that <insert claim
IMPORTANT: Lack of evidence does NOT mean the claim is "refuted," If the articles simply don't mention somothing (c.g., they say nothing about India's China's population), then that's not refutation it's "Insufficient evidence" (more on this later).
3. "Conflicting" evidence means that the articles contain both supporting and refuting information. It can also that the claim has multiple possible interpretations, and depending on which interpretation you choose, the claim can be supported or refuted. For example, Article 1 says China has the largest population, while Article says India has the largest population. Another example: the claim is "India is larger than China, and Article 1 says India has bigger population than China, while Articie says China is bigger than India in terms of land mass.
If you find conflicting evidence, your next if: (A) there is clear resolution to the conflict from 2021, the
is to determine
from 2020 and Article difference in publication dates might explain the discrepancy, suggesting a possible timeline shi ft. rather than a contradiction OR (B) there is no clear resolution to the conflict (i.e., both sides appear equally valid and there's obvious way determine which is correct).
(c.g., if Article
Do NCT resolve conflicts based on which article seems more credible trustworthy. That is not valid basis for determining clear resolution.
21

---

Published as a conference paper at ICLR 2026
Annotation Instructions (Continued)
4. A claim. may have one or more "parts. You can identify these parts by asking yourself: "What must be true in order for the entire claim to be true?" In the example claim above "As of 2021, India overteck China the country with the largest population" are three things that must true order for the entire claim be true: (1) India became the country with the largest population, (2) this happened as of 2021, and (3) prior to 2021, China had the largest pepulation.
there
Now, here are the rules for labels:
1. If all parts of the claim are supported by the articles, then you should select label 1: "All parts are supported."
2. If at least one part of the claim is refuted by the articles, then you shculd select label 2: "At least one part refuted".
Fer example, even if the articles confi that India now has the largest population (part 1) and that China used to have the largest population (part 3), if they indicate that India overtook China in 2022, not 2021 claimed, then part 2 is refuted, so you would select the "At least one part is refuted" label.
3. If there is conflicting evidence that does not have a clear resoluticn for at least one part of the claim, then you should select label 3: "Conflicting evidence with clear resolution."
you feel the conflicting evidence does have clear resclution, then you should just treat that evidence as supportive or refuting, whichever resclution is preferred.
4. If none of the above options apply, then you should select label "Insufficient evidence (none of the above)". This covers cases where: (1) no parts of the claim are refuted by the articles (or eise we would' ve selected label 2) AND (2). no parts of the claim have conflicting evidence that can't t be resolved (or else we
would' ve selected label 3) AND (3). at least one part of the claim is missing support.
For example, if the articles indicate that India surpassed China as the most populous country, but there are no dates provided, then the claim is not refuted and there's no conflicting evidence, but it's also not fully supported, Insufficient evidence (none the above) label.
you would select the
5. If you don't understand the claim for whatever reason, then you should select label 5: "I don't understand the claim."
You should follow the order of priority described above when assigning labels. For example, one part of the elaim is refuted by the articles (label 2), but another part has conflicting evidence with ne resolution (label 3), then you should assign label 2.
IMPORTANT: NOT use any personal knowledge or outside sources in the fhec-checking process. For example, if the claim is "The capital of the United States is Washington, D.C., but the articles don't refutes this claim, it doesn't
contain any evidence that supports matter that you know the claim. is true. You must still select the Insufficient evidence (none the above) " label.
### Column D: Uncertain
Fact-checking is often subjective - there won't always be clear-cut "correct" answer. If you feel uncertain about the labal you selected for any reason, please enter TRUE Colunn
You will NOT be penalized for indicating uncertainty - in fact, it's valuable for our analysis, as it helps identify potentially tricky" cases that are ambiguous or open to interpretation. If
22

---

Published as a conference paper at ICLR 2026
Annotation Instructions (Continued)
you're re confident in your Label, you can simply Leave Column D blank or enter FALSE.
Potential reasons you might feel uncertain include:
You found some relevant evidence, but you're unsure whether it's strong enough to support or refute the claim. You' re not sure whether you interpreted the claim correctly.
**# Column Evidence
you selected one of the following labels, then you must provide at least one direct quote from the articles to justify your choice:
1. "All parts are supported" Provide evidence that supports all parts of the claim
"At least one part is refuted" Provi evidence that di directly
contradicts the claim or strongly implies that it's false
3. (Cenflicting evidence with no clear resolution" Provide evi dence that both supports and refutes the clain
Your evidence does NOT need to exhaustive. For example, if there are 10 quotes that say India's population is larger than shina's, it' sufficient to provide one of them. However, if the claim has supported,"
multiple parts and you selected the label "All parts make sure that the evidence you include (whether it's single quote or multiple) collectively supports all parts.
Do not include commentary or explanations in this column - only paste the relevant quote Save any comments for Column F.
### Column Comments
This column is optional but highly recomnended especially if you felt uncertain or selected the "Conflicting evidence with no clear resclution" label. Use Column F to:
Briefly explain your reasoning behind the label
Clarify how you interpreted the claim
Note anything unusual or borderline about the evidence
These comments help review your work and understand your decislon- making process.
# Process & Tips
Here's the recommended process follow for each folder: Open articles doex and skim. through the content to get familiar with the topic area.
2. Read the first claim carefully. Make sure you fully understand the claim. If it's unclear or confusing for any reason, select the label "I don't understand the claim." Also consider whether the claim has multiple possible interpretations that you need to consider.
Break down the clain into parts (or determine that there's s only one part). Ask yourself: "What must be true in order for the entire claim te be true?"
4. Search for evidence in the articles. Look for sentences that support er refute any part the claim.
Tip: Use the "Find" function (Ctrl+F on Windows, Command-F on Mac) to search for key terms.
If you've ve searched theroughly and found no relevant evi evidence, select the "Insufficien
evidence (none of the above) label.
If yeu've ve found evidence that seems to be conflicting (i.e., there is both evidence that supports at least part of the clain as well as evidence that refutes it), determine whether
think there's clear resclution to the conflict, not.
Fill cut Columns C, D, E, and F, using guidance above.
6. Move on to the next claim!
23

---

Published as a conference paper at ICLR 2026
Annotation Instructions (Continued)
Tip: Some claims may overlap, For exanple, consider Claim Jil1 is John's s manager" and Claim 2 "Jonn asked his manager Jill for promotion. If you find that Clain 1 is refuted, you already know that part of Claim 2 is also false. You can reuse the same evidence when labeling claim
# Additional Clatificatoons
When shculd you the label "Conflicting evidence with clear resoluticn"?
This label should only be used when you have both evidence that supports the claim well evidence that refutes the claim, and you're sure which is correct. It can so used you feel there are multiple possible interpratations of the claim, one of which is supported by the evi deiden,, another one of which is not, and it's net clear which interppeetaiio/resolttttt
is correct.
there's Insufficient evidence that feels relevant, or only very weakly relevant, you should pick "Insufficient evidence."
If there is some relevant evidence and you're just not sure whether it's "strong" enough to count supporting then you should decide whether you: (A) lean towards it being strong enough to count as supporting (or refuting), so pick "all parts are supported" (or "at least ene part is refuted"), or (B) lean towards it NOT being strong enough count pick "Insufticient evidence". In both cases, it would be good set "Uncertain" to True (don't be afraid to do this!). Hut, to reiterate, the "conflicting label is for when you feel it's roughly equally likely that the claim is supported real option for the label (e.g., supported) and you' re just sure if the evidence is strong enough to warrant that label.
refuting) the claim,
supporting
refuting),
refuted, NOT when there's only
24

---

Published as a conference paper at ICLR 2026
C VERITRAIL DETAILS
C.I ALGORITHM
Algorithm details VeriTrail's verification procedure for single claim The procedure assumes access to DAG G = (V, E) representing generative process (see $ including the terminal node u" root nodes Vo, and the source function src.
Algorithm 1 VeriTrail
Require: claim c, max NofFullySupported iterations
evidence tatincl < []
2: consec_not_supp x-
3: checked +
4: roots settt_tthee t- 9
5: all verdicts [[]
6: nodes_lo tceckcck & src(u*)
7: while true do
8:
(evidence, nodes_ nodes_wille with_ev) < get_evidencefe, nodes chcckc)))
9: if evidence = then verdict NotFullySupported
10:
11: else
12:
add evidence to evidence_trat
13:
rools_wilhheev +- rools_willl wilh_ev U (nodes_wilh-_e n Vo)
14:
verdict +- get_ vee__vedicct(,, evidence, nodes writh_ev)
15:
end if
16:
add verdict to all uerdicts
17:
checked + checked U nodes checkcc
18:
if verdict = NodFullySupported then consec_nol supp consec_not poonsec_nol_supe +
20:
nodes nodes_to_eed ttt_tccccee + Unenodes_to_cheed tocccckk stc(n)
else
22:
consec_rod ses__not_ssupp_ supp t 0
nodes_lo, fites_loocheck +- U UnEnodes_with src(n)
24:
end if
25:
nodes_to ttcess_tt__oe theck + (nodes_toe tt___ttcccccc \ checked) roots_ roots_with with_ev
26: if nodes to check roots_with_____ witheen ev then
27:
break 28: end if
29:
if nodes Lls__oo__hhee check = then
30:
verdict NotFullySupported
31:
break end if if consec_not_et = then
33:
break
35: end if
36: end while
37:
return (verdict,, evidence_ -trall, all _evediccs))
25

---

Published as a conference paper at ICLR 2026
C.2 PROMPTS
C.2.1 EVIDENCE SELECTION PROMPT
Evidence Selection System Prompt
You are an extremely smart, thorough, and neticulous assistant. You will be given a collection of excerpts from one or more sources. Each excerpt preceded by label like I[1lI, and each sentence excerpts has an ID. You will also be given question the form "Is there any information in the excerpts that indicates sproposition??" Your task is to answer the questicn.
Note the following rules:
Sometimes the proposition can be further decomposed into sub- propositions. For example, if the propositi on is "There have been advancements clean energy and desalination technologies," the sub- prepositiens are: "There have been advancements in clean energy" and "There have been advancements in desalination technologies." If information in the excerpts strongly implies the truth or falsehood of at least
sub-proposition, it should be included in your answer
You will only include information that STRONGLY implies sub- proposition's s truth or falsehood. You will NOT include weak implications If you are not sure whether sub-proposition is STRONG WEAK implication, you should defer towards including it your answer.
You will put yourselt in the shoes of a careful reader who interprets the text holistically, considering both explicit statements and implied meaning. For example, if the claim is "John emphasizes the importance of mentorship programs", and John never explicitly says that mentorship programs are important but it's clear that he values them because he speaks of his attempts to establish mentorship programs and he comes across as passionate about them, then careful reader would find that the proposition strongly implied.
If the proposition is something like "John found x", "John reported x", "John emphasizes x", etc. (where John can be replaced with any entity entities), it should be interpreted what John says does. highlights that transparent cormunication is a critical part of Project Alpha", and the excerpts cate that transparent. communication is critical part of Project Alpha, but they are missing the critical context that this is sonething John highlights, then they would NOT strongly imply the truth or falsehood proposition. Let's call this the Statements and Actions Rule.
statement about example, if the proposition is "John
the
You will NOT use any external knowledge beyond what is stated in the previded excerpts.
It is EXTREMELY important that you cite the correct IDs. You will heavily penalized if you attribute information to the wrong ID.
Your cutput must adhere to the following format exactly.
& Question: <insert full question> Proposition: <insert proposition>
H Step 1: Decompose proposition into sub-proposit cntrroosiiiona. cions that cannot be further decomposed (two rounds) <Decompose the proposition propositiens SP [SP1, 522, l..l. the proposition cannot decomposed into multiple independent sub-propositions, return single-label list. Make sure to follow the Statements and Actions
into list of independent sub-
Rule. Ensure that the S do not contain any unverifiable components
26

---

Published as a conference paper at ICLR 2026
Evidence Selection System Prompt (Continued)
e.g.r "extensive", "signi ficant", "substantial" etc.) from P. You will de this in two rounds, to ensure that the sub-propositions cannct be decomposed any further. For example:
"As the CEO Company X, John's frequent emphasis the importance of solar and wind energy has contribated to their mainstream. acceptance."
Round 1: SP without unverifiable components "John is the CEO of Company x", "John has emphasized the importance of solar and energy", "John's emphasis on the importance of solar and wind energy has contributed to their mainstream acceptance"
Round 2: SP without unverifiable components
"John is the CEO of Company x", "John has emphasized the importance of solar energy", "Jhhn has emphasized the importance of wind energy",
"Jchn's s emphasis on the importance of solar energy has contributed to its mainstream acceptance",
"John's s emphasis on the importance of wind energy has contributed to its mainstream acceptance"
}>
step Provide an overview sentences
<Provide an accurate overview of the sentences in the excerpts with respect to the question, without adding any interpretations or making any assurptions. The overview should be fully entailed by the excerpts. For example, if the question asks whether there have been advancements in clean energy and a sentence says there is potential for advancements in clean energy, the overview will NOT say mentiens advancements in clean energy" sentence; it will say "mentions potential. for advancements in clean energy". It can be very helpful to organize the sentences by excerpt Provide a point for each sentence WITHOUT quoting it. If there aren 't any relevant sentences, state "NO RELEVANT SENTENCES" and terminate your output here. It is EXTREMELY important that you do not overlcck any relevant sentences.>
this misrepresents the
H Step 3: Test each sentence or each range of sentences
<For each sentence or range of sentences you ideatified in step 2, print the sentence ID or range of sentence IDs then complete ALL of the bulleten statements below. faith completion for a statement: (i.e., you should NOT claim that the sentence states something when it does not, or that it fails to state scmething when it does), statement. Remember that you NOT alloxed to outside the provided excerpts. You MUST cover ALL the sentences or ranges of sentences you identified in step 2.
it's not possible to make good
should put "N/A" for that any information
SP <insert the SP from Step 1 that is most relevant to the sentence range of sentences>
the following quote argue that the sentence (s) strengly implies (NOT necessarily explicitly states) the truth or falsehocd of SP: "..."
One might
One might use the follewing (uote from the remaining sentence ID (s) in the excerpts
additional context: "..." "N/A"
A careful reader trained to look for STRONG IMPLICATIONS, which weaker standard than explicit statements, and to consider the sentence (s) holistically would reason as follows: <insert step-by- step reasoning, then clearly state the conclusion about whether or not it could be interpreted you're re net sure whether it's a strong implication, you should defer towards including it>.>
strong implication; remember that if
27

---

Published as a conference paper at ICLR 2026
Evidence Selection System Prompt (Continued)
# Step 4: Final submission
<Insert ETTHER (1) "The excerpts do not contain any information on that strongly implies any sub-proposition" OR (2) "The following sentences provide a strong implication: [<insert ALL sentence IDs where strong implication is the conclusion from Step 3; do NOT include any excerpt labels, e.g., [[1]]:5 is incorrect vs. is correct; ranges allowed for consecutive sentence IDs, e.g., 5-10>] with the following sentence (s) providing essential context: [<insert ALL sentence IDs needed as context for the sentence IDs that provide strong implication; if no context is needed because the sentence IDs independently provide streng implication, leave this empty>] Here is complete summary covering ALL information in the sentence that relevant to least one sub-proposition and ALL context necessary to understand them and their connection to the (ub-proposition wi thout mentioning what is implied or indi cated: <insert an accurate description of the information contained in the sentence (s) and their connection the sub-proposition(s) entities when they are provided; do NOT just quote the sentences; do not speculate about what is implied or indicated.> Here are some comments on what is missing unclear: <insert here, "N/A">
always
full names for
Evidence Selection User Prompt
Excerpts:
(excerpts)
Questicn: [questien]
Example sub-propositions (SP) that may need to deconposed further: (sub_claims)
C.2.2 VERDICT GENERATION PROMPT
Verdict Generation System Prompt
You are an extremely smart, thorough, and met cul ous assistant. You ieticulon
will be given a collection of excerpts from one or more sources. Each excerpt is preceded by label like [[]]], and each sentence in the excerpts has an ID. You will also be given claim. Your task is to answer the following question: Do the excerpts justify the entire claim?
order for the excerpts must STRONGLY imply that the entire clain is true. This means that a careful reader of the excerpts would naturally infer the entire claim witheut needing to make any assumptions or access any external information. Note that streng implication is a weaker standard than explicit statement. Also note that WEAK implication is NOT sufficient For example, if the claim is "John high' ights the importance of coll lollabrraiion in driving innovation" and the only relevant evidence in the excerpts is that John worked on several team projects, the excerpts would NOT justify the entire clain.
justify the entire claim, the excerpts
There are 4 possible cases where the excerpts do NOT justify the entire claim: 1. The excerpts contradict least one part of the claim
28

---

Published as a conference paper at ICLR 2026
Verdict Generation System Prompt (Continued)
2. The excerpts strongly imply that at least one part of the claim is false
3. At least one part of the claim is only weakly implied by the excerpts
At least one part of the claim not addressed by the excerpts
Note the following rules:
The claim. is extracted from answer to question about collection of documents. Therefore, if the claim is something like "x is mentiened" or "X is discussed, it should interpreted as a statement about what is mentioned or discussed in the documents.
If the claim is something like "John found X", "John reported X", John emphasizes x", etc. (where John can be replaced with any entity or entities), it should be interpreted as a statement about what John says er does. For example, if the claim is "John highlights that transparent communication is a critical part of Project Alpha", and the excerpts indicate that transparent comnunication is a critical part of Project Alpha, but they are missing the critical context that this is something John highlights, then they would NOT justify the entire claim. Let's call this the Statements and Actions Rule. You will NOT previded excerpts.
any external knowledge beyond what is stated in
You will put yourself in the shoes of a careful reader who interprets the text holistically, considering both explicit statements and implied meaning. For example, if the claim is "John emphasizes the importance of mentorship programs", and John explicitly says but it's s clear that he values them because he speaks of his attempts to establish mentorship prograns and he cones across as passionate about them, then careful reader would find that the excerpts justify the entire claim.
the text that mentorship prograns are important
You will operate under the assumption that the excerpts contain all information required to make a determination. For example, if the claim is "John led three teams" and the excerpts are from interview where John only mentions argue that the excerpts do not provide a comprehensive 1ist all teams that John led so a determination cannot be made. Instead, you will censider the excerpts to be the only source of truth and since they only support the conclusion that John led one team, the excerpts do NOT justify the entire claim. Similarly, if one excerpts provides a list of teams and another source indicates that some teams were led by John, it iS valid to cross-reference the lists to determine the number of teams John led.
tean that he led, you will NOT
in the
Your cutput must adhere to the following fornat exactly. Do NOT remove the instructions. 1: Claim <insert claim>
Does the Claim have multiple possible interpretations? yes, specify them, then clearly state which one you believe most people would agree with you will use this interpretation for the rest of your cutput. If there are distinct aspects of the Claim that must be true for the Claim to be true, enumerate them (e.g., "John worked at (1) Cempany and (2) Company 2"). Also identity any unverifiable components of the Claim (e.g., "extensive",
finnnt,, "substantial", etc.) Print "Clari fiedClaim <insert clarified version of the Clains".
Quete the relevant sentences in the text with respect to the ClarifiedClaim without any interpretations or judgments, making sure
29

---

Published as a conference paper at ICLR 2026
Verdict Generation System Prompt (Continued)
to include the sentence IDs. Do NOT cover sentences about the lack of information, e.g., "there is no explicit nention of x" If there aren't any relevant sentences, state "NO RELAVANN SENTENCKS" and terminate your output here. If there sentences, pick the 10 most important ones.
Likely
than 10 relevant
<insert stream of consciousness thought process; use bullet points oz numbered lists if needed>
Identify ALL pieces evidence from step 3 that are CONFLICTING i.e., one piece of evidence indicates X is true while another indicates X is false), outline the possible resolutions, and determine whether or not the excerpts STRONGLY imply that one resolution is preferred over the other If yes, clearly state which one is preferred, and use this information in your final deliberation. If not, you will DISCARD this issue in your final de1 iiberattion (1.e., you will treat it as if the resolution is unknown it cannot be used to make determination). Make sure to include the sentence IDs in your output.
5: Identify ALL pieces of evidence from step 3 that are DEBATABLE (i. people could reasonably disagree on what the evidence means, what it implies with respect to the ClarifiedClaim, and/or the strength of the implicatien), outline the possible conflicting positicns, and determine whether or not one position is more compelling than the other If yes, clearly state which one is more compelling, and use this information in your final deliberation. If not, you will DISCARD this issue in your final deliberation (i.e., you will treat as it the resolution used to make a determination) Make sure to include the sentence IDs in your cutput.
unknown, so cannot be
List sentence IDs from step that were NOT included in steps and 5, then quote them. These pieces of evidence are CLEAR in their meaning and implication for the ClarifiedClaim.
Given your analysis the evidence in steps 4-6, and considering that there may be parts the ClarifiedClai that are NOT addressed by the evidence, does the NON-DISCARDED evidence from the excerpts justify (i.e., STRONGLY imply) the ENTIRE claim? Remember that strong implication is weaker standard than explicit statement, but weak impl implication and speculations are NCT sufficient. First, walk through your reasoning step-by-step; do NCT jump straight to the conclusion Then print, "I submit the following answer: <insert 'excerpts justify the entire ClarifiedClaim' or 'Excerpts do not justify the entire ClarifiedClaim 'Cannot determine if Excerpts justify the entire Clarifiedllsim'> Only use 'Cannot determine if Excerpts justify the entire ClarifiedClaim' if all evidence was DISCARDED.
Verdict Generation User Prompt
Excerpts:
(excerpts)
Claim: (claim)
The output "Excerpts justify the entire ClarifiedClaim" corresponds to VeriTrail's "Fully Supported" verdict. The output "Excerpts do not justify the entire ClarifiedClaim" corresponds to the "Not Fully Supported" verdict. The output "Cannot Determine if Excerpts justify the entire ClarifiedClaim" corresponds to the "Inconclusive" verdict.
30

---

Published as a conference paper at ICLR 2026
D COMPUTATIONAL COST
Table compares the average cost per claim for VeriTrail, baseline methods except the NLI models, and human annotation 10 Any retries due invalid outputs were also factored into the estimates. We note that RAG consists two parts: (I) a one-time embedding of all source text chunks, and (2) per-claim processing (embedding the claim, retrieving & chunks, and generating a verdict). We compute the amortized per-claim cost, which divides the one- embedding cost across all claims and adds the result the average per-claim processing cost. For completeness, the one-time embedding cost for RAG was $0. 17 for DiverseSumm+ and $0.02 for FABLES+.
Table 2: Average cost per claim ($) for the FABLES+ (F) and DiverseSumm+ (D) datasets. VeriTrail's hyperparameter specifies the number of consecutive "Not Fully Supported" verdicts that will trigger termination of the hallucination detection process. RAG's l hyperparameter specifies the number of top-ranked chunks that were retrieved.
$/Claim
Method
VeriTrail (DeepSeex-V3.g=1)
0.06
0.12
VeriTrail
0.09
0.14
VeriTrail (misst:al-1arrge-4111.--
0.46
0.83
VeriTrail (gpt-40-2024-0006.6-
0.90
1.22
VeriTrail (gpt-4o-2024-0006..-1
1.15
1.61
VeriTrail
1.20
2.12
RAG (gpt 4o99t-44-2024-0800 ==))
0.06
0.03
RAG (ptt-4c-2024-080, k 15)
0.12
0.04
RAG (gpt (gpt-4o-200) 2000--4-2224-08065 k 25)
0.21
0.06
GPT-4.1 Mini
0.06
0.54
Gemini 1.5 Pro
0.38
3.37
Human Annotation
1.65
2.41
The cost estimates in Table combined with the performance results in Table 8 Appendix E.7 demonstrate that Veri Trail can achieve strong performance at low cost. For example, using Gemini- 2.5-Flash with q 1, VeriTrail costs only $0.09 $0. 14 per claim, while still outperforming the baselines. These results are particularly noteworthy given VeriTrail's significantly larger verification burden. Unlike the baseline methods, which only provide a faithfulness verdict by comparing claims directly to the source material, VeriTrail also provides traceability by constructing an evidence trail through intermediate outputs. For example, in DiverseSummt, the baselines only evaluate -3K root nodes, whereas VeriTrail must evaluate an additional 1 10K intermediate nodes.
In terms of time complexity, the worst-case for VeriTrail is ((V)) per claim, where |V| the number of nodes in the input DAG. However, this upper bound assumes that the claim must be verified against all nodes. In practice, considerably fewer nodes are evaluated. For example, when q = 3. the average percentage of nodes verified was 47% for FABLES+ and 1% for DiverseSumm+; when & 1, the corresponding averages were 38% and 0.3%, respectively.
VeriTraii'ss efficieny stems from the following aspects of its design:
1. Early Termination. If claim reaches q consecutive "Not Fully Supported" verdicts, the verification process terminates, even if the root nodes have not yet been reached. Therefore, lower q values allow the process to terminate earlier, reducing computational cost. Table confirms that lower values are associated with lower average cost per claim.
2. Selective Verification. As described in S 3.1.4 after "Fully Supported" or "Inconclusive" verdict, only the source nodes of nodes that yielded evidence are verified not all possible
FoFor DiverseSummt, our cost estimate for human annotation is the total amount spent on the annotation study ($1,350) divided by the number of claims in the dataset (560). For FABLES+, we used the cost reported in the original FABLES paper ($5,200) divided by the number of claims in their dataset (3, 158). We do not include the II.I models their costs vary depending on the type of GPU used.
31

---

Published as a conference paper at ICLR 2026
source nodes. This approach avoids wasting computation nodes that are unlikely to contain relevant information.
Reverse Traversal. VeriTrail verifies claims the reverse order of the generative process: it starts from the source nodes of the terminal node v" and proceeds towards the root nodes Vo- Because selective verification progressively narrows the search space and low g values enable early termination, VeriTrail tends to verify larger proportion of nodes in later stages (closer to v²) than in earlier stages (closer to Vo). Since carlier-stage nodes are typically larger (e.g., book chapter is larger than chapter summary), verifying fewer of them reduces cost.
32

---

Published as a conference paper at ICLR 2026
E ADDITIONAL EXPERIMENTS
All experiments in this section were conducted on random subset of the data: 6 books from FABLES+ and questions from DiverseSummt. This subset includes 31% % of claims from FABLES+ and 34% from DiverseSumm+, totaling 415 claims. Unless otherwise noted, VeriTrail and RAG results in this section were produced 4pt-40-02------ with q for VeriTrail.
E.1 UNDERSTANDING VERITRAIL'S PERFORMANCE
In this section, analyze which factors contributed to VeriTrail's performance gains. As noted in $4.3 all LM-based methods used the same verdict prompt, so VeriTrail's Verdict Generation step cannot explain why outperformed the baseline methods. Instead, VeriTrail differs from the baselines in two key ways: (1) it traces claims through intermediate outputs rather than checking them directly against the source material, and (2) uses LM-based Evidence Selection prior to Verdict Generation.
To isolate the contribution of each component, we created variant of VeriTrail that retained its tracing mechanism and Verdict Generation prompt but replaced its LM-based Evidence Selection step with the embedding based retrieval approach used in RAG, the best-performing baseline. At each iteration, this variant retrieved the top-k nodes most similar to the claim and used them as input for Verdict Generation. We tested & E (5, 15, 25). If the verdict was "Not Fully Supported," verification terminated (as the original VeriTrail with q = 1); otherwise, the source nodes the retrieved nodes were evaluated in the next iteration.
We refer to this variant as VT-RAG (VeriTrail-Ree hybrid), and compare to the original VeriTrail with (VT) and RAG. These methods can be summarized as follows:
VT tracing LM-based Evidence Selection
VT-RAG tracing embedding bbasdd Evidence Selection
RAG no tracing + embedding- based Evidence Selection
If tracing through intermediate outputs the primary driver of VeriTraill' performance gains, expect VT = VT-RAG > RAG. LM-based Evidence Selection is the primary driver, we expect VT> VT-RAG and VT RAG. If both factors contribute, expect VT VT-RAG RAG.
Across both datasets, the original VeriTrail (VT) outperformed the other methods. For FABLES+, we observe VT RAG VT-RAG, suggesting that VeriTrail's Evidence Selection step is the main reason it outperforms the other methods. For DiverseSumm+, we observe VT VT-RAG > RAG, indicating that both the Evidence Selection step and tracing through intermediate outputs contribute to Veri Trail's performance gains.
Results are shown Table 3
Table 3: Hard prediction results (%) on the FABLES+ (F) and DiverseSumm+ (D) datasets, com- paring standard VeriTrail with q=1 (VT), RAG, and a hybrid method (VT-RAG). For RAG and VT-RAG, & denotes the number of top-ranked chunks that were retrieved. We report macro Fi, balanced accuracy (Bal. Acc.), and class-specific precision and recall for fully supported (FS) and not fully supported (NFS) claims. Bolded values indicate the highest score per column.
Macro FN
Bal. Acc.
Precision Ps
Recall Ps
Precision
Recall Ps
Method
k
F
D
F
VT
69.2
81.2
80.5
85.8
96.6
95.7
77.2
83.0
38.2
62.9
83.9
88.6
RAG
5
58.9
71.5
66.5
71.0
91.7
85.5
71.7
87.4
26.8
58.5
61.3
54.5
VT-RAG
59.6
75.5
75.1
82.5
96.7
96.2
63.0
74.1
28.4
53.3
87.1
90.9
RAG
15
66.6
72.2
72.7
70.2
93.1
84.5
81.0
92.6
36.4
67.7
64.5
47.7
VT-RAG
15
61.5
77.7
75.4
82.8
96.1
94.7
66.8
79.3
29.9
57.6
83.9
86.4
RAG
25
63.9
69.3
70.0
67.1
92.4
82.9
78.8
93.3
32.8
66.7
61.3
40.9
VT-RAG
25
51.9
73.8
70.4
78.3
96.9
92.0
50.5
77.0
23.5
53.0
90.3
79.5
33

---

Published as a conference paper at ICLR 2026
E.2 VERITRAIL INPUT SIZE LIMIT
As explained in $31.2 and 3.13 VeriTrail allows users to set an input size limit per prompt for both Evidence Selection and Verdict Generation. If no limit is specified, the LM's context window size is used. Separate limits can be specified for root and non-root nodes.
In our experiments, we set the Evidence Selection limit to 40 sentences per prompt for all nodes. For Verdict Generation, set the limit 200 sentences for non-root nodes, with no limit for root nodes. This means that for Evidence Selection, nodes were split into sentences and divided into prompts of up 40 sentences each. For Verdict Generation, the prompt was capped 200 sentences no root nodes were included in the evidence; if any root nodes were included, the input limit defaulted to the context window size.
We evaluated the effect of the input size limit through an ablation study. We did not vary the limit for Verdict Generation because all evidence must in single prompt. Decreasing the limit would only force compression or removal of evidence, which is unlikely to improve performance. Increasing the limit would have minimal effect: for non-root nodes, our default limit was rarely exceeded, and for root nodes, the limit was already set to the full context window size. Instead, we focused our ablation study on input size limits for Evidence Selection. Unlike Verdict Generation, Evidence Selection allows any number of prompts, making it more meaningful setting for studying the impact of input size limits.
We hypothesized that the key risk of lower limit (i.e., using many short prompts) is context loss. For example, consider a node containing the following sentences: "John began his career at Company A. He later worked at Company B." If both sentences appeared in the same Evidence Selection prompt, they would likely be selected as evidence for the claim "John worked at Company A prior to Company split across different prompts, the second sentence might not be selected because it would be unclear who "He" refers to without the preceding sentence.
B." " However, if the sentences
Conversely, hypothesized that the key risk of higher limit (i.e., using few long prompts) is reduced recall. LMs are known to struggle with needle-in -nallliin---hattckk retrieval, where they must identify specific pieces of information within a long context (Kamradi) 2023). Evidence Selection is even more challenging than traditional needle-in-a-hayttan sentences ("needles") may exist, and (b) complex reasoning sometimes required to assess the logical relationship between each sentence and the claim.
tasks because (a) multiple relevant
We evaluated four input size limits in addition to our default of 40 sentences per prompt: 20, 80, 160, and 320 sentences. Table 4 shows the results. On DiverseSumm+, the default setting achieved the highest macro F1 and balanced accuracy. On FABLES+, the 160-sentence condition performed best.
Table 4: Effect of input size limits (in sentences) for Evidence Selection on hard prediction per- formance for FABLES+ (F) and DiverseSumm+ (D). We report macro Fi, balanced accuracy (Bal. Acc.), and class-specific precision and recall for fully supported (FS) and not fully supported (NFS) claims as percentages. Bolded values represent the highest score in each column.
Macro F,
Bal. Acc.
Precision Fs
Recall Fs
Precision NFS
Recall
Input Limit
F
F
F
D
D
D
D
20
66.2
80.3
81.1
84.6
97.9
95.0
71.9
81.4
34.1
62.3
90.3
87.8
40
67.4
81.0
78.6
85.6
96.1
95.8
76.6
81.4
35.7
62.9
80.6
89.8
65.4
72.4
78.4
76.0
96.6
89.9
72.9
76.4
33.3
52.9
83.9
75.5
160
67.8
76.1
82.1
82.7
97.9
96.3
74.0
73.6
35.9
54.9
90.3
91.8
320
65.4
75.4
81.7
81.7
98.5
95.4
69.8
73.6
33.3
54.3
93.5
89.8
One possible explanation for this difference is the nature of the source material. Claims from Divers- eSumm+ were evaluated against many short, self-contained articles, while claims from FABLES+ were evaluated against a single long book composed of interdependent parts. Articles also tend to be written in compact, expository style, whereas fiction books typically more verbose, with ideas unfolding gradually through narration and dialogue. As a result, evidence required to verify FABLES + claims is less likely be concentrated within narrow span of text, and therefore, may
34

---

Published as a conference paper at ICLR 2026
benefit more from context preservation. However, in both datasets, performance declined at the highest limit we tested (320 sentences), suggesting that regardless of source material type there may be tipping point where the benefits of context preservation outweighed by losses in recall.
E.3 SOFT PREDICTION
In the soft prediction setting, introduced 4..3 the probability that claim is fully supported. Only AlignScore, INFUSE, and Llama-3.1-PP -Bespoke- MiniCheck-7B natively produce continuous scores. For the remaining methods (except Gemini 1.5 Pro, which was excluded due to cost constraints), we sampled three verdicts at a temperature of 0.2 and calculated the proportion labeled "Fully Supported." This approach predicated on prior work demonstrating that the consistency of an LM's outputs across samples can be used proxy for confidence (Wang et al. 2023 Tian et al. 2023
methods produce a continuous score representing
Recall that VeriTrail (a) performs Evidence Selection before Verdict Generation, and (b) generates interim verdicts as it traverses the DAG before producing a final verdict. To approximate different confidence thresholds for these intermediate steps, we tested three thresholds, t E (1,,,,3) (1,2.3)
For each setting during Evidence Selection, we generated three outputs using temper- ature of 0.2. A sentence was included as evidence only if it was selected in at least & runs. If no sentence met this condition for a given claim, verification was terminated. As no final verdict was generated in these terminated cases (10% and 11% of claims for FABLES+ and DiverseSumm+, respectively), we excluded them from soft prediction evaluation across all methods.
For Verdict Generation, we likewise generated three outputs using temperature of 0.2. For each interim iteration, a verdict "passed" appeared in at least outputs. multiple verdicts passed, we selected the one that appeared most often across the outputs. If no verdicts passed, or passing verdicts tied in frequency, the verdict for that iteration was to "Inconclusive."
We used the proportion of "Fully Supported" verdicts from the final iteration as the soft prediction score.
evaluated soft predictions using Area Under the ROC Curve (AUROC), which measures method's ability to distinguish supported from unsupported claims across varying classification thresholds. Table 5) shows the results for both datasets. All VeriTrail variants outperformed the baseline methods, with the t = 2 variant achieving the best results.
Table 5: Soft prediction results for the FABLES+ (F) and DiverseSumm+ (D) datasets. For RAG and AlignScore, we report the best-performing configuration by macro Fi: RAG uses k = 20 for FABLES+ and & 15 for DiverseSumm+; Llama- 3.1 -es--esso--einii MiniCheck-7B (denoted "Bespoke- MiniCheck-7B") uses l = 4 for FABLES+ and = for DiverseSumm:; AlignScore uses k = for FABLESS and & 2 for DiverseSumm+. Bolded values represent the highest AUROC in each column.
AUROC
Method
VeriTrail (t 1)
0.86
0.79
VeriTrail (t 2)
0.88
0.87
VeriTrail (f 3)
0.85
0.80
RAG
0.81
0.76
0.79
0.79
GPT-4.1 Mini
0.67
0.61
AlignScore INFUSE
0.57
0.68
0.64
0.58
35

---

Published as a conference paper at ICLR 2026
E.4 FULL HYPERPARAMETER CONFIGURATIONS
Figure 4 and Figure 5 report hard prediction results for all hyperparameter config- urations tested for AlignScore, INFUSE, Llama-3. -ea.aaa..--espoo--MMee Drspook--inniCeekk-1. and RAG, respectively. Hyperparameter definitions are provided in $4.2
Figure 2 Figure 3
AlignScore Results
Accuracy [%)]
17.4
55.5
54.3
61-7
46--
60.0
$4.8
60.2
61.3
41.3
S6.
96.8
50
06
11.2
06.8
63.4
17.0
06.1
08.1 35.8
37.0
26.0
15.1
91-4
41.5
62.7
65.7
652
60.9
67.0
60-1
63.4
60.0
94.1
60-
16.8
94.
16.6
1611
48.
412
19.8
17.1
15.2
10.2
32.5
10.9
25-9
18.5
27.4
26.
25.9
65.0
66.3
12.7
60.6
00.3
99.
56.3
66.2
60
55.4
56.
56.0
57.5
57.3
56.9
20.2
68
24
15
Mecro 1:1 (16) for Diversebuumm+
Uel lenaecee Accuracy (26) for DiverseSuuTT
0.3
43.0
49.5
45.0
40.2
50.0
50.8
50.0
50.3
40.0
02
02
20.3
54.0
53.4
53.0
617
53.6
50.1
40
46.4
48.0
48.3
S4.5
F4.5
53.7
52.3
40.4
40.5
48.3
49.1
M6.1
Mir W/
DS0 N/W
161
663
11.
10.5
50
192
16.5
366
320
83
00.6
372
3510
36.3
13.3
07.5
10.8
12.9
13.0
29.8
26.
579
573
260
50
#2-7
10.3
06.2
10.
26.3
20.5
300
32.
02.5
30.3
320
12.0
Figure 2: Hard prediction results for all AlignScore configurations on the FABLES+ and Divers- eSumm+ datasets. We varied the threshold used convert entailment probabilities into binary labels (r) and the number of chunk-level probabilities averaged (k). Each value shows the performance for specific (r, k) pair.
INFUSE Results
Macro F1
Balanced Accuracy
80
80
Sclancoo Acc Irayy Ph1
60
55.2
Pocro (%)
80
32.1
-00
24.8
20.0
19.7
19.
17.3
19.7
0.6
3.7
09
0.5
0.7
DiverseSuumm
DiverseSumm
Figure 3: Hard prediction results for all INFUSE configurations on the FABLES+ and DiverseSumm+ datasets. We varied the threshold used to convert entailment probabilities into binary labels. Dashed lines indicate the best result across all methods from Table
36

---

Published as a conference paper at ICLR 2026
Llmm-3--deppote-Miiii------ Results
Macro F1 (86) FABLES
Balanced Accuracy (%) for FABLES
30.0
15.2
23.0
24.4
0/2
30.6
561
43.2
24.0
22.0
15.6
340
9/.4
16.4
53.1
525
50.4
0.
342
340
0.
53.5
53.0
50.0
Mecro F1 (16 for DiverseSumm+
Bal laa lancedd Accuracy (36) for DiverseSumm
03
10.1
12 13.7
61.
67.0
59.
56.5
56.
64.3
70.6
752
T0
63.7
614
02
02
60: 668
6..4
68.4
66-4
63.5
05.
833
29.2
37.
X1.9
72. M8.
12.3
20:8
08.2
07.1
10.2
60.
01.0
393
362
102
65.7
65.7
26.3
33.6
52.0
18.3
61.8
12.1
37.0
56.6
111
2911
27.5
28.3
73.4
13.5
72.
00.
06.0
65.8
t3.0
63.2
05.9
20.3
37.4
56.3
315
55.9
552
Figure 4: Hard prediction results for all Llama-3.1 Beeeeoooe--iiiChheeeeee configurations on the FABLES+ and DiverseSumm+ datasets. We varied the threshold used convert entailment probabilities into binary labels and the number of chunk-level probabilities averaged (k). Each value shows the performance for specific (r, k) pair.
RAG Results
Macro F1
Balanced Accuracy
25.1
71.5
7y2 742
00.5
$7.5
70
67.4
26.3
20.
Natro (ea)
80-
Salancos
.00
23
TRBLES
DiverseSummr
Figure Hard prediction results for RAG configurations on the FABLES+ DuverseSumm+ datasets. We varied the top k chunks retrieved. Dashed lines indicate the best result across all methods from Table
37

---

Published as a conference paper at ICLR 2026
E.5 ALTERNATIVE VERDICT GENERATION PROMPT
Table reports hard prediction results for RAG, Gemini 1.5 Pro, GPT-4.1 Mini, and VeriTrail, using two different prompts for verdict generation: (1) VeriTrail's default prompt (see Appendix C.2.2).
and (2) a prompt from the original FABLES paper, shown below:
You are provided with context and statement. Your task is to carefully read the context and then determine whether the statement true or false. Use the information given in the context to make your decision.
Context: {context)
Statement: {claim)
Question: Based on the context provided, is the above statement True False?
Answer:
Table 6: Hard prediction results (%) for the FABLES+ and DiverseSumm+ datasets using VeriTrail's prompt for Verdict Generation ("Orig.") and prompt from the original FABLES paper ("Alt."). We report macro F1, balanced accuracy (Bal. Acc.), and class-specific precision and recall for fully supported (FS) and not fully supported (NFS) claims. Bolded values indicate the better score in each Orig./Alt. pair.
Macro F;
Bal. Acc.
Precision
Recall
Precision
Recall
Dataset
Method
Orig.
Orig.
Orig.
Orig.
Alt.
Orig.
Orig.
VeriTrail (q 1)
67.9
66.6
79.0
70.4
96.1
92.1
76.7
84,5
36.6
37.5
81.2
56.2
GPT-4.1 Mini
63.2
66.1
60.2
62,8
88.4
89.1
98.4
97.4
70.0
64.3
21.9
28.1
FABLES+
RAG (R: 3)
56.2
54.9
63.5
$7.7
90.7
88.1
69.0
76.6
24.0
21.8
58.1
38.7
RAG (R: = 5)
58.9
58.2
66.5
61.7
91.7
89.4
71.7
78.3
26.8
25.9
61.3
45.2
RAG (k 10)
60.4
59.7
64.9
62.8
90.6
89.7
78.3
80.4
28.6
28.0
51.6
45.2
VeriTrail (g
81.0
70.4
85.6
68.2
95.8
82.4
81.4
93.6
62.9
70.0
89.8
42.9
GPT-4.1 Mini
60.6
59.7
54.4
78.4
92.9
98.6
56.5
71.4
26.5
10.2
DiverseSumm+
RAG (f:
84.8
85.4
90.5
52.3
RAG (k:
58.4
85.6
79.1
86.9
94.2
57.1
55.6
54.5
22.7
RAG (k- 10)
66.7
58.9
65.3
58.4
82.6
79.0
89.8
96.4
56.2
64.3
40.9
20.5
E.6 VERITRAIL TERMINATION CONTROL
Table reports hard prediction results using different values for VeriTrail on FABLES+ and DiverseSumm+. At most iterations are possible for FABLES+ and 5 for DiverseSumm+. Therefore, their maximal q values are and 5, respectively; these settings, VeriTrail always verifies at least one root node. We also include results for RAG, the best pest-performing baseline method, for direct comparison. All VeriTrail variants outperformed the baseline.
E.7 ADDITIONAL MODELS
Table reports hard prediction results for VeriTrail and RAG (the top-performing base- line method) with the DeepSeek-V2 gemini-2 5- - 04-17, and mistral-large-241 models.
38

---

Published as a conference paper at ICLR 2026
Table 7: Hard prediction results (%) for the FABLES+ (F) and DiverseSumm+ (D) datasets for VeriTrail at varying values and RAG. For RAG, we report results using the best-performing k value by macro Fi (k=15 for k=30 for D). We report macro Fi, balanced accuracy (Bal. Acc.), and class-specific precision and recall for fully supported (FS) and not fully supported (NFS) claims. A dash (-) indicates that the configuration was not evaluated. Bolded values indicate the highest score in each column.
Macro Fi
Bal. Acc.
Precision PS
Recall
Precision NFS
Recall n P8
Method
Setting
F
D
F
D
F
D
F
D
F
D
F
D
1
69.1
80.7
80.5
85,5
96.6
95.7
77.0
82.4
38,2
61.9
83.9
88.6
2
80.4
76.5
85.4
74.7
96.5
86.9
90.2
92.6
58.1
71.4
80,6
56.8
VeriTrail
3
85.7
76.9
87.6
73.5
96.6
85.7
94.5
97.1
71.4
84.6
80.6
50.0
4
74.6
71.3
84.6
97.1
83.3
45.5
= 5
74.6
71.3
84.6
83.3
45.5
RAG
best-k
66.5
74.3
72.7
72.9
93.1
86.1
80.9
91.2
36.4
66.7
64.5
54.5
Table 8: Hard prediction results (%) the FABLES+ (F) and DiverseSumm+ (D) datasets for Veri- Trail and RAG with the DeepSeek-v4 (DeepSeek), gemini-2 5---225-.... w-.5ffllssppppp sallash-peiiees 04 1:4-- -17 (Gemini 2.5 Flash), and mnstrall-aarge-i performing & value based on macro Fi: DeepSeck 3/30, Gemini = 5/10, and Mistral = 5/10, for F/D, respectively. We report macro F1, balanced accuracy (Bal. Acc.), and class-specific precision and recall for fully supported (FS) and not fully supported (NFS) claims. Bolded values indicate the best-performing method for each dataset and metric.
(Mistral) models. For RAG, we use the best-
Macro F,
Bal. Acc.
Precision
Recallps
Precision
Recall yes
Model
Method
F
F
VeriTrail (g = 1)
61.7
68.0
70.9
73.2
93.4
89.8
73.1
68.8
29.7
46.3
68.8
77.6
DeepSeck
RAG
59.4
66.0
63.1
63.9
90.0
80.1
79.3
97.2
27.3
78.9
46.9
30.6
VeriTrail (g
70.0
69.9
80.2
72.3
96.2
87.2
79.2
77.3
39.4
50.8
81.2
67.3
Gemini 2.5 Flash
RAG
66.6
68.9
73.2
93.4
82.5
36.2
65.6
44.9
VeniTrai (g 1)
49.2
67.0
67.2
73.8
95.5
91.7
49.7
64.7
20.6
44.8
84.6
83.0
Mistral
RAG
55.2
67.6
62.1
66.5
90.8
82.2
70.4
88.2
21.9
56.8
53.8
44.7
39

---

Published as a conference paper at ICLR 2026
F NLI BASELINE DETAILS
In 84.2 we provided an overview of the NLI basclines evaluated in our experiments: AlignScore (Zhal
al. 2023). INFUSE (Zhang al. 2024a), and Llama-3.1 we----ooe---iiihheeeee
(Bespoke Labs
2024). We implemented all methods using the official repositories and default hyperparameter settings. For AlignScore and INFUSE, the only modification we made was applying our sentence-spliting method (described in 331.2 to ensure consistency with VeriTrail.
For AlignScore, used the top-performing model from the original paper: AlignScore--arge (355M parameters), which is based on RoBERTa-large (Liu et al. (2019). For INFUSE, we used the same 2020), fine-tuned on MNLI (Williams et al. 2018) and VitaminC (Schuster et al. (021). with 58.7M parameters. We accessed
NLI mode as the original paper: an ALBERT-xlaare -xlarge model (Lan
Llama-3. Beenalll-BBpoo----iicehe-------- MiniCheck-BB via Hugging Face The model based on InternL.M2.5-7B.
Chat (Cai et al.
2024), fine-tuned on ANLI (Nie et al. 2020) and ceathetically-gneeaatee data.
For FABLES+, each claim was compared single document: the corresponding book. For Divers- eSumm+, which contains multiple documents (i.e., articles), we tested all pairwise combinations of documents and claims. For AlignScore and Llama-3. Deepoke-MMinihhell A chunks (across single document for FABLES+; across all documents for DiverseSumm+) with the highest probability of entailment for a given claim. For INFUSE with DiverseSumm+, we used the document with the highest entailment probability for each claim.
we selected the top
AlignScore and INFUSE were run for 12 and 24 hours, respectively, on an NVIDIA A10G GPU. Llama-3.1 1.-ma--.-BBesooke Bewpooee-innChheeeeee MiniCheck-7B run for 20 hours an NVIDIA A100 GPU.
G LIMITATIONS
To better understand VeriTrail's limitations, we manually reviewed 60 error cases per dataset. For each dataset, we randomly sampled claims from the q = and y settings, evenly split between false negatives (claims incorrectly labeled as "Fully Supported") and false positives (claims incorrectly labeled as "Not Fully Supported"). All analyzed outputs were produced using go 202 -08---000000000 08-06.
We identified two primary causes false negatives. The first was invalid inferences. For example:
Claim (from FABLES+): Charlie Brown is eager to reunite with her husband Harry
Assigned Label: Fully Supported
Model's Reasoning: "The evidence strongly supports the idea that Charlie is eager to reunite with Harry... The evidence does not explicitly state that Harry is her husband, but it is implied through the anticipation of reunion."
Our Analysis: Charlie's excitement about reuniting with Harry does not, on its own, imply that he is her husband.
The second cause of false negatives was overlooking components of the claim. For instance:
Claim (from DiverseSumm+)) NATO and its member states have supplied weapons to bolster Ukraine's defense capabilities [in response to Russia's aggression)
Assigned Label: Fully Supported
"The evidence from the excerpts clearly indicates that NATO member states, such as the United States and various EU countries, have supplied weapons and military aid to Ukraine... The evidence does not explicitly mention NATO as organiza- tion supplying weapons, but does strongly imply that NATO member states are actively involved..."
Model's Reasoning:
The evidence supports the role of member states, but not NATO as an organization. The model acknowledges this fact but ultimately glosses over in its final judgment.
Our Analysis:
ch
40

---

Published as a conference paper at ICLR 2026
For false positives, the sources of error more varied. One example omission of relevant context from the evidence:
Claim (from DiverseSumm+)): Nike's apparel units in North America declined by high- single digits
Assigned Label: Not Fully Supported
In the Evidence Selection step, the model correctly selected the following text evidence: "Region where Nike saw decline in apparel units by high-single digits." However, failed to include a related header, "NORTH AMERICA (GEO):", that appeared several lines earlier. As result, it generated the following evidence summary: "Sentence 12 states that Nike saw a decline in apparel units by high-single digits in a region." In the Verdict Generation step, reasoned follows: "The only relevant sentence indicates a decline in apparel units by high-single digits in region but does not specify that the region is North America."
Model's Reasoning:
The model identified the correct evidence, but missed the broader context needed to interpret it. Our Evidence Selection prompt instructs the model to include sentences that provide critical context; however, this example demonstrates that identifying such sentences can be challenging when they are dispersed or presented in an ambiguous form (e.g., sentence fragements).
Our Analysis:
We also observed differences in enror types across the datasets. In FABLES+, some error cases involved multi-hop reasoning (i.c., combining multiple pieces of evidence, often located different parts of the source text, support refute claim). For example:
"Weyward" chronicles the interconnected narratives of three women named Altha, Violet, and Kate
Claim (from FABLES+):
Assigned Label: Not Fully Supported
Model's Reasoning:
"The excerpts clearly mention narratives involving Altha, Violet, and Kate... However, the excerpts do not provide strong evidence that these narratives are interconnected."
This claim is difficult to verify because the evidence linking the three characters is subtle and dispersed throughout the story. It is unlikely that single Evidence Selection step would surface all connections needed to fully support the claim.
Our Analysis:
In DiverseSumm+, we did not observe multi-hop reasoning errors. This is likely because its source material consists of self-contained articles on distinct topics, as opposed long narrative with interdependent parts.
unique error type in DiverseSumm+ was the use of parametric knowledge, presumably due the real-world nature of the source material. For instance:
Claim (from DiverseSumm+): NVIDIA collaborates with other tech giants [besides itself]
Assigned Label: Fully Supported
Model's Reasoning: "The evidence ... clearly indicates that NVIDIA has partnership with Alphabet Inc.'s Google Cloud Platform, which is a tech giant. Sentence 45 suggests that Oracle provides cloud infrastructure for NVIDIA's DGX Al supercomputer. and Oracle is generally considered tech giant."
Our Analysis: The model correctly identified relevant collaborations. However, the desig- nation of Alphabet and Oracle technology giants was neither explicitly stated nor implied by the evidence and likely reflects reliance on parametric knowledge.
Collectively, these findings highlight several opportunities for future work, including mitigating common reasoning emrors (e.g. invalid inferences, reliance on parametric knowledge, etc.), supporting multi-hop claim verification, and exploring how the nature of the source material affects error patterns. We also flag that VeriTrail is not tied to using LM for verification. The Verdict Generation step is modular and can be replaced with any verifier that assesses claim's 's faithfulness based on evidence, experimentation with alternative verifiers is another opportunity for future work.
41

---

Published as a conference paper at ICLR 2026
H ERROR STAGE ANALYSIS
In this section, we address questions:
1. Which stages are the most common sources of hallucination in the MGS processes we evaluated: hierarchical summarization (used in FABLES+) and GraphRAG (used in Divers- eSumm+)?
2. How consistent is error stage identification across VeriTrail variants? For instance, if we run VeriTrail with 1 instead of = 3 on a set of claims, how similar are the resulting error stage distributions?
To answer these questions, we analyzed the VeriTrail variants used in our ablation studies, cover- ing different input size limits (Appendix E.2), confidence thresholds (Appendix E.3), values of (Appendix E.6p, and models (Appendix E.7) We included 13 variants for FABLES+ and 15 for DiverseSummt.
For each variant, we identified its true positive claims, defined the set of claims that met all the following conditions:
The claim from the subset of FABLES+ DiverseSumm+ described in Appendix (since not all variants were evaluated on the full datasets);
The claim was correctly labeled "Not Fully Supported"; and
At least one error stage was identified for the claim (see $3.2 for cases where error stage identification is not possible).
The average number of true positive claims per variant was 24 for FABLES+ and 32 for Divers- eSumm+.
H.1 WHICH STAGES ARE MOST PRONE TO IILLUCINATTION??
For given VrriTrail variant and possible enror stage let:
true positive claims for where identified as error stage Po.s
# true positive claims for v
In other words, stages with a higher value of were more frequently identified as sources of hallucination by variant v.
We computed Pe.s for all combinations of u and s. As noted in Appendix B.I.I and Appendix B.2. there 4 stages FABLES+ and in DiverseSummt. However, root nodes (stage cannot the source of hallucinations, leaving 3 and 5 possible error stages, respectively.
To aggregate across variants, we computed the weighted average for each stage s:
Ee Rey Pe.s
En Tly
where nyy is the number of true positive claims for variant v. This weighting reflects the intuition that variants with more true positives are likely more reliable and should have greater influence on the overall estimate
Let CTP denote the all true positive claims across all variants. To estimate uncertainty, we applied bootstrap resampling over Crp. Specifically, we performed 1,000 iterations in which we sampled with replacement the same number of claims as Crr and recomputed p. for each stage. The resulting distribution was used to compute 95% confidence intervals. Final estimates for both datasets are reported in Table 9
We also tested unweighted averages and observed negligible differences.
42

---

Published as a conference paper at ICLR 2026
Table 9: Error attribution rate (Pa) for each possible error stage in FABLES+ and DiverseSumm++ This metric captures the proportion of correctly identified hallucinated claims (i.e., true positives) for which given stage was identified as a likely source of error. Values are averaged across VeriTrail variants, weighted by the number of true positives per variant. 95% confidence intervals were estimated via bootstrap resampling. Bolded rows indicate the most frequently implicated stage for each dataset.
Dataset
Stage
Error Attribution Rate (95% CI)
2
0.20 [0.16, 0.24]
FABLES+
3
0.47 [0.42, 0.53]
(Hierarchical Summarization)
4
0.32 [0.27, 0.37]
2
0.15 [0.12, 0.18]
3
0.09 [0.07, 0.12]
DiverseSumm+ (GraphRAG)
4
0.41 [0.36, 0.45]
5
0.13 [0.10, 0.16]
6
0.22 [0.19, 0.26]
H.2 HOW CONSISTENT ERROR STAGE IDENTIFICATION?
We assessed the consistency of error stage distributions across VeriTrail variants using two metrics:
1. Jensen-Shannon Divergence (JSD; Lin [991] measures the similarity between two prob- ability distributions. It bounded between and 1, with lower values indicating greater similarity. We computed the JSD for all pairs of variants and observed a mean JSD of 0.02 for FABLES+ and 0.03 for DiverseSumm+, suggesting that the error stage distributions were highly consistent across variants.
2. Spearman Rank Correlation (Spearman 1961) measures the similarity between rankings assigned
of items in case, the relative frequency of different error stages. ranges from -1 (perfect inverse agreement) to (perfect agreement). Across all pairwise comparisons, the mean correlation was 0.67 for FABLES+ and 0.66 for DiverseSummt, indicating substantial agreement in the ranking of stages across variants.
43


Reviews:
[{"review_id": "r36z8DUs0J", "reviewer": "Reviewer_1wLF", "summary": "This paper proposes a hallucination detection framework for multi-step generation process. It models the tracibility of generated content (the chain of rationale) with a directed acyclic graph and convert the hallucination detection of the final problem to a traverse on graph. Experiments on book summary and news summary shows that it outperforms previous method.", "questions": "See the weaknesses above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ The paper proposes a new framework for hallucination detection in multi-generation process.\n+ It curates two datasets for faithfulness evaluation in MGS based on previous benchmarks.\n+ The proposed VeriTrail framework shows a substaintial improvement over previous baseline.", "weaknesses": "+ Missing intermediate results. The ablation analysis In Appendix E shows that evidence selection using language model plays an important role in final performance so it would be necessary to include the accuracy of the evidence selection in evaluation.\n\n+ Since VeriTrail uses stage-by-stage verfiication and does not have to always checks the source materials if the verfiication process exits early, I would suggest analyzing the API token cost of VeriTrail and comparing with the baseline method.\n\n+ a minor comment is to use vector graph in Figure 1 for better readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5UmvfmMMDD", "reviewer": "Reviewer_xoWU", "summary": "This paper addresses closed-domain hallucination in complex, multi-step generative (MGS) processes, where error risk is amplified. The authors argue that simple detection is insufficient for MGS and propose \"traceability\", the ability to pinpoint where errors are introduced. They present a novel method that models the generation process as a graph to achieve both detection and traceability. To evaluate this, new datasets with intermediate steps were created. Experiments show the proposed method outperforms baselines in detection while also providing this novel traceability.", "questions": "Please refer to the previous section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper tackles the critical problem of error propagation in MGS processes. The focus on \"traceability\" beyond simple detection is a key conceptual contribution.\n\nThe method is tested against strong, comprehensive baselines.", "weaknesses": "A primary weakness of this method is its recursive reliance on Large Language Models (LLMs). Core steps of VeriTrail, including sub-claim decomposition, evidence selection, and final verdict generation, are all dependent on LLMs. This creates a fundamental \"verifier's dilemma\": the LLM used for verification may itself hallucinate or make faulty inferences.\n\nThe paper's own error analysis in Appendix G concedes that the verification model itself can make \"invalid inferences\" or improperly use \"parametric knowledge.\" This directly undermines the claimed reliability of the detection method, as the verification process itself may introduce new errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oBTjXDVzQ1", "reviewer": "Reviewer_8wrP", "summary": "In this paper, the authors analyze hallucination detection in generations of autoregressive LLMs in a closed-domain setting. As opposed to just single-step generation (such as with RAG), the paper analyses hallucinations in a more complex and challenging setting, involving multiple generative steps (MGS). Toward this, the authors propose VeriTrail, which assesses the faithfulness, as well as provenance tracing and error localization using a multi-stage graph-based framework. The method begins with the final output and uses backward traversal to identify the stage at which hallucinations may have been introduced. The paper further introduces two new datasets FABLES+ and DiverseSumm+, and demonstrates the efficacy of VeriTrail over existing baselines like RAG and long-context models toward hallucination detection.", "questions": "Kindly refer to the questions mentioned in the weaknesses section above. I would be happy to raise my score further if these could be adequately addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper studies a pertinent problem of detecting hallucinations in the generations of autoregressive LLMs, while also focusing on provenance tracing and error localization with multiple-generation steps, which is highly relevant given complex, multi-agent systems that have gained prominence in recent times. By identifying the intermediate stage at which hallucinations are introduced in a closed-domain setting, specific and actionable modes of improvement can be identified and undertaken in these complex systems in a transparent manner.\n\n2) The DAG-based framework adopted by VeriTrail is principled, and helps assess the veracity of claims and sub-claims in a systematic manner. Furthermore, selective verification helps reduce unnecessary and wasteful computation over nodes that correspond to uninformative texts relative to the primary claim. \n\n3) A significant and notable contribution of the paper is the introduction of two new datasets, FABLES+ and DiverseSumm+, that are specific to the MGS setting, and include intermediate outputs as well as human-annotated evaluations on faithfulness. Furthermore, the proposed method VeriTrail is shown to be effective, and out-performs strong existing baselines on these datasets.", "weaknesses": "1) The proposed method VeriTrail is not simple, and requires a fairly complex setting up of the generative process as a DAG, and ensure that each intermediate output is carefully produced and analyzed in a multi-step manner with verification by an LLM. This also places a constraint on its scalability in practice for real-time applications, given the need for multiple LLM calls for each claim/sub-claim in the final output. Could the authors also kindly provide some metrics with respect to the actual average runtime required for the overall verification pipeline, and compare with more standard approaches like RAG itself?\n\n\n2) This complexity also manifests in the computational overheads and costs involved for improved performance. For instance, as presented in Appendix D, when comparing with the same LLM model (gpt-4o-2024-0806), VeriTrail  appears to cost 5x-9x more than RAG itself. These added costs can further limit practical usage in several real-world settings. \n\n\n3) The quality of VeriTrail's analysis depends critically on the decomposition of the final output into factual claims, using Claimify. Given this critical dependance, if Claimify makes an ambiguous phrasing of a claim, or introduces an error in itself, the proposed method cannot recover in a meaningful manner.\n\n\n4) Stage-assignment analysis: Must this setup for stage-assignment logic be explicitly programmed by a developer for each new MGS system, or does the framework offer any mechanism to automatically infer or propose a logical stage hierarchy from the process graph itself? Furthermore, how sensitive is the accuracy and utility of the 'error stage' localization to the granularity of this breakdown? Would this be more dependent on the actual facts presented, as opposed to the stage-specific assignments given the closed-domain setup?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3xNLibXr9m", "reviewer": "Reviewer_xtw1", "summary": "**Summary**\n\nThe paper introduces an algorithm for detecting hallucinations and provide traceability in processes with multiple generative steps (MGS). Processes with multiple generative steps involve multiple calls to an LLM and typically involve feeding output of a step as the input of a subsequent step. In the context of the paper, traceability includes identifying the source from which a claim was derived and also, in which subprocess any error(s) were introduced.\n\n**Contributions**\n- An algorithm for detecting hallucinations in processes with MGS\n- Two new datasets for faithfulness of LLM outputs that include outputs of intermediate steps", "questions": "- With regard to the weakness 1 mentioned above, do you think VeriTrail is applicable to tasks other than summarization? If yes, please mention them and how would you apply VeriTrail in such tasks?\n- What was the rationale for each step in VeriTrail being done with a large monolithic prompt? Have you considered breaking it down into smaller steps with more specific instructions? I am asking this because I see that the prompts in the appendix are written in a step-by-step fashion, asking the LM to do one after the other in a single prompt.\n- I am curious to see if using something like DSPy [1] for optimizing the prompts would lead to better performance. \n\n**References:**\n\n[1] DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (Khattab et al: [https://arxiv.org/abs/2310.03714](https://arxiv.org/abs/2310.03714))", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses an important and underserved area of combating hallucinations in LLM outputs. Namely, in complex tasks that involve several LLM calls with potential chaining from output of one call to the input of another. \n- I commend the clarity with which they have described their framework.\n- I understand that in newer research directions, it can be challenging to find strong benchmarks/baselines to compare. So, I appreciate the effort in making their own datasets (FABLES+, DiverseSumm+).", "weaknesses": "- As the authors state at the end of the paper, VeriTrail is a reference-based method, meaning that it relies on relevant source text and compares claims with this source text. So, its applicability is inherently limited to transformative tasks like summarization (and not, say, to question-answering when the LLM is relying on its internal knowledge). Even the benchmarks that are used in the experiments are both essentially summarization tasks. While I don't mind a method limited in applicability, this limitation should be presented clearly.\n- None of the baselines assume or use the DAG of the generative process with MGS. IMO, this makes them weak baselines for comparison.\n- The prompts used in VeriTrail have some examples or clarifications. How were these chosen? The effect of these on the performance of VeriTrail is unclear. I am afraid that if they were added in an ad-hoc fashion, would these necessarily translate to other tasks or benchmarks?\n\nUltimately, I am giving the paper a score of 2 due to the above reasons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)