Token-Guard: Towards Token-Level Hallucination Control via Self-Checking Decoding
Authors: Yifan Zhu
Huiqiang Rong
Haoran Luo
Venue: ICLR 2026 Poster (2026)
Keywords: token_level
hallucination control
self checking
Primary area: interpretability and explainable AI
TL;DR: Token-Guard applies self-checking decoding for token-level hallucination control, enhancing LLM quality and reliability.

Abstract: Large Language Models (LLMs) often hallucinate, generating content inconsistent with the input. Retrieval-Augmented Generation (RAG) and Reinforcement Learning with Human Feedback (RLHF) can mitigate hallucinations but require resource-intensive retrieval or large-scale fine-tuning. Decoding-based methods are lighter yet lack explicit hallucination control. To address this, we present \textbf{Token-Guard}, a token-level hallucination control method based on self-checking decoding. Token-Guard performs internal verification at each reasoning step to detect hallucinated tokens before they propagate. Candidate fragments are further evaluated in a latent space with explicit hallucination risk scoring, while iterative pruning and regeneration dynamically correct detected errors. Experiments on HALU datasets show Token-Guard substantially reduces hallucinations and improves generation accuracy, offering a scalable, lightweight solution for reliable LLM outputs. Our code is publicly available\footnote{\url{https://github.com/rhq945/Token-Guard}}.

Published as a conference paper at ICLR 2026
TOKEN-GUARD: TOWARDS TOKEN-LEVEL HALLUCI- NATION CONTROL VIA SELF-CHECKING DECODING
Yifan Zhu', Huiqiang Rong', Haoran Lu
Beijjng University of Posts and Telecommunications *Nayyang Technological University {yi syifan_zhu, rhq] @bupt edu. en, haoran. luo@ieee.org
ABSTRACT
Large Language Models (LLMs) often hallucinate, generating content inconsis- tent with the input. Retrieval-Augmented Generation (RAG) and Reinforcement Learning with Human Feedback (RLHF) can mitigate hallucinations but require resource-intensive retrieval large-scale fine-tuning. Decoding-basee methods are lighter yet lack explicit hallucination control. To address this, we present Token-Guard, token-level hallucination control method based on self-checking decoding. Token-Guard performs internal verification at each reasoning step to detect hallucinated tokens before they propagate. Candidate fragments evaluated in a latent space with explicit hallucination risk scoring, while itera- tive pruning and regeneration dynamically correct detected errors. Experiments on HALU datasets show Token-Guse substantially reduces hallucinations and improves generation accuracy, offering a scalable, modular solution for reliable LLM outputs. Our code is publicly available"
further
INTRODUCTION
Large Language Models (LLMs) (Zhao et al. 2025) have achieved widespread success in natural language processing tasks. However, in knowledge-intensiee proprietary scenarios, they still suffer from the hallucination problem (Zhang al.) 2023). generating inaccurate or mislead- ing content. To improve factual consis- tency, strategies such as RAG (Lewis et al, 2020 Luo al. 2025b) and RLHF (Sti-) ennon et al. 2020) have been pro- posed. These methods alleviate hallucina- tions but heavily rely external knowl- edge retrieval or large-scale fine-tuning, which is resource-intensive. To address this, decoding-based methods have been adopted further enhance the model's generation process, improving both qual- ity and output reliability.
Guord Certo hwl.eintteessseeeeeeoeeeeeeee Inngeenteirsserr
Madispeth
gramaakkn
Sefebock
Swring
Complted agrapted patss
Paggend
Takea
Frammor Pad
Pavogg: 1525-11 sane lage máb fresies 1541: Crince Kan, croac da Gla oier se rafo undo concting
1335
lead lang amy Nincery.
Questiv How many yours aood the begervide
Talim-Gruud
Assrec: 14
Figure I: An illustration of TToeen-Gurrd
Existing decoding methods, including Auto- Lugo-Regressive Thain-of-Though((CoT)) (Wei et al. 2022 Tree-of-Thought (Yao et al. 2023), Guided (Xie al. 2023), and Predictive Decoding (Ma al.) 2024]. share three main characteristics: First, outputs are generated progressively through step-wise reasoning, allowing the model to build multi-step chains of thought. Second, exploratory generation is often adopted, where multiple reasoning paths are expanded in parallel to increase the chance of reaching a correct solution. Finally, intermediate self-evaluation provides feedback each reasoning step, enabling partial assessment of coherence and consistency before proceeding further.
Corresponding author. thwb. http:s////githhbb.ooo/////g/////

---

Published as a conference paper at ICLR 2026
Perforaace -1I Scare (1)
70-
39.1
30
47,6
30.8
20:3
20.3
Li
choosceBBeoch
BAGtratt
CosldQA
Drop, hiderr
Drop, atl
PubmeotjA
Hatuoral
Aerg
Comparison of F1 scores across HALU benchmarks, illustrating model performance on various tasks and effectiveness in hallucination mitigation.
Figure 2:
Despite these advances, current decoding methods still face three major challenges in hallucina- tion control: (i) Limited token-level hallucination checking. Most existing decoding methods do not implement an explicit token-level hallucination checking mechanism, relying instead on gen- eral representations without self-validation. This may allow token-level errors in reasoning steps to propagate and compound (Zhang et al., 2025). (ii) Hallucination risk not explicitly quantified. Traditional probability scoring methods fail to explicitly quantify hallucination risk in the encoded space, leading to undirected token selection and unstable reasoning results (Zeng et al. 2024). (iii) Limited dynamic correction capability in iterative and generation stages. Most methods sup- port only single-pass generation limited iterations, lacking mechanisms for dynamic correction and resource allocation, which undermines output consistency and may incur unnecessary time and computational costs (Ugare et al. 2024).
To address these challenges, we propose Token-Guard (see Figure a token-level hallucination- controlled decoding method. Token-Guse Guard introduces three key innovations: (1) Token-level Hal- lucination Control. At each reasoning step, we score candidate tokens in the latent space and prune low-confidence tokens, enabling the model to effectively detect and suppress hallucinated tokens, improving factual precision. (2) Segment-level Explicit Hallucination Scoring. We group relevant tokens into candidate fragments and assign hallucination risk scores, guiding more reliable and accu- rate fragment selection, thus improving relevance. (3) Local Enhancement and Global Iteration. When hallucinations are detected, local fragment regeneration and global iteration strategies are ap- plied to dynamically correct prior fragments. For multi- fragment reasoning paths, these strategies maintain logical consistency while avoiding unnecessary time and computational overhead.
We evaluated Token-Guse on multiple standard HALU datasets. Experimental results show that Token-Guaal when integrated with Ilm, achieves substantial improvements over both conventional decoding-based approaches combined with large models and the base large models themselves. Specifically, Token-Guard demonstrates significant gains in generation accuracy, with relative im- provements of up to 16.3% over the strongest baseline, and notable enhancements in output quality, including logical and factual consistency, as illustrated in Figure
RELATED WORK
LLM Hallucination Control. LLMs often produce hallucinations outputs that are factually in-
correct unsupported. Existing approaches address this through RAG (Lewis et al. 22000] Shuster et al. 2021 Luo et al. (222aa) with domain-specifer pipelines (Li et al. 2024) and benchmarks like
RrGTruth (Wu tiano et al. 2017; Stiennon et al. 2020 Bai et al. 2022) with multi-modal interactive exten- sions (Gunjal et al. 2023 AI- Yaychli et al. 2025). and post-generation detection via uncertainty estimation (Ledger & Mancinni 2024) or classifiers like HaloScope (Du et al. 2024). These meth- ods can be computationally intensive or domain-specific, limiting practical use.
2023). alignment methods including instruction fine-tuning and RI.FF (Chris-)
Decoding and Decoding Methods for Large Language Models. Decoding strategies significantly improve reasoning quality and efficiency. Chain-of-Thoune (CoT) enables explicit multi-step rea- soning. Methods such DoLa (Chuang al. 2024) and KCTS (Choi et al. 2023) introduce dif-
2

---

Published as a conference paper at ICLR 2026
ferent forms of token-level hallucination detection, allowing models filter unreliable tokens and enhance factual consistency during generation. Chi-Decoding (Xu et al.] 2025) uses adaptive prun- ing and foresight sampling to improve computational efficiency and better control hallucinations. Building on multi-level control (Hiriyanna & ZZhao| 2025) and self-reflective decoding (Ji et al.) 20236), introduce Token- Guard, decoding method designed to provide explicit and robust hallucination reduction throughout the LLM reasoning process.
3 PRELIMINARIES
We summarize key concepts and fundamental principles underlying token-level generation, multi- level supervision, and iterative verification processes in modern autoregressive models:
(a) Token-Level Auto- auto-Regressive Generation. A sequence of tokens ......; dr is generated au- toregressively. Formally, the probability of the entire sequence given input defined
P(a oT x)) II Po(a, x.
where T is the sequence length, Oct fai...... a } denotes the tokens generated before step and P3 represents the model aasiined conditional probability a,
(b) Multi-Level Supervision. Token probabilities can be modulated by hierarchical signals:
C2 = Σ wi gi(as x act, C2-1),
(2)
IEC
where is the set of supervision levels, is the control function at level 1, wy is its weight, and Ct-1 is the previous combined score. Ct represents composite guidance score.
(c) Iterative Verification. Generated sequences can be refined iteratively:
(((), ( S(K()) QQ) Q):
S(K 2T
((++1)
(3)
S(K [Q) <T
where ((()) is the sequence at iteration 4, Q the input query. S( 1Q) is a score measuring correctness hallucination risk, is the threshold, and R(-) is refinement function applied to low-scoring sequences to improve accuracy and reduce potential hallucinations.
METHODOLOGY: TOKEN-GUARD
In this section, as illustrated in Figure 3
we introduce Token-Guard, including token-level self- checking initialization, iterative token scoring and local fix, pruning-base candidate selection, global iratation and final response generation to control hallucinations during decoding.
4.1 PROMPT SELF-CHECKING
Before the Token-Guard pipeline, we use a unified prompting strategy combining a general tem- plate with domain-specific constraints. The general prompt enforces reliance on the input and standardized output format ("Answer:[...""), while domain-speeific prompts add restrictions such as "Yes/No/Maybb" in PubMedQQ, financial reporting in FinanceBench, input-dependent answers History, CovidQA, and RagTruth. Full details are provided the Appendix A.I
4.2 TOKEN-LEVEL HALLUCINATION SELF-CHECKING
Llken-Guard establishes controllable generation space Sp and performs token-level self-checking to reduce hallucination risks. Each token a is verified before being propagated, limiting local hallucinations and supporting multi-step reasoning.
Latent Token Environment Initialization. At generation step t, a latent environment S, is con- structed store semantic representations 8, and contextual states hy of accepted tokens a, for t.
3

---

Published as a conference paper at ICLR 2026
Token Self-cheeking
Segment-Level
Global-Level
Latent Space
O
8p--18p11
O
O
1igh
plosent) BAU
licraion
(R)
H
F halu
Frail9))
Foog
User Input
Iteration
Output
Figure 3: Overview of the Token-Guard framework: iterative token-level decoding trajectory with self-checking, hallucination scoring, local fix, and pruning to ensure reliable output.
To address the initialization of the latent environment at = 1, we define the mean hidden state of the input context tokens an initial anchor:
h
1
Σ 2=1
(LL(L-I)
hidden (x:), h<1 == hr-
(4)
where x = (Z1...... 11)) denotes the input context token and LLM(L-1) layer hidden representation produced by the model. This ensures that trajectory coherence for the first candidate token is well-defined and that hallucination scoring is applicable from the start.
hidden denotes the penultimate-
Candidate Tokens and Hidden States. The LLM generates a candidate token set A, {a ,...,,,
at (Ad) }, each with a hidden state 1, defined by:
h (
LLM (at (i) aa
(5)
where is the sequence of previously accepted tokens and denotes the input context.
Hybrid Token-Level Hallucination Scoring. For each candidate token a.") first compute the mean hidden state of previously accepted tokens:h<ı - E/= hy, 1, where hy is the hidden state of token aj. Then, the hybrid token-level hallucination score is then defined as
11 -het. (1 P(a.) (ace;));
haauu (a: (6)
8x) X
(6)
|
where A [0, 1] balances semantic consistency and token probability. The weighted combination yields an interpretable score and supports easy tuning of the comantic-probabilee trade-off. In our experiments, we set À = 0.6.
Candidate Token Selection and Environment Update. Select the next token at from the candidate set A, by
Of = arg max YEAr
1 1( (a, (st) 2 Ttoken
Jotokeen (6)
3:41 = st (h (e))).
Consecutive tokens passing the token-level threshold form candidate segments Cs. preserving se- mantic coherence for segment-level scoring. In our experiments, we Ttoken 0.4.
Memory note: Only a running average her is maintained for scoring; token hidden states are tem- porarily buffered until segment formation, then released, keeping memory O(Lmax Kactive d) independent of total generation length T'
Proposition 1. Token-level self-checking reduces expected hallucination of generated sequences.
Proof. Experimental results provided in Section 5.4 and theoretical proofs in Appendix B.1
4

---

Published as a conference paper at ICLR 2026
4.3 CANDIDATE SEGMENT REPRESENTATION
Let a candidate segment C₁ consist of a sequence of self-checked tokens {. Ot, J. where each token (t, has associated hidden state n, obtained from the token-level self-checking process. We construct a segment-level representation H& as a weighted average of these token hidden states, where the weights w, are derived from the token-level hallucination scores Pttoken (at, (t,)) using softmax function to emphasize more reliable tokens:
exp (ptokenn (a,, st.))
W
H Σ
(8)
exp( halu (at, st,)))
This ensures tokens with higher reliability contribute more to HR. capturing the most trustworthy semantic content. Each segment C2 then evaluated using three complementary aspects. First, the weighted token-level hallucination score aggregates individual token reliabilities:
Ptaale (Cx)=
Σ
wi Etallu (ac, St.
(9)
Second, local consistency measures the smoothness of semantic transitions between adjacent tokens, with abrupt changes potentially indicating hallucinations or logical breaks:
Consistency(C&)== = 1
Σ
1 h ((+11)
(10)
Third, global alignment evaluates how well the segment aligns with the overall input context repre- sentation Hz, computed as the cosine similarity between the segment representation H₃, and Hg:
Aligmment(Ca) = H₂ H
(11)
The segment-level hallucination score is computed by weighted sum of these three components:
J.eg (C*) = aaatu (Cx) + 3 Consistency(CA) + Alignment
(12)
where a,B,7 E 1] represent the relative importance of the three parts, respectively, satisfying + In experiments, we set = 0.5, 3 0.3, y 0.2.
Tang and high are refined, while those below -lo are discarded to keep only reliable, coherent content. low Segments above hhigh are valid and stored or used multi-step reasoning. Segments between Targ
'seg
During local refinement, only the target segment C is updated. The hidden states of downstream segments C CR+2,... are retained to preserve global structure, avoiding full regeneration of the remaining sequence and ensuring efficiency, following the principle of selective local refinement as demonstrated in (Tang al. 2025 Lyu et al. 2025).
For local refinement, the lowest-scoring token aoow is identified within the segment, and a local win- dow w fa:-1. w!! a is formed including its immediate neighbors. This window is used to correct the weakest part of the segment while preserving surrounding context. The LLM generates refined tokens conditioned on the surrounding context 0ci--1 0>i+1 and the segment representation
Hy, producing a refined window w-(l)' that replaces the original window in the segment:
k
w (1)' LLM LLM__eeineeW:!"
(1ci-1, 05141 HR), C% Cs \ W/ UW!""
(13)
After refinement, the segment score Freg (C)) is recomputed. If it reaches -high the segment is accepted: otherwise, iteration continues up to Ninx steps. This process ensures controlled, step- low
wise improvement of segment reliability. We set seg 0.55, seg
high = 0.75, Nmax 3.
Memory note: During segment formation and local refinement, token hidden states within the seg- ment are temporarily buffered. Once the final segment representation H₃ is computed, temporary hidden states released. Downstream segment vectors are retained only as compact representa- tions, bounding memory to O(Lmax + K d), independent of total generation length T.
Proposition 2. Segment-level with local refinement improves quality and reduces hallucinations. Proof. We provide experimental results in Section 5.5 and theoretical proofs in Appendix B.2
5

---

Published as a conference paper at ICLR 2026
4.4 GLOBAL ITERATION AND CORRECTION
In Stage Three, Token-Guard globally re-evaluates candidate outputs from Stage Two by assembling reliable segments into reasoning chains I = {Ci, C2,..., CR ). We do not rely on the original seg- ment order, since exploring multiple branches can improve global consistency and factual accuracy. inspired by |Saaa et al. (2024). Segments are clustered via TF-IDF and KMeans, and candidate chains are generated by selecting the chain closest to each cluster centroid. We set the number of clusters K for large-scale datasets, while for small datasets such as HaluEval we use K to reduce computation. Each segment Ck has a vector representation H3, token-level confidence vector f.. and cnowledeevverifcatiee score Eg comparing generated content with relevant knowl- edge sources. The factual consistency of a chain is computed as:
K
[ff Ex
Fiact (R)
= K
Wy Feeg( (Ck),
Wk Σラー E
(14)
where (Cx) means segment-level hallucination score, and wa combines token confidence and evidence alignment to emphasize longer segments.Logica coherence is measured between adjacent segments using cosine similarity of Hş and HK+1, weighted by contextual factor Ag:
K
H6+1 |H Hk+1ll'
H₂
F Flogic
A simetx(CCC Ck+1),
(15)
K-1
R=
where simetx provides multiplicative contextual adjustment and is defined as:
Simetx(Ca: Ck+1) 1 +
Σ TEk
1 R2
Ek,i
(16)
with Ek, denoting the input token embedding of segment Cg. This factorized formulation com- bines hidden-state continuity through cos( Hos(H,, Hk+1) and semantic alignment between segment texts through simetx, which uses only input embeddings reconstructed from token IDs.
The global score Fgloball (R) combines factual and logical components via a soft minimum:
Finct (R) Fiogie (R)
Fglobal Fisce (It) + Flogie (R) - Fisci (R) (Rt)
(17)
triggering e-generation if Fglobal (R) Tglobal. which set 0.7, and returning "cannot answer"
both Fract and Flogic are below 0.5. To improve stability, segment thresholds are dynamically adjusted by adjustment margin Ar:
high 'sng
+ AT, if Fiact is low and Flogie is high,
ndj
(18)
seeg
low
AT, if Foggie is low and Fiact is high.
'seg
where seg is the adjusted threshold. Each iteration produces candidate chain R( and recalculates f:(() globati the chain is accepted if "global 2 Tglobal before Mmax iterations, otherwise the system outputs "cannot answer." In our experience, we set Tglobal = 0.7, AT = 0.1, Mmax 2
Memory note: Global alignment and scoring operate exclusively compact segment vectors [H&) and token-level confidence vectors fç. No full token hidden sequences are stored, keeping memory usage bounded by O(K- d), independent of total generation length T'
Proposition 3. Global iterative refinement over candidate sequences reduces hallucination.
Proof. We provide experimental results Section 5.6 and theoretical proofs in Appendix B.3
EXPERIMENTS
This section presents the experimental setup, main results, and analysis. We answer the following research questions (RQs): RQ1: Does Token-Guard outperform other methods? RQ2: Does the main component of Token-Guard work, and how is its comparative analysis? RQ3-5: How well does Token-Guard improve the overall generation quality, in terms of factual precision and cor- rectness, relevance to the query, and logical consistency and completeness in multi-step reasoning? RQ6: How does Token-Gure affect time efficiency and memory consumption during decoding? RQ7: How does Token-Guane perform across backbone models of different scales?

---

Published as a conference paper at ICLR 2026
Method
Financelemch RACTrudh
CovidQA
DROP
history
DROP
PubmedQQ
Halueval
Avg
EM
F1
EM
F1
Fi
EM
F1
EM
F1
EM
F1
EM
FL
EM
Fl
Mefo-Lioma-
BaseModel
0.16
16.00
0.00
14.71
0.02
32.33
0.21
44.21
0.30
39.10
0.00
9.55
0.32
42.16
0.15
26.29
Guided Decading
22:21
55.95
34.73
Predictive Decoding
D.09
8.79
0.00
29
0.20
11.09
38.00
24.43
Chain--f-Thoought
0.11
11:01
0.00
20.47
0.08
36.84
0.30
49:26
0.34
49.21
0.00
20.33
0.40
55.32
0.18
34.63
Tree- of oh-ouhhhhh
0.10
14.44
0.00
21.33
0.10
43.30
0.22
47.73
0.24
37 69
0.00
12.38
0.38
56.02
0.15
33.33
Token: Gurrs(Ours))
0.30
30.50
0.02
43.94
0.08
47.64
0.48
68.52
0.44
58.10
29.67
78.54
0.29
$1.03
Qwenn3-0
BaseModel
0.20
26.67
0.00
36.12
0.05
12.57
0.11
65.10
0.39
57.02
0.00
22.45
0.43
59.83
0.22
41.25
Guided Decoding
0.21
23.56
0.00
39.35
0.07
43.19
0.52
66.11
0.43
53 66
0.00
24.76
0.62
69.88
0.26
45.79
Predictive Decoding
0.04
11.25
34,51
31.92
0.06
23.85
0.00
23,41
19.44
0.00
21.29
0.01
23.67
Chain-of Thoughhts
22.77
Tree-off Thouuhh
0.28
34,91
0.00
36.07
0.00
37.3 34
0.49
69. 12
0.34
54.67
0.00
26.
0.39
57.33
0.21
45.12
(aken-Guard((Or)))
0.45
45.37
0.06
45.89
0.09
44.01
0.66
71.83
0.66
67.69
0.00
28.91
0.51
74.15
0.35
53.98
Table 1: Performance comparison across datasets and base models. Each dataset reports EM and F1 scores. Best values in each column are bold.
5.1 EXPERIMENTAL SETUP
Datasets. To evaluate the performance of Token- Guard, we conduct experiments across six stan-
dard HALU datasets (Ravi et al.
2024): FinanceBench (Tslam et al. (023). DROP (Dua et al.)
2 COVID-QA (Möller et 2020). PubMedQA) @Jin al. 2019), HaluEval (Li al. 2023)
and RAGTruth (Wu et al. 2023). 3. More details are in Appendix
Baselines and Backbone LLMs. We compare Token-Guard with Basemodels, Chain- of oh-uhhsshss (Wei et al. 2022). Tree-of-Thought (Yao et al.) (023). Guided Decod-
ing (Xie et al. 2023) and Predictive Decoding (Ma et al.) 2024) at two models
Meta-Llama-3.1 8B- Instruct (Grattafiori et al. (use) the default backbone in all
subsequent experiments) and Owen3-8B (Yang et al. 2025). More details are in Appendix
All the experiments are implemented on NVIDIA A40 GPU(40GB). The softmax temperature is set to 0.3, the sampling temperature is set to 0.4. More details are in Appendix H.3
Evaluation Metrics. We evaluate Token- Guard and the baselines with three metrics: Exact Match (EM), F1, and BLEU. EM checks exact matches, Fl measures token-level overlap, and BLEU eval- uates n-gram quality and fluency (Luo al.) 2025c). These metries together assess factual accuracy, semantic coverage, and generation quality; formal definitions are in Appendix G
Hyperparameter Details. The hyperparameters of Token-Guard follow practical design; their rationale, functional roles, tuning procedure, and resulting configuration are detailed in Appendix H
5.2 MAIN RESULTS (RQ1)
We compare Token-Guard with baselines across different backbone models. As shown in Table Guard achieves the highest average EM/FI across both LLMs:00.2/51I 1.03 on Meta-Llama- 3. 3.1-8B-Ine ----strruct and 0.35/53.98 3.98 on Qwen3-8B, which demonstrats stable overall performance..
The relative advantage of Token-Guard varies across datasets due to its adaptive self-checking. On tasks requiring multi-step reasoning and strict factual consistency (c.g.. DROP _f0,_ token-level and segment-level scoring suppress hallucinations and improve logical correctness, leading to the largest gains. On nowledge-intensiie tasks (e.g., PubMedQA, CovidQA), improvements are smaller, as Token-Guard reduces hallucinations but cannot compensate for missing domain knowledge. No- tably, EM scores on RAGTruth and PubMedQA are near zero because these datasets provide long- form, flexible reference answers, so even factually correct outputs rarely match exactly.
Overall, Token-Guase provides robust, cross-task reliability, with improvements depending the dataset's emphasis on factual verification versus domain knowledge.
5.3 ABLATION STUDY (RQ2)
We evaluate the contributions of Token-Guard's core components: prompt initialization (P), token- level scoring (T), segment-level scoring (S), and global iteration (G). Table 2 shows that removing any component degrades performance. Token-level scoring most affects EM and F1, due to its crucial role in guiding carly-stage token generation, while prompt initialization and segment-level scoring provide stability. Notably, global iteration boosts BLEU, enhancing linguistic fluency.

---

Published as a conference paper at ICLR 2026
Method Variant
DROP history
RAGTruth
Avg.
F1
BLEU
EM
BLEU
EM
F1
BLEU
Token-Guard
w/o Prompt
0.32
55.23
51.48
0.01
32.50
27.92
0.17
43.87
39,70
w/o Token Level Scoring
0.28
47.51
44.88
0.00
27.10
25.05
0.14
37.31
34.97
who Segment- Level Scoring
0.43
60.10
59.55
0.01
33.08
49.65
46.32
w/o Global Iterationn
0.42
63.05
52.37
0.01
41.05
20.14
0.22
52.05
36,26
Table 2: Ablation study of Token-Guard representative datasets.
5.4 ANALYSIS OF TOKEN-GUARD" EFFECT ON FACTUAL PRECISION (RQ3)
As shown in Figure 4 to evaluate Token- Guard''s effect on factual precision, we analyze it from (a)
token-level F1 across four datasets and (b) accuracy of estimated step values.
Ours- 43.9
47.6
58.1
29.7
1.0
60
Token-wise Acc. Acc.
34.1
10.1
1004
Token-wise
0.6
40
36.1
-30
0P
14.6
32.1
28.0
9.7
0.2-
20
RAGTruth
CoviaRA
DROP .el
PubMedgA
Only
Fixed-Size
Beam
Ours
probability
Flltering
search
Token-level Fl
(b) Step Value Accuracy
Figure 4: Comparison of Token-Guare and token-level decoding methods on factual precision.
Token-level Precision Advantage. Token-Guard consistently outperforms other token-level meth- ods, including Fixed-Size Filtering (FSF), Beam Search (BS), and Only Probability (OP) Wang
al. 2025), demonstrating its effectiveness in retaining factual tokens.
Reliability Across Generation Steps. The token-level accuracy is positively correlated with the correctness of the final answer. By ensuring high accuracy each generation step, Token-Guard improves overall task performance, highlighting the importance of its token-level self-correction mechanism. Please refer to Appendix for details.
5.5
ANALYSIS OF TOKEN-GUARD'S IMPACT ON RELEVANCE (RQ4)
As shown in Figure to evaluate Token-Guard's impact on relevance, we analyze it from (a) segment-level Fl across four datasets and (b) BLEU scores across six datasets.
Improved Segment-level Precision. As shown in Figure 5p, Token-Guard substantially outper- forms SAS (Self-Adaptive Sampling) and LSTM (Long Short-Term Term Memory based decoder) (Ali et al. 2024) in segment-level Fl across all datasets, which demonstrates its superior ability to pre- serve relevant segments and suppress irrelevant fragments during decoding.
Robust Relevance Across Benchmarks. As shown in Figure 5b. Laken-Guurd consistently attains the highest BLEU scores across six datasets, e-g., 63.19 on DROP confirming its robustness in maintaining output relevance and fluency under diverse tasks.
DROP_history history and 75. on HaluEval,
1STM
BLEU Score
63.2
SAS
17.9
30.2
28.0
40
$3.00
34.9
LSTM
19.4
32.6
33.1
11.0
10
RAGTruth
CovidQA
DROP
PubMeagA
CovidGA RAGTruth
DROP hotory
DROP af
HuluEva val
PubmedOA
(a) Segment-level Fl
BLEU Scores
Figure 5: Comparison of Token Guard and segment baselines on relevance.
8

---

Published as a conference paper at ICLR 2026
5.6 ANALYSIS OF TOKEN-GUARD 'S ROLE IN LOGICAL CONSISTENCY AND DYNAMIC RESOURCE MANAGEMENT (RQ5)
As shown in Table and Figure 6 we evaluate Token-Guard from the perspective of its ability to improve generation quality, illustrating how its multi-stage filtering process preserves factual correctness and stabilizes reasoning under ambiguous or underspecified contexts.
Token-Guard Performance Across Key Dimensions.
Token-Guard demonstrates clear gains across multiple evaluation dimensions, with notable improvements in Factual Precision, Logical Coherence and Relevance. Its token- and segment-level checks help suppress halluci- nations and select more reliable fragments, while itera- tive refinement maintains logical consistency throughout the reasoning process. The slightly lower score in Con- ciseness reflects its tendency to provide more detailed ex- planations, prioritizing accuracy and completeness over brevity. Please refer to Appendix G for details.
Logical Coherence
Relevance
Factual Precision
Correctness
50.3
55.8
73.3
Completeness
Concisatiss
Bee-sffhasght Ohirro
Racomodel
Preductiv Secading
Case Study Illustration. Table 3 presents an example from DROP _ROP_nff: baseline methods either report "insuf- ficient information" or misreport values, while Token- Guard correctly identifies the 61-yard scoring play, pre- serving both semantic and factual correctness through dynamic local regeneration. Please refer to Appendix A.2 for passage.
Gabded Decading
Takoo dian
Figure 6: Generation Evaluations.
yorte gree
Oolden Ancowars
Merwods
Tree of Thought
Rosemsddl
Cheik of Thoughtts
cressooning atog tuee deitty the minart nethene from
crenooning stapso- idwariy the wnt nerterze from yarcs nathing in the fire hall and scored on dinip run belon
helow leeing
yoirts nodhing in the for aia and scowel on didtyaediin nnn
The conoa waion "Sasp Potrect
ind
anough informaiion ordeer the quosion about hall
cresscntng adtpao The pansage does not prowide
lenving with concussionn Sten Parrett the infervationn
Generation
passaage sure that Gove sushec 148 pande fnd hoot
aco the frg Blep 2: Detumrano arowor. The
yordh rosh ii gin the
mentore that Gore sat franch record with 140 doos maniion
number of yards PB Frant: Gore pooredir the hall Soop Deterrin annoow nenedt passoggo 61 sard juit does snolic stale and nanyy yartk he nooeed that -ihoas oniing shopsh
necteed intermation The
spectly how yarcs he scored tola durnng the
hast chressoning stepse-
passage
camsseer The passaaee does not provide enough
Intomadton.carmaeen
information on determine the sumber of yarta Dore scored in the first h/ll://wweeee
detemnine the number yards Gene scared the first halt-c/unswer
Exssatton
Spore
0.00
0.00
0.00
0.00
8.00
0:00
Methode
Gudded Decading
Predictive Decoding
Token- Mawn-GuarrDDee Dunn))
en foonc foo a Toochiio mecert with yands nathing
lderrly the volovnt in femmatio in pazaage:
the intomaation in the passage: The passage state
thur rtss frank Gore tranches recond with 148
Mentty
creasoning, atepas The ressccing stace and 1-Tha poanage states that RiS trank tore aet munchas record with yortss bafore loving with concuscion on 2 The quesiion osks speo Roally for the number of pareh Gore sconed
The passage doca menien Gone scoring 148 yorts, only
that he scored on eit sand run Howover & docss prow de
Gord a surds in the ligt tull The passage stades that dors
with rousuiiin Exrect the apeeile infomaaoon about
pents
pansage
relecunt datal about
The quention mnko about the rumber of yards Gore
run before loaving with concusioon Extrat the
pards in the first halt Ancwer the quosdor:
al sconed
perfomance The
nakiig
thail hart, his moniooned the first hal the syrd fun Thoretore the arever the yards gained that scoring play. <ascwenn-0 yards-lanswerr
Honwwar based the
the connec
passage sush rig pands, not sooving yords vivoasoiing shepto
canswern1140 148 yunds.rlenwwer
secfressoning stopss
<answers 140 slowerr
Faskation
F1
BLSU
BLEU
F1
BLEU
0.00
0.00
0.03
0.00
65.77
50.00
Table 3: Case study on generation quality under query from DROP nfl, passage is in Appendix.
5.7
ANALYSIS OF TUKEN-SUARD'S TIME AND MEMORY CONSUMPTION (RQ6)
Here we examine Touen-Guard's 's computational efficiency, analyzing time overhead and memory usage across decoding stages. We focus on two aspects: (a) runtime, including per sample latency and throughput, and (b) memory across token level, segment level, and global level stages. These measurements reveal Token-Guard''s resource implications.
Time and Throughput Analysis. As shown in Table Token- exhibits distinct time, to- ken, and throughput patterns across datasets. Its dynamic iteration enables efficient computation on PubMedQA and HaluEval, while allocating more runtime and output tokens for challenging datasets such CovidQA and RAGTruth. Notably, despite higher token budgets in some settings,

---

Published as a conference paper at ICLR 2026
Token-Guus Guard maintains competitive superior throughput, reflecting its ability to balance iterative refinement with efficient decoding.
Method
RAGTruth
CoviQQA
PubmedQA
HalnEval
Time:
Tokes
Tolemsfies
Time
Tokcnn Tokens/ke
Time
Tokess
Tokenslace
Time
Tokens
Tokershkee
Time
Tokenss
Tokessfice
Chain Thaln-o--Thuughte Thoughts
11 859 80.71
9 620 75.14
11 998 $8.45
14543
77.33
9,66
777.08
79,41
Tree-f- Thouhhf
57 3687- 64.49
67. 3423 $1.09
58 4253
-13.3--33.00 73.58
35 2786
79
$4.29 3537. 19 67.11
Prodictive Decodling
107 12058 112.53
138 538-6643... 47.017
79: -4-66116 106.06
58: 5954
102
95.50
6225 61 92.54
Guided Dreading
110- 17234- 155.96
189 13525 71,64
103 18733 -12703
77. 11-8 151.15
119. 19..38
15295.11 140.20
Tolen-Guard
69 18004 262.81
901- 17474 58.10
12. 19.7799- 240.58
54: 5254 97.54
113.20
12112.95
164.76
Table 4: Time, output token consumption, and normalized throughput (tokens/sec).
Memory Consumption Analysis. On the HaluEval dataset, Token-Guard shows a multi-stage
memory profile: token-level states are buffered efficiently (1.2-9.4%) segment-level briefly peaks (21.6-19.6%), and global-level stabilizes around 10.3-19.0% Overall, peak (21.6%) and average ( (16.3%) memory remain competitive with baselines. Table summarizes peak memory, average memory and runtime for Token-Guare and other methods.
Method
Peak Memory (%)
Avg Memory (%)
Runtime (8)
Guided Decoding
21.4
15.0
77.31
Chain-of-Thhn Thouhhtss
21.3
14.5
7.02
Prec-of-Thoughe
21.4
17.8
35.04
Predictive Decoding
4.1
3.7
58.26
Token-Guard
21.6
16.3
53.89
Table 5: Memory and runtime comparison for Token-Guard and baseline methods.
5.8 ANALYSIS OF TOKEN-GUARD'S PORTABILITY ACROSS BACKBONE MODELS (RQ7)
Table 6 reports the performance of Token- Guard and several baseline decoding strate- gies, evaluated both Llama-3.2-3B-Instruus and L.ama-3.2-13B-Instrru backbone mod-
Table 6: Performance on 3B and 13B models.
Method
RAGTruth
DROP history
HalnEval
Fl BLEU
BLBU
Fl BLEU
els. Predictive Decoding (PD) is omitted on the 13B backbone because it triggered out-of- memory (OOM) errors during decoding.
Model
MascMadel
12.11
29.31
42.95
Chain of Thoughts
32,70
25.48
31.60
26.04
20.88
48.44
Tree of Thought
27.05
21.16
39.84
36.14
59.22
55.11
Prodictive Decoding
31.077
57.55
Guided Decoding 30.50
Token-Guard achieves the highest F1 and BLEU on DROP _istory and HaluEval, showing consistent gains on both model scales. These improvements indicate that Token-Guard scales favorably with larger backbone capacity.
Token- Guard(Ours)
68.21
138 Model
ReseMdel
22.47
15.26
46.32
42.74
35
31.40
Chain-of-Thought
32.25
27.67
$3.00
30.06
40:43
44.01
Trae-of-
32.93
23.70
43.97
38.25
02. (2.20)
58.07
Pradictive Deweding
Guaded Decading
20.34
22.01
40.4
36.71
61.42
58.72
Tolen-Ganed(iarr
33.14
30.41
51.17
47.64
72.66
68.81
On RAGTruth, Token-Guard performs lower on the 3B backbone due to the dataset's long-context structure, which is challenging for smaller
models. The 13B model alleviates this issue to some extent, offering more stable decoding. Over- all, Token-Guard remains effective across backbone sizes, with limitations mainly arising from tremely large-context settings. The results confirm that Token-Guard is portable across model scales without requiring additional tuning.
CONCLUSION
In this work, we introduce Token-Guard, a token-level hallucination-controllee decoding frame- work for large language models. By embedding self-checking at each reasoning step and leveraging a three-stage filtering pipeline, Token-Guard detects and suppresses hallucinated tokens, ensuring multi-step reasoning preserves factual and logical accuracy. Experiments on HALU benchmarks show that Token-Guare substantially improves generation quality, achieving higher F1 scores and stronger alignment with ground -ruth responses compared to conventional decoding methods, Over- all, Bokknn-Guand maintains efficient resource usage, demonstrates portability across model seales, offers reliable approach for mitigating hallucinations in large language model generation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work is supported by the National Key Research and Development Program of China un- der Grant 2024YFC3308500, Beijing Municipal Natural Science Foundation under Grant L,510422 National Natural Science Foundation of China under Grant 62406036, China Postdoctoral Science Foundation under Grant 2025M781 457, and also sponsored by the State Key Laboratory of Net- working and Switching Technology under Grant NST20250110.
REFERENCES
Borkan Ahmed AI- Al-Yaychli, Noor F. Mohammed, Mohammed Safar, Farooq Safauldeen Omar, and Mohammed H. Rasheed. Automating rihf-based hallucination tracking. Journal Européen des Systèmes Automatisés, 58(7):1477-1488 2025. doi: 10. 10.18880//esa.580715 URL httpps:// // doi. org/ 82800 / /eee////eeeaaeeeeee jesa. 580715 Received: 27 May 2025;
Revised: 30 June 2025;
Accepted: 22 July 2025; Available online: 31 July 2025.
Rizwan Ali, Jin Xu, Mushahid Hussain Baig, Hafiz Saif Ur Rehman, Muhammad Waqas Aslam, and Kaleem Ullah Qasim. From data to decisions: enhancing financial forecasts with Istm for ai token prices. Journal of Economic Studies, 51(8):1677-1693, 2024.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfiel-Doddss, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204.0588 008622, 2022. URL https: /aps:///ap.... semanticschollar.c..m..
Corpus :
Sehyun Choi, Tianqing Fang, Zhaowei Wang, and Yangqiu Song. Kets: Knowledge-coostrainee tree search decoding with token-level hallucination detection, 2023. URL https://arxiv.
org/ ht///bbb/23111 09044
Paul Francis Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017. URL
https://api. www.eeeeeniss.e.ee semanticscholar
Atgg/oorpusiii 4787508
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models, 2024. URL
org/ mabs/2333 03883
Xuefeng Du, Chaowei Xiao, and Yixuan Li. Haloscope: Harnessing unlabeled Ilm gener- ations for hallucination detection. ArXiv, abs/2409. 17504, 2024. URL https:
semant i cscho ar. www.or///oo.oo.o TD: 272910799
Dheeru Dua, Yizhong Wang. Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2368-2367 2378,, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18153111/N11911111 1211//1991126....
URL https:
//aclanthology: hrw.lanntooooo..or////....
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle
laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics. 9:346-361. 361, 2021. doi: 10.1162/tacl a 00370.
URL URIhttts:: htt.::ttpt///t http///allanacttcc
antho h wratthhoooy..rrgg.oo ogy. .tall-1..2..

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab Al- Badawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, et al. The llama 3 herd of models, 2024. URL
org/ abs 2407 22407.21783
Anish Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In AAAI Conference on Artificial Intelligence, 2023. URL htttps/:://ap semanticscho ar. ott.://..........ooooooo.0000110
Sachin Hiriyanna and Wenbing Zhao. Multi-layered framework for Ilm hallucination mitigation in high-stakes applications: A tutorial. Computers, 14(8), 2025. ISSN 2073-431X. doi: 10.3390/ computers1408D0 4080332. URL. httptt:/t///wwt http:::::/wwwwwpp....mo com http:c/000333333///////////33
Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vid- gen. Financebench: A new benchmark for financial question answering, 2023. URL https:
Ziwei Ji. Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigat- ing hallucination large language models via self-reflection, 2023a. URL httpttt//s//xxi org htg/abs//en 2310. 06271
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating LLM hallucination via self reflection. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1827-1843, Sin- gapore, December 2023b. Association for Computational Linguistics. doi: 10. 10.18353/112022. findins--emnlp.15 1233 URL https::/aclanthol..../ wwt.ooo....oo...oooooo. 2023. findings-
123/
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP) pp. 2567-2577, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10. 10118853/1/D11 1259. URL httph:tt//// /tttps://ac//. htt///aaaannn www.caannolooooooooooooooooooooo hrttchooooyyorrgg 119-1259/
Grant Ledger and Rafael Mancinni. Detecting IIm hallucinations using monte carlo simula- tions token probabilities. TechRxiv, June 2024. URL httpt::/p/////. //doi org/10 36227/ wtw.briiw..182239.... 01118893366611111///33 Cite as: Grant Ledger, Rafael Mancinni. TechRxiv. June 12, 2024.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. ArXiv, abs/2005. 14001, 2020. 218869575
URL https://app semanticscho org.:/ooooooooroooooooooo Corr//orrsss::
Jiarui Li, Ye Yuan, and Zehua Zhang. Enhancing Ilm factual accuracy with rag to counter hal- lucinations: A case study domain-specific queries in private knowledge-bases. ArXiv, 0bs/2403.10444, 2024. URL hhtpss: :ttass://p.... semanticscholatt
268550113
12

---

Published as a conference paper at ICLR 2026
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. HaluEval: A large-scale hallu- cination evaluation benchmark for large language models. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 6449-6464, Singapore, December 2023. Association for Computational Linguis- Utthttss:: htt.thpppp/...aaaaaooooo.oo.... ogg.oggg...g.3.....
tics. doi: 10. 10.18663//1/22233mmma/o/
em. main. 3977
Haoran Luo, Haihong E, Guanting Chen, Qika Lin, Yikai Guo, Fangzhi Xu, Zemin Kuang, Meina Song, Xiaobao Wu, Yifan Zhu, and Luu Anh Tuan. Graph-r1: Towards agentic graphrag framework via end-to-end reinforcement learning,
org. /bs/
2507 21892
Haoran Luo, Haihong E, Guanting Chen, Yandan Zheng, Xiaobao Wu, Yikai Guo, Qika Lin, Yu Feng. Zemin Kuang, Meina Song. Yifan Zhu, and Luu Anh Tuan. Hypergraphrag: Retrieval-augmented generation via watperrrapp-sttuuureee knowledge representation, 2025b.
URL
Haoran Luo, Haihong E. Yikai Guo, Qika Lin, Xiaobao Wu, Xinyu Mu, Wenhao Liu, Mcina Song, Yifan Zhu, and Luu Anh Tuan. Kbqa-ol: Agentic knowledge base question answering with monte carlo tree search, 2025c.
org/ abs 2501 18922
Zhiyi Lyu, Jianguo Huang, Yanchen Deng. Steven Hoi, and Bo An. Let's revise step-by-step. A unified local search framework for code generation with Ilms, 2025. URL htttps://axii
org/abs/2508. 07434
Chang Ma, Haiteng Zhao, Junlei Zhang, Junxian He, and Lingpeng Kong. Non-myopic generation of language models for reasoning and planning. ArXiv, abs/2410.17195. 2024. URL https:
//api stmant icschollrr (tt.:olarrrrr/oooott....D 2735000::227377111
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. 2023. URL fettps://l //aaiiv///riivia//aa abs//33333 17651
Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. COVID-QA: A question an- swering dataset for COVID-19. In Karin Verspoor, Kevin Bretonnel Cohen, Mark Dredze, Emilio Ferrara, Jonathan May, Robert Munro, Cecile Paris, and Byron Wallace (eds.). Proceedings the 1st Workshop on NLP for COVID-19 at ACL 2020, Online, July 2020. Association for Computa- tional Linguistics. StRlhttps:.
anthology. hrg. wtt.oo/////...........oo. wwwcoviiiiiliemmm
18/
Selvan Sunitha Ravi, Bartosz Mielczarek, Anand Kannappan, Douwe Kiela, and Rebecca Qian. Lynx: An open source hallucination evaluation model. 2024. URL
abs/2407 08488
Swamaadep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch- solve-merge improves large language model evaluation and generation, 2024. URL https:
//arxiv.org /aaaxii..rr//bb///111 15123
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces ballucination in conversation. In Marie-Francine Moens, Xuanjing Huang, Lucia Spe- cia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3784-3803, Punta Cana, Dominican Republic, November 2021. Associa- tion for Computational Linguisties. doi: 10.18653/w1/2011 wttp:///w//22....nnngnnnnnnon mndings-emmlp.320.. URL https:
//aclanthology 200/2021.1...ooooo findings emn 3207///
3207|
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feed- back. ArXiv, abx/2009.01325, 2020. URL https:/ /api. tttpapa..aaaaaaaaaa......aaaaaaa semanticscholas org/ CorpusTD:22166511
13

---

Published as a conference paper at ICLR 2026
Zeyu Tang, Zhenhao Chen, Xiangchen Song, Loka Li, Yunlong Deng, Yifan Shen, Guangyi Chen, Peter Spirtes, and Kun Zhang. Reflection-window decoding: Text generation with selective re- finement, 2025.
03678
Shubham Ugare, Rohan Gumaste, Tarun Suresh, Gagandeep Singh, and Sasa Misailovic. Iter- gen: Iterative semantic-aware structured Ilm generation with backtracking. In International Conference on Learning Representations, 2024. URL oxt//oorpuSD::2733333
Haoran Wang, Xiongxiao Xu, S Yu Philip, and Kai Shu. Beyond tokens: survey on decoding methods for large language models and large vision- langunl models. Authorea Preprints, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le. and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35. pp. 24824-24833 Curran Associates, Inc., 2022. URL
htw.ggn.nniss.....eeeeeeeeeeeeee wwwpppppppp..ppppppppppppepe..ee fi fiw.s/ /atpapprrrrileespppee/22222222 fi File/ 2tt.9//60001111444444411001011.3
Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Cheng Niu, Randy Zhong. Juntong Song, and Tong Zhang. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Annual Meeting of the Association for Computational Linguistics, 2023.
URL htttss://app.. Semanticschee
org/
Corpuorpppppcccc ID: 2666:36665555
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 41618-41650. Curran Associates, Inc., 2023. URL httppppprr/ppppee.......rrrrrsss http://ww....oooooooo/oooaeaaaaa
files/paper/2023/ file/B 81 1a
Fangzhi Xu, Hang Yan, Chang Ma, Haiteng Zhao, Jun Liu, Qika Lin, and Zhiyong Wu. Φ- decoding: Adaptive foresight sampling for balanced inference-time exploration and exploita- tion. ArXiv, abs/2503.13288, 2025. URL https: /api. www.nticsccooarr.coooooo
Corpus ID ID: ID:271004404
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: http///axvvvv.. org/ 2b05 09388
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In A. Oh. T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 11809-11822. Curran Associates, Inc., 2023. URL httpss://prcceeeddeeeee/// new. cwtppppa..pppppppppppp.ppppppe.. http://aeeeseeeeeee/e///2
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. ArXiv, 11.22404.11999, 2024. URL hhtpps://aa)
semanticscholar: http:/chhhhhaac..ooo..ooopaaaaaa
Muru Zhang, Ofir Press. William Merrill, Alisa Liu, and Noah A. Smith. How language
model hallucinations can snowball. ArXiv, abs/2305.13534, 2023. URL https: /pp
semant cscho laroollrroooooo wtw.oo//oooppssstt228888880
14

---

Published as a conference paper at ICLR 2026
Tunyu Zhang, Haizhou Shi, Yibin Wang, Hengyi Wang, Xiaoxiao He. Zhuowei Li, Haoxian Chen, Ligong Han, Kai Xu, Huan Zhang, Dimitris N. Metaxas, and Hao Wang. Token-level uncertainty estimation for large language model reasoning. ArXiv, abs/2505.11737, 2025. URL https:
//api. semant cacholaa seacholar.org.. org Cttpchoolr..rr///rrrtttttt::
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. A survey of large language models, 2025. URL https: / //arxvvvv org/orgggaaoo
2303 18223
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A INPUTS USED IN TOKEN-GUARD
A.1 TOKEN-GUARD PROMPT
As shown in Figure g
we detail the dataset-specific prompts used in our experiments.
In PubMedQA, answers must start with "Yes/NNo/Maybe. and provide a concise one-sentence conclusion, preserving key medical terms and conditions. In FinanceBench, outputs must exactly match the ground-truth format (USD amounts, percentages, ratios) and only use provided finan- cial data without intermediate calculations. In History, answers must use only numbers and enti- ties explicitly mentioned in the passage, with insufficient information handled by a fixed rejection statement. In perieval-augmented QA like RagTruth, answers limited to the given passages, including all factual details, or else return standardized refusal.
Combining domain-specifis constraints with fallback general prompt improves answer consistency and reduces hallucination risk across domains.
peoegtue-ggn 1
"Asewor prece hittee paoe parso mnt YeuteeiMayye
Deganmen eonch
"Youth eheech M'Man
72.
caunty
wardd io"
plraace and roud them parepe dr beceect with cravomm
ti, Amapetin tacluk eppiisty ovedit
Recomelaresie""
recommen cnteenaat cup
intemuaion
"Tee aualy runaadh saclya Auarnc queatica arl Maddy tu parsidi
faid
enloula ens yonsiing pommeniury
15, blacs chress
(1ST221HE7" sirk wattern (S4459 biliees, (7547907 pereeeggges "eraaeeie encemmr, sumes "Lannic givce 2712311 incooal
eutiic oo our
drentt avalaeec
"Zaaepic Uoi much (112
darasaab)
Annvo Auweartyuuuuu
1t. pres pessmaativid a quesiaccio"
rxpostad
mander 'abut saervbblliin vila"
alliag
Case paoige arsens taae chuage saadilly
12. inlecssdion Far
boutfiin wokee lil passue
mangleidde sikee wnnce wilh parsige ge par wangh calke bromen shopeii
Clnice
cuppe Aecoue lwa""
Tagrod
girce und cacoian, Follow dew aupeda"
"Aassoo cardial
de de
aiteon (Ciabll mon lanel gives /oo
aall goampt-
Figure 7: Prompt for Token-Guard.
A.2 CASE STUDY PASSAGE
San Francisco kept Detroit in the game with missed opportunitics, then made just enough plays for two-game winning streak and its first road win. RB Frank Gore set a franchise record with 148 yards rushing in the first half and scored on 61- yard run before leaving with concussion. wishing it hud a bigger lead after out gaining Detroit 247-102 yurds und recovering fumble without giving up tumover. The 49ers had chances to go ahead big in the third quarter because Detroit had two tumavers on its first three plays, but they came away with only field goal and 13-point lead. Gore then caught a 7-yard pass, wobbled off the field and didn't return. QB Alex Smith's fumble midway through the third quarter set score that helped the Lions get back in the game, The 49ers drove down the field set up Joe Nedney's 's fourth field goal, key score because made Detroit go u TD instead of kicking tying field goal late the game. S Keith Lewis intercepted QB Jon Kitna's pass the 49ers' with 2 minutes left and San Francisco now 4-5 picked up the one fitst down it nceded to seal the game winning consecutive games since 2003. RB Frank Gore finished with career-high 159 yards rushing and San Francisco's QB Alex Smith was 14 of 20 for 136 yards with fumble. WR Arnaz Battle caught six passes for 55 yards, and converted third-and-4 on the final drive to allow the 49ers to run out the clock. The 49ers' defense again came up huge, allowing only 273 total offensive yards and forcing turnovers. LB Brandon Moore yet again the story, leading the with tackles, sacks and forcing tumovers. With the win, the 49ers advanced to 4-5.
Francisco led 13-3 halftime after scoring on three of
four drives,
the second
Figure 8: Prompt for Tuken-Guard
16

---

Published as a conference paper at ICLR 2026
B THEORETICAL PROOF
B.1 PROOF OF PROPOSITION
Proposition Token-level self-checking reduces expected hallucination of generated sequences.
Proof. Let a, denote the token generated step with model probability pa(a; St), where s, is the current state (including context and previously generated tokens). Each token is associated with hallucination score Fhalu (at) [0, 1], measuring its likelihood of being factually incorrect.
Token-Guard modifies the generation probability to penalize ballucinations:
Pguard (a; sz) (a (a: (s) exp ( ( >/hnauu (at))
Ze
(19)
Ze =
Σ po(a 8.) exp ( 7-atuuaa))) AEV
where y > controls hallucination penalization strength, Ze ensures normalization, and denotes the set of all possible tokens in the vocabulary.
Define the expected hallucination at step
E Eeeaaea [Fhalu]
Σ Peuard Se) Fhalu (a.).
(20)
Information-theoreti view: Define the conditional entropy of the token under the original model and
Ho(At Sc) = Σ
Parar 8z)) log (a sz),
(21)
H Hguard AA Se)
Σ ni
Pguard (ax sx) log Penard (az 8t).
Here, "Facts" denotes the factual information or evidence relevant to the generation task. The con- ditional mutual information between the generated token and factual evidence
[(A Facts s;) = HA, s,) H(A, Facts,
(22)
Since Token-Guart weights high-hallucination tokens, we have
Expcare [Fhatu] Ey [Fnatu]s
(23)
I (A,; Facts s:) I(A,; Facts sz)o.
i.e., expected hallucination decreases and mutual information with factual evidence increases.
Summary: Token-level self-checking both reduces expected hallucination per token and increases factual alignment, providing theoretically grounded basis for subsequent segment-level and trajectory-level selection. This effectively improves the model's factual precision during genera- tion.
B.2 PROOF PROPOSITION
Proposition 2. Segment-level scoring with local refinement improves generation quality and reduces hallucination.
Proof. Let a candidate segment Ct- consist tokens (at,,..., at,, ). each with hallucination score Fhatu (a,, and weight C wi 1). Define the segment-level score:
Feog (Cx)
Σ wi Fnatu (ar,) + 3HH,
(24)
where H2 measures structural or semantic consistency, and B > 0 balances hallucination vs. struc- ture.
17

---

Published as a conference paper at ICLR 2026
Expected segment-level hallucination:
Feel
Σ
w; E, (ala (a:.)) [x]
(25)
When Fseg (Cx) falls below threshold Teeg. a local refinement window W = {,,- a low is extracted, where a is the lowest-scoring token. The refined window is generated by
a,+1
W LLLM_refio([H) C& WW)),
(26)
yielding an updated segment C₁ Ck W
WWY
Information-theoreli view: Let the mutual information between segment and the factual source
be
I I[Ck; Facts)
w.J(ar FFcts) + [H2;; Facts).
(27)
Refinement improves the weakest sub-window, so that
(C Facts)] 2 E[T(Ck; Facts)),
(28)
ensuring that corrected segments more consistent with factual knowledge.
Summary: Segment-level scoring combined with local refinement guarantees that unreliable sub- parts are iteratively improved. This reduces expected hallucination while enhancing semantic coher- ence, factual alignment, and generation relevance, thereby improving the overall generation qual- ity.
B.3 PROOF OF PROPOSITION
Proposition 3. Iterative refinement over candidate sequences reduces hallucination and improves final response reliability by enhancing logical consistency.
Proof. Let { be candidate trajectories, each with segments {C(²) (.f,,...))) Define the
trajectory-level score:
K
T
Frinal (Ti)
Σ
Feeg (C(()) (1
Fbatu (a (),
A A [0,
(29)
Select the optimal trajectory hierarchically via clustering:
Tj = min FTina(TT)), ECC,
(30)
Tinal = arg min Frina(T)))
(31)
Iterative refinement for global consistency. To further enhance logical coherence, we introduce global iterative update:
F(())(T)) a fift) (T:) + (1 - a)
Σ Sim Tj), jft
(32)
N
where Sim((T, Tj) measures logical agreement between trajectories and a (0,1) balances local reliability and global consensus. After convergence, the final trajectory is
Tinal arg min (T))
(33)
which incorporates both hallucination minimization and global logical consistency.
Informationntheeretii view: The mutual information between trajectory and factual knowledge
K
T
I(TE Facts)
Σ
I(C
Facts) +
I(a (2) Facts).
(34)
18

---

Published as a conference paper at ICLR 2026
Iterative refinement maximizes [(;;; Facts) under consensus constraints, ensuring that the selected trajectory carries maximal factual content and consistent logical reasoning.
Global state selection with iterative refinement reduces hallucinations,, enforces logical consistency, and dynamically corrects low-confidence fragments while controlling computa- tional resources.
Summary:
C TOKEN-GUARD ALGORITHM DETAILS
To illustrate the mechanism of Token- Bukknn-Gurr, we present its full workflow in Algorithm | compris- ing three key stages designed to mitigate hallucinations through iterative self-checking and global consistency evaluation.
Algorithm Token-Guard: Multi-Stan Stage Hallucination-Contee Generation
Require: Input x, language model LLM, thresholds (Teoken: Tang Fong Tglobal): ). steps Nmax
Ensure: Final hallucinationncomtrooe answer y
I: // Phase I: Token-Level Generation & Verification
2: Initialize latent environment 81 f- 0, reasoning chain
3: for to to T
4: Sample candidates {a, (i)) and hidden states {A, from LLM
5: Compute score Frookmn FEatu" (a, |ss)) via semantic similarity and probability
U (h(d)) (n(a)) Append of to current segment C₁
6: Select až satisfying Ttoken: 81+11
7: end for
8: // Phase 2: Segment-Level Evaluation and Local Refinement
9: for each completed segment Ca ((e,............................ Or, do
Compute He Σ wilk, and segment score Piin (Ct) 11: if Faalu (Ct) 2 Tues then Add C& to valid pool and append A
10:
12: else if Tsog <. Phatu (Ci) < high then
low
13:
while Fann (C&) < high and r < Nmax do alow + arg min; Eetokeen, W(l + LLLM_reinc context, Hz)
14:
Update Cs t (C&s W (1) ( Recompute Ha and Fiai (C&): e e
16:
end while
17: if F 2 high then Accept Cs; append to R else Discard Cg.
18: else Discard Ca
19: end if
20: end for
21: // Phase 3: Global Re-thinking and Feedback Correction
22: Initialize iteration
23: while < Nmax do
24:
Compute Frace (R) and Logical Coherence Flogte (R) via segment representations (Hu}
Flace Flogis
25:
Compute Global Score F Fgtobsa Frees Flogle Firet Plogto
26:
if Felobal (R) 2 Tglobal then return answer = R
27:
else Adjust Tang <- f--((Faa [(Freet ); Refine chain via Stage logic; i +- i +
28:
end if
29:
end while
30:
return "cannot answer"
Exceeded Nmax global iterations
D COMPREHENSIVE EVALUATION
To evaluate the potential of TOKEN-GUARD across variety of datasets and tasks, conduct ex- periments on seven QA and reasoning benchmarks as well as the open-domain Strategy dataset. the LLaMA-7B model and compare against the De-
For StrategyQA (Geva al.] 2021), coding by Contrasting Layers method DoLa (Chuang et al. 2024), following the accuracy-based evaluation protocol reported in the original DoLa paper.
19

---

Published as a conference paper at ICLR 2026
Method
Accuracy (%)
Base Acc. (%)
Improved
Improvement Rate (%)
DOLA TOKEN-GUARD
64.1
60.1
4.0
6.66
69.4
62.3
7.1
11,40
Table 7: Results on the STRATEGYQA dataset using LLaMA-7B.
From Table 7, we observe that TOKEN-GUARD achieves 69.4% accuracy, outperforming DoLa by 5.3 points and nearly doubling the improvement rate (11.40% vs. 6.66%). This demonstrates that explicit token-level hallucination scoring yields benefits even in open-domain settings where external grounding is limited. The method forms more reliable reasoning trajectories and reduces emrors caused by locally inconsistent token transitions.
D. MULTI-DATASET BENCHMARK RESULTS
To better assess the robustness of Token-Guard. further incorporate several widely used dfinement-basee baselines, includiig SELF-REFINE(Madann et al. (023), Self-RAGAAsl al., 2024). and Self-Reflection(Ii al. 2023ap. covering both retrieval-ugmmentdd and self-
improvement paradigms, These baselines allow comprehensive comparison across different fami- lies of hallucination aallucination-mitiaation strategies.
Method
FinanceBuuch
RAG'Truth
CovidQA
DROP-h
Avg
BaseModel
16.00
14.71
32.33
44,21
39.10
9.55
42.16
28.29
Self-RAG
42.74
32.91
42.48
50.13
45.29
43.21
86.70
51.78
SELF-REFINE
17.09
28.93
38.66
42.64
47.25
19.47
71.75
37.97
Self Reflection
19.37
14.82
33.76
47.48
42.05
12.83
46.21
30.93
Token-Guard (Ours)
30,80
43.94
47.64
68.52
Table 8: Fl results across seven QA datasets.
Across the seven datasets, observe clear differences among the baselines. SELF-REFINE and Self-RAG exhibit notable gains on short-text or low- low-complexity tasks, but their improvements di- minish substantially on long-context, high-compositionality datasets such DROP and CovidQA. Self-RAG in particular is highly dependent on the reliability of its constructed evidence corpus, leading to sharp drops in performance when retrieval becomes unstable. Self-Reflection also shows limited effectiveness, as repeated rounds of regeneration often fail to meaningfully reduce the dis- crepancy between the predicted and gold answers. contrast, Boken-Guard delivers consistent and robust improvements across all datasets without requiring retrieval or external corpus construction, resulting in strong absolute gains even on the most challenging benchmarks.
E DATASET DETAILS
We conduct experiments six widely-used In-Context HALU benchmarks selected from HaluBench (Ravi et al. 2024), covering specialized questions across multiple domains where large models prone hal hallucinate in their generated answers.
FinanceBench (Islam et al.
2023) A dataset of 10,000 financial document QA pairs including
tables and bullet points, designed to mimic real-world questions asked by financial analysts.
DROP (uua et al. 2019): An English reading comprehension benchmark evaluating reasoning ability over passages; we use its history and n f.l subsets for experiments.
CovidQA (Möller et al., 2020): dataset of 2,000 QA pairs annotated by biomedical experts on COVID-19 related scientific articles.
2019): A biomedical QA dataset collected from PubMed abstracts, requir- ing Yes/No/Maybe answers to research questions, with fong-form evidence-basee answers.
PubMedQA (Jin al.
HaluEval (Li et al. 2023y: A hallucination evaluation dataset containing user queries and Chat- GPT responses, with specifcc examples from QA, knowledge mmowledge-grounded dialogue, and sum- marization (we the qa qa__amples amples subset).
20

---

Published as a conference paper at ICLR 2026
RAGTrutth (Wu al. 2023): A corpus with word-level hallucination annotations for LLM- generated text.
F BASELINE DETAILS
Our experiments compare Token-Guard with four representative reasoning-base based baselines:
Auto- uuto-Regressive (CoT))(Wei et al.
2022 Produces chain-of-thoughh reasoning through auto-
regressive language generation, but is prone to hallucinations due to the lack self-correction.
Tree-of- Tre-uhh-Thouhht (ToT) (Yao et 2023): Builds tree structure for given problem, where each node represents reasoning step. We adopt the BFS implementation, which improves exploration and reduces local ballucinations.
2023): Utllizes self-evaluation each step to perform stochastic beam search, effectively filtering low-quality reasoning paths and mitigating token-level halluci- nations.
Guided Decoding (Xie et al.
Predictive Decoding (M et al.
2024): Proposes look-ahead strategy leveraging Model Predic-
tive Control to reweigh LLM distributions, enabling non-myopic language modeling and reducing long-range hallucinations.
G EVALUATION DETAILS
We evaluate the performance of Token-Guard using four widely adopted metries:
(i) Exact Match (EM). EM measures whether the predicted answer exactly matches the ground truth. Let norm(-) denote the normal- ization function:
(iii) BLEU. BLEU evaluates the n-gram preci- sion of the predicted answer with respect the ground truth, with a brevity penalty (BP) to pe- nalize overly short outputs. For n-gram order up to K:
EM Σ (nomm(y)) = norm(y!)) (35)
BLEU BP
log
(37)
(ii) F1 Score. The Fl score measures the token- level overlap between the predicted answer y: and the ground ttrth answer yf using the har- monic mean of precision and recall:
where P.. is the modified n-gram precision, wn is the weight (uniformly 1/K), and BP min ex, [y' Tol
1 N
2 [tokens(w) (x) tokens(yi))
Fl
[okens(y;) tokens(y?)) (36)
(iv) Token-Level Accuracy Across Generation Steps. To evaluate the reliability of Laken-Guard at each generation step, we compute the step-wise correctness:
Acc. = 1(a, a;),
(38)
where is the predicted token and at is the ground-truth token at step i, and 1(-) is the indicator function.
The overall token-level accuracy for sequence of length T is then
Token-level Accuracy
Ace:
(39)
t=1
H PARAMETER OPTIMIZATION DETAILS
H.1 CONFIGURATION OF CORE PROCESS PARAMETERS
As shown Table we summarize the effect different Token-Guard parameters. Increasing and AT generally improves F1 but also increases time per sample (TPS).
Each Token-Gune hyperparameter has a clear functional role: À and (a, 7) control the interac- tion between token-level confidence and segment-level semantic consistency, while Ar, Nmax and
21

---

Published as a conference paper at ICLR 2026
Mmax define the scope and iteration budget for local refinement. A grid search was performed across the full training corpus all benchmark datasets to select a single, robust configuration balancing Fl and TPS.
ID
Ar
(,,8,
N tuax
M mos
F1
TPS
0.40
0.05
(0.3,0.3.0.4)
2
45.32
95.21
2
0.40
0.10
(0.4.0.3.0.3)
3
48.76
142.35
3
0.40
0.15
(0.5.0.25.0.25)
4
50.21
187.64
(0.3.0.3.0.4)
49.12
110.87
0.50
0.10
(0.4,0.1.0.3)
3
3
52.45
156.78
(0.5.0.25.0225)
$5.03
201.95
0.60
0.05
(0.3.0.3.0.4)
2
50.12
122.56
0.60
0.10
(0.5,0.3.0.2)
$1.03
113.29
9
0.60
0.15
(0.50022.00225))
53.87
180.12
10
0.70
0.05
(0.3,0.3.0.44)
2
$1.56
131.45
12 0.70
0.15
(0.5.0.25.0.25))
56.78
198.45
0.80
(0.3.03..0.4)
2
48.9
104.36
14
0.80
0.10
(0.4.03..33)
3 50.67
148.72
15
0.80
0.15
(0.5,0.25.0225)
4 57.34
205.89
Table 9: Grid search results for Token-Guane Highlighted row indicates the selected high- performance, cost-efficient configuration.
Increasing Max and Mmax leads rapid, multiplicative growth in TPS, especially for complex datasets, illustrating the trade-off between accuracy and efficiency. Our chosen configuration (ID 8) with 0.60, AT 0.10, (a, 8, 7) (0.5, 0.3, 0.2), Nmax 3, and Maax achieves F1=51.03 at TTPS=113.29 demonstrating high cost-effectiveness
H.2 THRESHOLD PROPAGATION FROM TOKEN TO SEGMENT AND GLOBAL LEVELS
Token-Guare thresholds are interdependent. We systematically propagate token-level thresholds Tonken segment- and global-level thresholds, ensuring semantic coherence and factual reliability.
Segment-Level Thresholds For candidate segment Cg with n tokens that passed token-level checking, let
Fise token
Σ
Ftoken (at, St.)
(40)
i=1
which denotes the average token-level hallucination score. Segment acceptance thresholds are de- fined as:
high
Earyy
+ k1 (Caeg 0.5).
(41)
see
low
Farg taken
k2(1 - Carg);
(42)
Feg
where k2 E (.2) control the threshold margin. High-threshold segments are accepted im- mediately, while moderate ones can undergo local refinement.
Global Threshold The global threshold is based on segment-level thresholds and expected con- sistency:
Tglobal = max high
Tseg
A1, pexpected]
fact
(43)
where A E [0.05, 0.1] and expected Rs 0.7.
Propagation Algorithm
1. Choose token-level threshold Ttoken
2. Assume segment consistency Oseg 2 0.7.
3. Compute segment average score geeg R3 Ttoken
4. Compute segment-level thresholds:
high
Fang taken
+ ki (Coeg 0.5),
low Teg
Fang
1z
Cseg.)
token
22

---

Published as a conference paper at ICLR 2026
5. Compute global threshold:
Tglobal max high
A1, pexpectedy
6. Optionally, fine-tune , 22 A empirically to balance Fl and TPS.
Tsolem
Low
hhihh
Tglobal
F1
TPS
0.30
0.52
0.70
0.68
42.17
125.34
2
0.30
0.54
0.72
0.70
43.89
132.58
0:40
0.55
0.75
0.70
51.03
113,29
0.40
0.57
0.72
49.03
110.87
0.40
0.58
0.79
0.73
50.21
118.46
7
0.50
0.60
0.80
0.74
52.45
140.12
148.77
9 0.50
0.63
0.84
0.76
$4.09
156.92
Table 10: Illustrative threshold propagation from token to segment and global levels, with FI and TPS. Row 4 highlights the high-performance, cost-efficient configuration.
H.3 SELECTION OF TEMPERATURES
We provide a brief justification for the choice softmax temperature of 0.3 and a sampling tem- perature of 0.4 in Token-Guard.
Token-Guard depends on stable token-level hidden-state trajectories to perform reliable self- verification. High temperatures introduce excessive stochasticity during decoding, destabilizing these trajectories and weakening the consistency signals that Token-Guard relies upon. Lower temperatures, in contrast, smooth local transition dynamics and preserve the fine- fine-grained patterss needed for hallucination detection.
Our choice of 0.4 is also aligned with the findings of Li et al. (2025), who show that low- temperature decoding (around 0.4) yields more stable distributional alignment in self-consistency reasoning, whereas higher temperatures distort the answer distribution and degrade consistency.
To further validate this choice, we conducted a small-scale sensitivity analysis across five tempera- ture pairs. The 0.3-0.4 configuration achieved the best balance between hidden-state stability and output reliability; higher temperatures consistently reduced hallucination drift.
ability to detect early
Softmax Sampling
HaluEval
RAGTruth
0.15 / 0.25
75.63
41.94
0.30 0.40
79.56
43.60
0.50 / 0.60
73.34
43.31
0.80 0.80
67.05
42.30
Table 11: Fl performance under different temperature settings on HaluEval and RAGTruth.
Based on both theoretical considerations and empirical evidence, we therefore adopt 0.3 (softmax) / 0.4 (sampling) as the default temperature setting.
I ERROR CASE STUDIES
This appendix presents five representative failure cases from comtext-dependent QA datasets, il- lustrating why some generations remain incorrect even after token-level self-checking and re- generation. Each case includes the dataset source, question, context, Token-Guard output, ground- truth answer, and a brief analysis of the failure reason.
Case 1 CovidQA
Question: [In addition to oral swabs, which tests detected the presence of 2019-nCOOV virus?] Context: [Detectable 2019-nCoV viral RNA in blood is strong indicator for the further clinical
23

---

Published as a conference paper at ICLR 2026
severity. The study retrospectively analyzed virus presence in pharyngeal (oral) swabs, blood, and anal swabs in patients confirmed with 2019-nCoV. In the blood cohort, 6 of 57 patients had de- tectable RNA, all progressed to severe symptoms. In the anal swab cohort. 11 of 28 patients were positive, with 8 developing severe disease. Some patients had sequential samples across multiple days showing fluctuating viral loads. Patient 1 had detectable viral RNA in pharyngeal swabs on days 5, 7, 8, and 11; blood was weakly positive on day 6; anal swab was high on day 13. Patient 2 had higher viral RNA in anal swabs than blood on the same day. Patients 3-6 also showed varying RNA detection in blood and anal swabs corresponding with progression to severe illness. The study highlighted the correlation of pulmonaul viral RNA presence with disease severity, suggesting monitoring blood and anal samples could aid early prediction. Furthermore, high RNA in anal swabs indicates potential replication the digestive tract. Limitations included missing stool samples and inability to measure bronchoalveolar lavage fluid due to procedural risks. Overall, results confirm virus RNA in pulmonary sites and support early monitoring beyond oral swabs.]
Boaen-Guard Output: [Blood and anal swab samples]
Ground Truth: [The 2109-nCoV RNA was readily detected in the blood (6 of 57 patients) and the anal swabs (11 of 28 patients).]
Analysis: Token-Guard correctly identifies the sample types but omits the numeric details and se- quential measurements. The failure stems from the challenge of summarizing long-form, densely detailed text into a complete answer, illustrating the common failure mode of generating accurate but abbreviated outputs for long, complex contexts.
Case 2 - DROP_RI _nnf
Question:
[How many points did each team have at halftime?]
Context:
[Hoping to rebound from their tough overtime road loss to the Raiders, the Jets went home for Week 8 duel with the Kansas City Chiefs. In the first quarter, New York took flight as QB Brett Favre completed an 18-yard TD pass to RB Leon Washington. In the second quarter, the Chiefs tied the game as QB Tyler Thigpen completed a 19-yard TD pass to TE Tony Gonzalez. The Jets would ground truth with Washington getting a 60-yard TD run. Kansas City closed out the half as Thigpen completed an 1-yyard TD pass to WR Mark Bradley.]
Token-Guard Output: [There isn't explicit information about the points scored by each team at halftime. However, two touchdowns have been scored by each team in the first two quarters: Wash- ington's 's 18-yard TD pass and the 60-yard TD run, and Gonzalez's 's 19-yard TD pass and Bradley's s 11-yard TD pass. Considering that each touchdown worth points, at halftime the scores are Jets 14 and Chiefs 12.]
Ground Truth: [14]
Analysis: The failure arises from the inherent difficulty of producing precise numerical answers when the context requires multi-step reasoning. Token- correctly identifies the touchdowns but does not apply exact scoring rules to match the reference answer, illustrating limitation in handling arithmetic reasoning within long contexts.
Case 3 PubMedQA
Question: [Is ultrasound equal to X-ray in pediatric fracture diagnosis?]
Context: [Ultrasound is currently not established for the diagnosis of fractures. The aim of this study was to compare ultrasound and X-ray beyond their use solely for the identification of frac- tures, i.e., for the detection of fracture type and dislocation for pediatric fracture diagnosis. Limb bones of dead young pigs served as a model for pediatric bones. The fractured bones were examined with ultrasound, X-ray, and CT, which served the gold standard. 162 of 248 bones were fractured. 130 fractures were identified using ultrasound, and 148 using X-ray. There were some advantages of X-ray over ultrasound in the detection of fracture type (80 correct results using X-ray, 66 correct results using ultrasound). Ultrasound, however, was superior to X-ray for dislocation identification (41 correct results using X-ray, 51 correct results using ultrasound). Both findings were not statisti- cally significant after adjustment for multiple testing.]
Token-Guard Output: [No. Ultrasound was not equal to X-ray in detecting fracture type, but was superior to X-ray in identifying dislocations in pediatric fracture diagnosis.]
Ground Truth: [Yes]
Analysis: This failure primarily arises from incomplete or ambiguous evidence in the context. Al- though the numerical differences exist between ultrasound and X-ray, the statistical sinniinnieee indicates comparable effectiveness overall. Token-Guane overemphasizes minor differences in frac-
24

---

Published as a conference paper at ICLR 2026
ture type detection, illustrating how subtle interpretations in biomedical contexts can lead to dis- crepancies even after re- re-generation.
Case 4 HaluEval
Question: [Which genus of moth in the world's seventh-largest country contains only species?] Context: [Indogrammodes is a genus of moths of the Crambidae family. contains only one species, Indogrammodes pectinicornalis, which is found in India. India, officially the Republic of India, is a country in South Asia. It is the seventh-largest country by area, the second-most populous country (with over 1.2 billion people), and the most populous democracy in the world.] Token-Guard Output: [Indgraammodes]
Ground Truth: [Crambidae]
Analysis: This failure is due to ambiguous context. Both the genus Indogrammodes and its family Crambidae are mentioned, allowing multiple plausible answers. Token- Guard outputs the genus, which is factually correct but differs from the reference answer, illustrating errors from ambiguous grounding.
Case DROP _hii_oistory
Question: [What religion was least prominent in 2008?]
Context: [Greater Manchester is home to a diverse population and is a multicultural agglomeration
with ethnic minority population comprising 8.5% of the total population in 2001. In 2008, there were over 66 refugee nationalities in the county. At the 2001 UK census, 74.2% of Greater Manch- esters residents were Christian, 5.0% Muslim, 0.9% Jewish, 0.7% Hindu, 0.2% Buddhist, and 0. 1% Sikh. 11.4% had no religion, 0.2% had an alternative religion and 7.4% did not state their religion. This is similar the rest of the country, although the proportions of Muslims and Jews nearly twice the national average. It contains the Heaton Park Hebrew Congregation, large Ashkenazi Orthodox synagogue in North Manchester. Greater Manchester is covered by the Roman Catholic Dioceses of Roman Catholic Diocese of Salford and Roman Catholic Diocese of Shrewsbury, and the Roman Catholic Archdiocese Liverpool. Most of Greater Manchester part of the Anglican Diocese of Manchester, apart from Metropolitan Borough of Wigan which lies within the Anglican Diocese of Liverpool.]
Token-Guard Output: [Buddhist]
Ground Truth: [Sikh]
Analysis: The model misidentified the least prominent religion due to the subtle difference between 0.2% Buddhist and 0.1% Sikh in the context. This illustrates a failure arising from fine- grained numerical reasoning within the context, where token-level self-checking alone cannot fully resolve minute quantitative distinctions.
I.1 SUMMARY OF FAILURE PATTERNS
Across these five representative cases, common failures arise from incomplete or ambiguous ground- ing in the context, where multiple plausible answers exist or key information is subtle. Some errors also stem from limitations in the model's knowledge, especially for rare or specialized topies.
Additional failures occur due to trade-offs between hallucination suppression and expressive com- pleteness, leading to omitted details numerical information, and from multi-step reasoning or long-form aggregation that exceeds token-level verification. These examples illustrate both the ef- fectiveness of Token-Guard in reducing hallucinations and practical limits of context-dependent QA under current model and dataset constraints.
LIMITATIONS AND FUTURE WORK
While Token-Guare demonstrates strong performance across diverse datasets and backbone models, several limitations remain.
First, the multi-stage decoding with token-level and segment-level scoring introduces non- negligible computational overhead, particularly on longer passages, which may limit scalability on smaller models or latency-sensitive applications.
25

---

Published as a conference paper at ICLR 2026
Second, Token-Guard primarily mitigates hallucinations during decoding but does not supplement missing knowledge; on tasks requiring extensive domain-specific knowledge, factual improvements constrained by the backbone LLM. In particular, when the backbone LLM lacks key knowledge, hallucinations stem from missing information rather than decoding errors.
Third, the clustering and global iteration mechanisms assume semantically coherent segments; highly fragmented or noisy input can reduce effectiveness of segment-level scoring and chain se- lection. Segment scoring remains limited messy or fragmented inputs, clarifying the boundary of Token-Guarde mitigation capabilities.
Finally, Token-Guard eurrently focuses textual knowledge and structured reasoning chains; tending it to molti--oddal inputs remains a promising direction.
26


Reviews:
[{"review_id": "3H8YUFET7U", "reviewer": "Reviewer_eh4g", "summary": "This paper presents Token-Guard, a decoding method to mitigate the hallucination through token level self-checking and iterative refinement. Token-Guard integrates three key components: (1) token-level hallucination scoring using latent-space similarity and probability weighting (2) segment-level scoring combining local consistency and global alignment and (3) global iteration with factual and logical verification across reasoning chains. Token-Guard prunes low-confidence tokens and segments during generation, aiming to prevent error propagation. Experiments across six benchmarks demonstrate improved factuality and consistency over four decoding baselines.", "questions": "1. The method selects tokens that are similar to the previously accepted tokens. Could this reduce generation diversity or lead to over-constrained outputs?\n2. Line 323, the author set the softmax temperature to 0.3, and the sampling temperature to 0.4. Is there any specific reason to set such low temperatures?\n3. Table 3 lists Time and output token comparison but did not provide a normalized comparison(e.g., token/sec). Providing such results would be more straightforward.\n4. Some tables and figures are far from their textual references. Repositioning them near relevant sections would improve readability.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper defines a well-structured three-stage pipeline, incorporating token-level, segment-level, and global iteration.\n2. The experiment spans six benchmarks, demonstrating remarkable performance compared with other baselines.\n3. The method solves a meaningful task. It detects and mitigates the hallucination at the token-level, segment-level, and globally, preventing hallucination propagation.\n4. The case study in Table 4 explicitly demonstrates token-wise correction, supporting the claim of local self-repair.", "weaknesses": "1. The paper claims that current decoding methods lack token-level hallucination checking mechanism. But layer contrasting method (e.g. DoLa, Contrastive decoding) already mitigate hallucinations at the token level and achieve strong results. Relevant baselines are not discussed or compared.\n2. In the latent token environment initialization, the method mentioned it requires initializing the accepted tokens a_j, but no details are provided.\n3. Some datasets (e.g., RAGTruth, PubMedQA) show near-zero EM but high F1 (Table 1), suggesting mismatched evaluation or tokenization inconsistencies, but no explanation was found.\n4. The experiment was conducted on limited LLM architectures (llama3.1 and Qwen 3), and model scale(3B and 8B).\n5. The paper could benefit from providing a thorough examination of failure modes, especially those failed cases after re-generation.\n\nTypo: Ht^{(i)} in line 194 should be H_t^{(i)}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VG9xkekdP2", "reviewer": "Reviewer_ATsM", "summary": "In this paper, the authors study decoding frameworks of Autoregressive LLMs in order to mitigate hallucinations in generated responses. To do so, the paper proposes Token-Guard, a self-checking three-stage scoring and detection pipeline at the level of individual tokens, candidate segments of tokens and finally a global iteration and correction step, in a sequential manner. Furthermore, TokenGuard is seen to achieve consistent improvements in accuracy over standard datasets, when compared to baseline decoding strategies.", "questions": "1) Kindly refer to the questions mentioned in the weaknesses section above. I would be happy to raise my score further if these could be adequately addressed.\n2) Line 189-194: The difference between the hidden state “h_t^(i)” and “H_t^(i)” which denotes the representation vector of candidate token a(i) was not apparent, could the authors kindly clarify this?\n\nMinor Typos:\n\nLine 151: “Full details are provided in the Appendixe A.1”\n\nLine 196: “where Ht^(i) denotes” -> “​​where H_t^(i) denotes” (the “t” should be a subscript to keep notation consistent)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The paper tackles a pertinent research avenue in autoregressive LLMs, namely that of controlled decoding to mitigate hallucinations in generated responses. While the proposed method is a little complex, the paper is well-written and presents each individual component in a fairly clear and lucid manner.\n\n2) TokenGuard is seen to achieve significant improvements in ExactMatch and F1 scores over several decoding methods including Chain-of-Thought and Tree-of-thought, and is demonstrated using standard LLM models such as Llama-3.1-8B-Instruct and Qwen3-8b over diverse datasets for benchmarking.\n\n3) The paper presents a detailed ablative study over several components of TokenGuard, and helps establish the relative importance of the different stages, namely prompt initialization, token level scoring, segment-level scoring, and finally global iteration. \n\n4) Furthermore, while Token-Guard does impose a large computational overhead when comparing with CoT and Tree-of-Thought, the overheads are only marginal when compared to other well-used decoding methods such as Guided Decoding and Predictive Decoding, while simultaneously achieving notable improvements in response accuracy.", "weaknesses": "1) While TokenGuard is seen to be quite effective, the overall method is considerably complex, and involves a large set of hyperparameter choices. This suggests that the methods may not be very practical in several realistic settings. This also raises the question of how these hyperparameters can be set - for instance in Appendix G, some sets are shown, but could the authors clarify if the F1 scores shown are for the final test data, or a hold-out validation set? Furthermore, could the authors clarify if the same exact hyperparameters are used for all evaluation settings over all datasets (and plausibly models as well)? \n\n2) In Stage 2, if a segment that occurs fairly early in the generation process is detected to be modified, would the scores of every later segment necessarily change? For instance, the hidden states of segments that occur after the modified segment would need to be recomputed, since the autoregressive context available is modified as well. Could the authors kindly clarify this specific aspect, since it was not entirely clear, particularly after referencing the overall algorithm presented in Appendix C. \n\n3) While the paper clearly presents the run-time and output token overheads in Table-3, the memory overheads necessitated by TokenGuard are less clear. For instance, the token-level score relies upon the averaged hidden state upto that token, which with a running average still requires a doubling of the hidden-state memory. For the second stage segment score and third stage global iteration, this overhead appears to be potentially even higher since cosine similarities are computed over hidden representations corresponding to these segments. For instance, in Stage 3, the chain closest to each cluster centroid is retained, but the memory overhead that occurs prior to this choice is not very clear at all!\n\n\n4) In Line 259, the paper mentions the use of an external verification score E_k in order to compute the factual consistency score. Could the authors kindly clarify if an external score is actually utilized for the Stage 3 global iteration? This is an extremely pertinent aspect, since the baseline comparisons would be very different once external verification tools can be incorporated. Additional details on the computation of the sim_ctx score which quantifies semantic alignment is also needed. \n\n5) Several scoring methods introduced seem to be very poorly motivated, for instance the convex combination of the cosine similarity of hidden embeddings and the final probability of a given output token such as in Eq 5. This does not seem well principled apriori, and could be motivated further. For instance, more clarity is needed on which layer is chosen, and how this potentially influences the proposed score in practice. Prior works such as INSIDE [1] use similar components like consistency of representations, but are much more principled and well-motivated!\n\n6) For the final stage, with global iteration and corrections, could the authors clarify how many chains are used? Why not just use the original order of the segments?\n\n7) Given the correction and verification steps, comparisons with prior works such as [2], [3], [4], [5] would be highly relevant baselines to compare and contrast with. \n\n\n\n\n[1] INSIDE: Llms’ internal states retain the power of hallucination detection, Chen et al., ICLR-2024\n\n[2] SELF-REFINE: Iterative Refinement with Self-Feedback, Madaan et al., NeurIPS 2023\n\n[3] Towards Mitigating Hallucination in Large Language Models via Self-Reflection, EMNLP 2023\n\n[4] Generating Sequences by Learning to Self-Correct, Welleck et al., ICLR 2023\n\n[5] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, Asai et al., ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GNH82mznct", "reviewer": "Reviewer_35wE", "summary": "This paper proposes Token-Guard, a token-level hallucination control framework for large language models (LLMs). Unlike retrieval-based or fine-tuning methods (e.g., RAG, RLHF), Token-Guard introduces a lightweight self-checking decoding process that dynamically detects, prunes, and corrects hallucinated tokens during generation.", "questions": "1. Add a detailed efficiency comparison using wall-clock time and memory consumption, not only token counts.  \n2. Include experiments showing sensitivity to threshold and weighting parameters to demonstrate stability of the method.  \n3. Expand to tasks beyond hallucination benchmarks, such as reasoning or summarization, to show broader applicability.  \n4. Visualize token-level and segment-level hallucination scores to clarify how the model filters unreliable fragments.  \n5. Explore alternative similarity or scoring metrics for the latent-space hallucination risk.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper proposes a clear and practical framework for controlling hallucinations at the token level using self-checking decoding, segment-level verification, and global iteration. This multi-stage design offers a novel angle on decoding reliability.  \n2. The experiments cover six benchmark datasets and two LLM backbones, demonstrating consistent and measurable improvements in both factual accuracy and fluency.  \n3. The ablation analysis is detailed and informative, helping isolate the contribution of each module and confirming that all components are necessary.  \n4. The latent-space hallucination scoring method and hierarchical structure are intuitively motivated and well integrated into the decoding process.  \n5. The approach is model-agnostic and easily deployable without retraining, increasing its practical value.", "weaknesses": "1. The method increases computation time and output length, sometimes approaching or exceeding heavy decoding frameworks such as Tree-of-Thought. The claim of being lightweight should be more carefully qualified.  \n2. The method relies on many fixed hyperparameters, but the paper provides no sensitivity or stability analysis to show robustness across different values.  \n3. The definition of latent hallucination scoring is intuitive but not empirically verified; it is unclear whether cosine similarity between hidden states best correlates with factual accuracy.  \n4. The benchmarks are mostly QA-style hallucination datasets, so it remains uncertain how well the method generalizes to open-ended or long-form generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XTA2w01kql", "reviewer": "Reviewer_5cxV", "summary": "This paper's Token-Guard method uses self-checking decoding to fix LLM token-level hallucinations. It targets the big problems with current decoding methods: no token-level checks, unclear hallucination risk, and weak dynamic correction. So it built a three-layer control: first check single tokens, then score segment coherence, finally adjust the reasoning chain globally.\nExperiments are solid enough. On HALU datasets (like FinanceBench, DROP), testing with Llama and Qwen models, its average F1 beats CoT, Tree-of-Thought—for example, Llama’s base model only gets 28.29, but Token-Guard hits 51.03. Case studies (like picking the 61-yard score in DROP_nfl) also show it’s more fact-accurate. Unlike RAG or RLHF that need lots of resources, it only tweaks the decoding step—good for scenes with limited resources.", "questions": "see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Interesting to perform fine-grained token-level hallucination control in decoding. Uses latent space scoring to filter bad tokens early, so errors don’t spread.\n2. No need for external retrieval or big human feedback datasets and just need adjusting decoding. Works on different model sizes (like 3B Llama-3.2) too.\n3. The experiments are quite rigorous. Ablation tests (Table 2) prove key parts (token scoring, global iteration) matter; cross-dataset tests show it’s stable.", "weaknesses": "1. Efficiency concerns: Its three-layer processing adds extra work. For long texts or tasks that need fast responses, how fast is it (like tokens per second) compared to light baselines like CoT? Can params like $N_{max}$, $M_{max}$ change automatically to balance accuracy and speed?  \n2. RLHF/RAG comparison: The paper says it’s lighter than RLHF/RAG, but no direct numbers—like how much hallucination each cuts, or how much compute they use per sample. What if we combine it with RLHF? Does that work better?  \n3. Edge cases: If inputs are messy and fragmented, does segment scoring fail? For LLMs that lack key knowledge (like rare medical terms in PubMedQA), hallucinations come from missing info, not bad decoding—can Token-Guard fix that?\n4. Probably this paper is not the \"first\" token-level hallucination detection work, check this out:\n\n[1] Sehyun Choi, Tianqing Fang, Zhaowei Wang, Yangqiu Song, KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection, EMNLP 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)