Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation
Authors: Yasser HAMIDULLAH
Koel Dutta Chowdhury
Yusser Al Ghussin
Shakib Yazdani
Cennet Oguz
Josef van Genabith
Cristina España-Bonet
Venue: ICLR 2026 Poster (2026)
Keywords: sign language translation
multimodal generation
vision-language model
hallucination detection
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce reliability, a token-level measure of visual grounding, to predict hallucination in sign language translation; it generalizes across models and datasets.

Abstract: Hallucination, where models generate fluent text unsupported by visual evidence, remains a major flaw in vision–language models and is particularly critical in sign language translation (SLT). In SLT, meaning depends on precise grounding in video, and gloss-free models are especially vulnerable because they map continuous signer movements directly into natural language without intermediate gloss supervision that serves as alignment. We argue that hallucinations arise when models rely on language priors rather than visual input.
To capture this, we propose a token-level reliability measure that quantifies how much the decoder uses visual information. Our method combines feature-based sensitivity, which measures internal changes when video is masked, with counterfactual signals, which capture probability differences between clean and altered video inputs. These signals are aggregated into a sentence-level reliability score, providing a compact and interpretable measure of visual grounding. 
We evaluate the proposed measure on two SLT benchmarks (PHOENIX-2014T and CSL-Daily) with both gloss-based and gloss-free models. Our results show that reliability predicts hallucination rates, generalizes across datasets and architectures, and decreases under visual degradations. Beyond these quantitative trends, we also find that reliability distinguishes grounded tokens from guessed ones, allowing risk estimation without references; when combined with text-based signals (confidence, perplexity, or entropy), it further improves hallucination risk estimation.  Qualitative analysis highlights why gloss-free models are more susceptible to hallucinations. Taken together, our findings establish reliability as a practical and reusable tool for diagnosing hallucinations in SLT, and lay the groundwork for more robust hallucination detection in multimodal generation.

Published as a conference paper at ICLR 2026
GROUNDING OR GUESSING? VISUAL SIGNALS FOR DETECTING HALLUCINATIONS IN SIGN LANGUAGE TRANSLATION
Yasser Hamidullah Koel Dutta Chowdhury² Yusser Al Ghussin Shakib Yazdani
Cennet Oguz!.2 Josef van Genabith 1.2 Cristina -Speña-
GGerman Research Center for Artificial Intelligence (DFKI GmbH), 2Saarland University, Saarland Informatics Campus, Germany
3 BBarcelona Supercomputing Center (BSC-CNS), Barcelona, Catalonia, Spain
ABSTRACT
Hallucination, where models generate fluent text unsupported by visual evidence, remains major flaw in vision-language models and is particularly critical in sign language translation (SLT). In SLT, meaning depends on precise grounding in video, and gloss-free models are especially vulnerable because they map continuous signer movements directly into natural language without intermediate gloss supervision that serves as alignment. We argue that hallucinations arise when models rely on language priors rather than visual input. To capture this, we propose a token-level reliability measure that quantifies how much the decoder uses visual information. Our method combines feature- based sensitivity, which measures internal changes when vidco is masked, with counterfactual signals, which capture probability dif- ferences between clean and altered video inputs. These signals aggregated into a sentence-lewe reliability score, providing a compact and interpretable measure of visual grounding. We evaluate the proposed measure on SLT benchmarks (PHOENIX-20447) and CSL-Daily) with both gloss-based and gloss-free models. Our results show that reliability predicts hallucination rates, generalizes across datasets and architectures, and decreases under visual degradations, Beyond these quantitative trends, we also find that reliability distinguishes grounded tokens from guessed ones, allowing risk estimation without references; when combined with text-based signals (confidence, perplexity, or entropy), further improves halluci- nation risk estimation. Qualitative analysis highlights why gloss-free models are more susceptible to hallucinations. Taken together, our findings establish reliability practical and reusable tool for diagnosing hallucinations in SLT, and lay the groundwork for more robust hallucination detection in multimodal generation.
INTRODUCTION
Despite being video-to-text task, sign language translation (SLT) differs fundamentally from other video understanding problems. Whereas most multimodal approaches focus on recognizing scenes or describing visual events, SLT must process a complete natural language, namely sign language, with its own vocabulary and grammar. Early approaches relied on glosses. which are explicit human annotations that map video textual sign labels (Camgoz et al. 2018 2020). Glosses provide strong supervision but limit scalability, since most large-scale SLT datasets are only weakly labeled and lack gloss annotations.
Recent advances in SLT increasingly favor gloss-free architectures that rely on large language model backbones (Gong et al. 2024 Chen et al., 2024bb Hwang et al., 2025). While this shift alleviates the costly gloss annotation bottleneck, it also changes the nature of mistakes these systems make. We show that when visual representations are weak whether due to ambiguous signing or low video quality, the language model tends to dominate decoding. This results in translations that may appear fluent yet fail to reflect the signer's actual message, resulting in hallucinations. Such behavior mirrors findings in large vannonne models (LVLLMs), where hallucinations often arise from the model's

---

Published as a conference paper at ICLR 2026
tendency to prioritize language patterns over visual evidence (Ghosh et al. 2025 Parcalabescu & Frank 2025).
In an ideal SLT system, strong visual representations ground the translation, while the language model complements them by ensuring grammaticality and fluency. Hallucinations emerge precisely when this balance is disrupted in favor of the language model.
Hallucination detection has been studied in other multimodal settings, particularly in LVLMs and image/video to text generation, where strong language priors override weaker visual signals (Chen et al. 2025 Min et al. 2025). In these domains, approaches often focus on aligning model attention with Visual content or using perturbation-basse analyses to identify unreliable outputs. However, SLT presents a more challenging scenario because the visual modality is not auxiliary (more than just understanding a scene, detecting objects and actions) but constitutes the source language itself. Consequently, hallucinations in SLT are directly equivalent to translation errors, making the detection and quantification of visual grounding critical. In this work, we define hallucination in SLT as the generation of content tokens that are not supported by the signed video.
crucial first step toward mitigating and reducing hallucinations the ability to characterize and detect them, and to assign reliability scores that allow ranking candidate translations by their risk of hallucination. Text-based signals such as confidence, entropy, and perplexity capture useful linguistic uncertainty. but they miss critical aspect in SLT: whether the output is actually supported by the signed input. We therefore introduce a complementary dimension of hallucination detection we refer to grounding usage which quantifies the extent which generated tokens rely video evidence rather than language priors. By quantifying grounding usage and combining this signal with established text-only baseline measures, from language model priors instead of leveraging the video stream. To this end, we define a reliability score that compares standard decoding with counterfactual runs where the video is masked or replaced. These perturbations yield two classes of evidence: (i) changes in internal states and routing (c.g., hidden-state angular shifts, cross- coossaattnttoon activations), and (ii) probability margins between clean and counterfactual outputs. A linear fusion of the two types produces token-level reliabilities, which are then pooled into sentence-level scores. Our experiments across datasets (PHOENIX-2014
can identify hallucinations arising when models guess
(Camgoz et al. 2018) and CSL-Daily (Zhou et al. 202 1) and architectures (T5 (Hwang al. 2025) and mBART (Chen et al. 2022) ) demonstrate that reliability not only surpasses text-only baselines in predicting ballucinations but also generalizes robustly in cross cross-dataset and cross- architecture transfer. Specifically, it achieves an accuracy of 97% and maintains competitive regression performance when approximating the level of hallucination rate, with Spearman correlation of p 0.72 (between predicted and computed hallucination rates). Our findings further reveal that, compared to gloss-based models, gloss-free models hallucinate more often because they systematically under-utilize visual information, defaulting to linguistic priors rather than grounding in the video. Our contributions and findings can be summarized follows:
To the best of our knowledge, this is the first work to investigate hallucination in SLT.
We introduce a novel measure referred to as reliability based on grounding usage that quantifies how much generated tokens rely visual input, providing direct measure of whether SLT outputs are grounded in the source video.
We design reliability scoring framework using counterfactual perturbations, capturing both internal model changes and output probability shifts.
We empirically demonstrate that reliability surpasses text-only baseline measures in detecting hallucinations and show why gloss-free models are particularly prone them.
2 RELATED WORK
Hallucinations are often characterized as the generation of content that is logically unsound or that departs from the source; in other words, model outputs that assert facts, events, or details not supported, or even contradicted, by the available evidence or ground truth (Ji et al. 2023). In multimodal learning, hallucinations have been widely studied in image and video captioning. particularly with respect to object mentions. For example, Rohrbach et al. (208)) introduced the CHAIR metric to detect and quantify hallucinated objects in MSCOCO image captions, showing that captioning models frequently insert visually ungrounded entities. Similar patterns have been
2

---

Published as a conference paper at ICLR 2026
RELIABILITY COMPUTATION
TOKEN-LEVEL ILLUSTRATION
CONTRAST GF vs GB
Sign Language
Video
Taken types:
Taken level signal distributions
content token (typically signed)
Log-margin
Video
Wrong
function word auciliary verb Sub: subword ending (BPE fregment)
Counteefactual
Vision-Language Moder
Type:
Gold: think the same about you
Hidden sensitivity angle
Hidden Stanes
Cross
Output Probabilkies
Feature based
Atrention
Type:
Aix
Aix
Pred: am hop ing you
same
GF
GB
Festure based
Countertsctual
Ft 64 631
036
6.22 0.5
0.93
Confidence (-log p)
Tediarly
Token-level reliability a(W," h W.
Grounded
Guessed
Hallucinated
GB
Sentence-level reliability
spinad
Ressing - ang of lowest 9%6 rt
Rucio F
GP weok videa senstiutty conrrnnfieent lang arge-driven
Desection Halluciated?
Risk estimation Predict CHAIR
lower nelibil by dity burion conta in mostly quemnd coloens.
GB: strongg video grounding, Wabdence DEDDES ormbubiiyy.
Figure 1: Overview of the reliability framework. Left: Token-level signals are extracted by comparing sensitivity under clean, no-video and wrong video inputs, quantifying each token's 's dependence on visual evidence. Center: Low reliability tokens correspond to weak visual reliance (guessed) exactly where hallucination arises. Right: Signal distributions contrast the language-dominant behavior of gloss-free (GF) models with the visually grounded behavior of gloss-based (GB) models.
observed in video captioning, where generated descriptions may include unsupported objects, actions, Wan 2023).
or temporal details (Ullah & Mohanta 2022 Liu
With the advent of LVLMs, hallucination detection techniques have expanded significantly. Ap- proaches now include reinforcement learning to align outputs with factual grounding (Gunjal et al.. 2024), fact verification methods that leverage external knowledge bases or retrievers (Yin et al. 2024; Sahu et al., 2024)), and tool-augmented strategies that integrate external APIs to assess factual consistency (Chen et al. (2024a). Another noteworthy case is SelfCheckGPT (Manakul et al. 2023) a sampling- approach that detects hallucinations by leveraging response consistency: factual knowledge tends to yield stable outputs across multiple generations, whereas hallucinations produce divergent or contradictory answers.
3 FROM GROUNDING TO GUESSING: RELIABILITY AS A LENS ON VISUAL USAGE
Intuition. At each decoding step, a multimodal model can either ground the next token in the video or guess from language priors or use mixture of both. Our goal a token-level reliability score Tt that increases when the decision genuinely relies on the video signal. Prior work in hallucination detection has mostly relied on text-intrinsic proxies such as entropy or perplexity (Kuhn al. 2023 Farguhar et al. 2024 Obeso et al., 2025). which are agnostic to visual input. While these capture uncertainty, they fail to tell whether a confident prediction is grounded in visual evidence or merely plausible under language priors. In multimodal generation tasks, where visual evidence is the source as in SLT, this distinction is critical (Chen et al. 2022, Guo et al. 2024). To capture visual grounding, propose reliability score. At each step, we (i) measure feature-based sensitivity, which captures how much the output hidden states and the cross-attention drift when the visual input is removed; and (ii) compare decoding with the correct input (which we call clean) to counterfactual runs in which the video is masked or mismatched. The core mechanism is illustrated in Figure []
Intuitively, reature-based signals reflect internal reliance visual context, while counterfactual signals capture external evidence that the video actually helps the prediction. Together, these provide robust, interpretable measure of visual grounding we refer as reliability.
3

---

Published as a conference paper at ICLR 2026
3.1 TEATURE-BASED SIGNALS: INTERNAL RELIANCE
We compare the decoder's hidden states and attention patterns with and without video input.
Hidden state sensitivity. If the decoder hidden state at step changes direction substantially when the encoder input is removed, the token is influenced by the video. We build prior work (Li et al. 2017 2024) by quantifying this directional change as the normalized angle between the hidden states with video (hyid) and with masked input (h.):
π arccos
|
|
where (.-)) denotes the inner product and || || the Euclidean norm. larger shid indicates stronger change in orientation, and thus stronger reliance on the video input. The division by π normalizes the angular distance (in radians) [0, 1] for scale consistency.
Cross-attention usage. Decoder cross-attention A (t.h) layer E [1, L] and head h E [1, H] (from decoder token to encoder position i) aggregated across all layers and heads (L: total decoder layers, H: attention heads). We compute the mean attention mass directed to video encoder positions and subtract the corresponding mass under a null (masked) encoder input, with per-sequence rescaling:
L
H
st scale
Σ
(LM)
Σ
(E.h)
(2)
LH
LH
LH
iEvidoo
h=1 iEmasked
where scale(.) applies quantile scaling to [0, 1] across all decoder tokens in the same output sequence.
A larger sittm indicates stronger reliance on grounded (video) information.
3.2 COUNTERFACTUAL SIGNALS: EXTERNAL EVIDENCE
While Ceature-based signals measure how much the decoder state changes when video is removed, they do not yet capture whether the video actively helps the chosen prediction. To complement the hidden state sensitivity measure, we introduce counterfactual signals, which quantify the probubility advantage of real video compared to wrong or no video.
Setup. We run three parallel decoder passes, all conditioned on the same prefix ot (i) with the true video giving (- c,, (ii) with no video Ø (cross-attention disabled), giving Po(: | (.), and (iii) with a mismatched video z', giving Pmis( Ct, 2').
Let y be the token chosen under the clean pass with We then define the counterfactual distribution as
Pet | ce) max {Po cc), Punis (V:: C4, a.)=
This counterfactual approach related to Chlon et al. (2025), but whereas they consider multiple counterfactuals, focus the single strongest alternative.
Signal definitions. From these passes we compute five complementary measures video contribu- tion (deltas and margins?
log log Poid (v)) - Por((())))
(log-probability margin);
St
s,
logit logit (ve) logit (yr
(scale-stable logit margin);
sp
prob -ss
(normalized probability gain);
A clean = Pvia(U:)= - -o( Amiss = Pvia (v:) Pmis (v.)
(advantages vs. no-video / mismatched);
Pvid (v:), Po(y))
(raw probabilities, optional).
We take the maximum rather than the mean avoid false positives: if either the no-video the mismatched- video condition already explains yt nearly well as the video, then the token should be treated as ungrounded.
Margins capture relative strength across scales, while deltas provide absolute probability gains; using both gives complementary views of video contribution.
4

---

Published as a conference paper at ICLR 2026
Interpretation. All measures increase when the correct video makes the token more likely than either counterfactual The scores glog and glogit capture raw margins on log- and logit-scales, robust to small changes in scale. gDroh maps these into a [0, 1] probability advantage, interpretable "confidence boost." clean and A mis isolate the effect of removing vs. replacing the video, helping diagnose whether errors come from lack of signal misleading signal.
Role in reliability. The counterfactual signal captures whether a token depends on visual input. Reliability is high when (y)) exceeds the corresponding counterfactual probabilities, and low oth- erwise, that is, when the token is more likely under the correct video than under its counterfactual variants. This directly links hallucination risk to the model's under-use of visual evidence.
3.3 TOKEN-LEVEL, AND SENTENCE-LEVEI FUSION OF RELIABILITY SIGNALS
Token-level fusion: For each token, we have two types of reliability signals:
feature-based signals h, = [ hid satta] weighted by Wib.
2. counterfactual signals ge = [s by Wef-
St St prob
logit
cloan Amis Pvia (ye)())), po(s:)], weighted
A token-level reliability score r. is then computed by a linear fusion with sigmoid activation:
o (wi,h,
werg b).
Sentence-level fusion: Since hallucination supervision is only available at the sentence level, we aggregate token reliabilities {re]z= into fixed-size feature vector using early pooling. For each sequence, we compute:
Reail
Σ
rs (average over lowest 9% tokens)
[TT]]
KElowest 9%
where qI' denotes the number of tokens corresponding to the lowest 9% of token reliabilities in the sequence (i.e., [qT] tokens). We primarily rely tail-10 pooling (R)) Other options include Rmuenn (average over tokens), Rharm (harmonic mean), or Rmin (minimum token reliability).
The resulting sentence-level reliabilities Reail-q: Rmeeau.... ... } serve inputs to downstream clas- sifiers or regressors, enabling consistent comparison across utterances of varying length without masking.
3.4 TEXT ONLY BASELINES AND COMBINATION WITH GROUNDING-BASES SIGNALS
To compare reliabilities across datasets and models, we fit monotone mapping from the reliability deficit to the hallucination rate CHAIR
CHAIR ~
(5)
where R. a chosen pooled statistic (c.g.. 10%l, and ISO denotes isotonic regression on a held-out split.
We benchmark reliability against text-only proxies computed from the clean run: confidence C2 (y)) token entropy H, En (w) w log (w). and self-perplexity PPL, = 1// (perplexity under the model's 'ss own predictive distribution), pooled with the same pooling operator (R) Finally, also evaluate combined META variant that concatenates grounding-basee signals with these text-side baselines (confidence, entropy, perplexity) and fits a linear layer on the pooled statistics. This allows us assess whether our visual grounding signal provides complemen- tary information beyond existing text-only measures.
3 Here, counterfactual means either removing the video input replacing with mismatched visual content. 'CHAIR (Caption Hallucination Assessment with Image Relevance) is standard hallucination metric, here adapted to SLT by checking entity overlap hetween generated text and visual content (Rohrbach et al. 2018.. Details found in Appendix F2
5

---

Published as a conference paper at ICLR 2026
Gloss free
PHOENIX
CSL
Gloss
Gloss froe: G 630, pm02.13
Gloss free: im D. 742, pa 706
0.-
00-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Raliaility [min-max 0-1]
Reliability [mins-max D-1]
Figure 2: Correlation between reliability (Rait-10) and hallucination rates (CHAIR) on PHOENIX- 2014T (PHOENIX) and CSL-Daily (CSL). Higher reliability aligns with fewer hallucinations for both gloss-free and gloss- based models.
4 EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL DESIGN
Experiments are conducted on two datasets, PHOENIX-2014T (DGS-DE) (Camgoz et al. 2018
and CSL-Daily (CHL (Zhou et al., (02T), using two competitive and reproducible systems: SpaMo, gloss- free (GF) SLT model (Hwang et al. 2025). and TwoStream-Shi SLT, gloss-basce (GB) model (Chen et al.) 2022).
First, reliability features are extracted following the methodology described in Section 3 detailed algorithm provided in Appendix D
with the
Next, train regression models for hallucination detection and for predicting CHAIR from these features, yielding reliability weights. The hallucination detection model is trained to identify hal- lucinated outputs, leveraging both our grounding-basse based reliability signals (visual features) and the baseline features described in Section 3.4 all integrated within the META (ours) framework. Ac- curacy is computed at 0.5 threshold against the CHAIR-based detection label, where samples with CHAIR > 0 are labeled as I (hallucinated) and others as O. Isotonic regression is employed to predict both CHAIR and its complement (1 - CHAIR). Evaluation metries include AUROC (AUC), AUPRC (AP), and ACC@0. (ACC). The resulting weights are further examined for their transferability across datasets and model families.
Finally, reliability behavior is analyzed under both visual and linguistic degradations. Stress tests include Gaussian noise, temporal downsampling, and random frame dropping on the visual side, as well as wrong-start prompt prefixes to assess language-mode model dominance during decoding. In addition to CHAIR, the analysis tracks how BLEU (Papineni et al. 2002), an n--ram precision metric, and chrF (Popović 2015). character-n-grant F-score sensitive to morphological variation, evolve under these perturbations and how their trends correlate with reliability.
Evaluation includes correlations between reliability and CHAIR (Pearson, Spearman), regression fits (R against 1 CHAIR), calibration robustness under cross dosssedaaasetmoodel transfer, and ablations com- paring feature vs. counterfactual signals and pooling operators (mean tail- 1il.) Implementation details appear in Appendix
4.2 RESULTS
Low visual grounding as hallucination signal. We begin by asking whether our reliability signal is meaningfully aligned with hallucination. Figure shows clear negative correlation between reliability (R) and CHAIR across datasets and model types: as grounding usage (reliability) increases, hallucination decreases. This confirms that our measure captures essential property of

---

Published as a conference paper at ICLR 2026
CSL GB
CSL GF
PHOENIX GB
PHOENIX GF
Detection (AUC/ AP ACCC)
Grounding (ours)
0.803 / 0.991 0.963
0.951 0.998 0.970
0.827 0,954/ 0.899
0.918 / 0.986 / 0.938
META (ours)
0.865 / 0.994 0.753
0.963 / 0.999 0.883
0.899/ 0.978 / 0.864
0.940 / 0.992 0.872
Confidence
0.647 / 0.976 / 0.508
0.953 0.998 0.868
0.618 / 0.887 / 0.644
0.926 0.989 0.860
Perplexity
0.609 / 0.980 / 0.610
0.952 0.998 0.842
0.595 / 0.892 / 0.561
0,888 / 0.979 0.713
Token entropy
0.729 / 0.985 0.665
0.958/ 0.998 0.866
0.639/ / 0.900 / 0.581
0.871 / 0.975 0.731
Regression (Pearson Spearman ISO)
Grounding (ours)
0.465
-0.682 -0.680 0.722
-0.458 / -0.400 0.493
-0.623 / -0,590 0.650
META (ours)
-0,518 -0.515 0.543
-0.736
-0.705 0.755
-0.610 -0.574 0.637
-0.650 / -0.613 0.675
Confidence
-0.253 -0.269 /0.294
-0.685
-0.657 0.714
-0.344 / -0.308 0..30
-0.612 -0.578 0.637
Perplexity
-0.079 -0.082 00.120
-0.594
40.661 0.719
-0.189
/00.229 0.296
-0.400 -0.542/ 0,609
Token entropy
-0.158 1..11/30.143/1 00.195 -0.670 -0.698 0.748
-0.275 -0.273 0.342
-0.468 -0.544 0.625
Table 1: Summary of performance across datasets (CSL-Daily (CSL) and
(PHOENIX)) and models (Gloss-based (GB) and Gloss-free (GF)). Top block reports detection metrics (AUC / AP/ / ACC), bottom block reports regression metries (Pearson Spearman ISO). Best and second best values per metric are bold-faced.
AUC
ACC
CSL
6.50
PHOENE
SL PHOENIKGE
PHOENIS
PHOENIX
PHOENIX
PHOENIX
Grounding (ours)
META (ours)
Best basaline (ContPPp/Entrcop)
Figure 3: Detection metries (AUC, AP, ACC) across datasets and models. We compare Grounding (ours), META (ours), and the best baseline (CoffideneePerppexxxyEEnno from Table ]
hallucination detection, Interestingly, the correlation is consistently stronger in gloss-free models compared to gloss-based models. We attribute this gap the higher hallucination rates in gloss-free free systems: more frequent errors provide richer supervision for the reliability signal, making the negative slope sharper. contrast, gloss-based based systems hallucinate less often, yielding a weaker but still consistent trend.
These results establish the foundation for our subsequent analyses. Grounding usage is not just an auxiliary statistic, it already shows strong correlation with hallucination rates, especially in the more challenging gloss-free setting where visual grounding is the only safeguard against hallucinations. This validates its relevance as hallucination signal and motivates the deeper evaluations that follow, including detection, regression, transfer, and stress tests.
Hallucination detection (binary classification), Table Table upperr block shows that our grounding- based reliability consistently improves over text-only baselines on hallucination detection. Across datasets and models, we gain up to +0.20 AUC and +0.30 ACC compared to entropy and perplexity, reaching near- near-perfect AP (=x 0.99). We present the detailed gain over the other baselines in Table 2 The META (ours) variant, which fuses grounding with text-side uncertainty signals, achieves the strongest overall detection performance, demonstrating that grounding contributes complementary information beyond text probability-basee baselines. At the same time, pure Grounding (ours) is slightly better than any text-only baseline in terms of accuracy, as highlighted in the Figure 3 showing that visual evidence can sharpen decision boundaries in ways text-only signals cannot. This advantage is especially critical in sign language translation, and even more in the gloss-free case,
7

---

Published as a conference paper at ICLR 2026
Detection
CSL
CSL
PHOENIX PHOENIX
Regression (P/S/1)
CSL
CSL
PHOENIX PHOENIX
(A/P/C)
GB
GF
GF
GB
GF
GB
GF
A:+0.155
A:-0.002
A:+0.209
A:-0.008
R:-0.186
R=+0.003
R:-0.114
R:-0.011
Confidence
P:+0.015
P:-0.000
P:+0.067
P:-0.033
Confidence
S:-0.157
S:-0.022
S:-0.092
S:-0.011
C:+0.455
C=+0.102
C:+0.254
C:+0,078
I=+0.171
1:+0.007
1:+0,663
1=+0.113
A=+0, 194
A:-0.001
A=+0.232
A:+0.029
R:-0360
R=-0.888
R:60269
R:-0.223
Perplexity
P:+0.010
P:-0.000
P:+0.062
P:+0.008
Paplexity
S:-0.344
S:-0.019
S:00.771
S:-0.048
C:40.353
C:40.128
C:40,338
C:+0,224
1:+0.346
1:+0.003
1:+
1:+0,041
A:+0.073
A:-0.007
A:+0.189
A:+0.047
R=-0.281
R=-0.012
R:0.118
R:-0.155
Token entropy
P:+0.005
P:-0.000
P:+0.054
P:+0.011
Token entropy
S;-02283
S:40.018
S:-0.127
0:00.04
C:+0.298
C:+0.105
C:+0.318
C:+0,207
1=+0.270
1:00.266
1:+0.151
I:+0.225
(a) Detection: A=AUC, P=AP, C=ACC.
A(Grounding-baselline))
(b) Regression: b(Greunddng--aselinne). R=Pearson, S=Spearman. I=ISO.
Table 2:
Summary of gains and drops over signals. Columns correspond to dataset-model pairs (GB Gloss-based, GF Gloss-free). Blue indicates gain, orange indicates drop.
where intermediate gloss supervision is available and reliable grounding in the continuous video stream is the only safeguard against hallucinations. More broadly, this underscores the need in multimodal generation go beyond text-only uncertainty: hallucination detection benefits from explicitly checking whether outputs are supported by the input modality.
Predicting CHAIR (regression). On the regression task (see Table Dbottom block), where CHAIR is approximated as a continuous target, text-based baselines achieve slightly higher correlations compared using only Grounding (ours) (e.g., token entropy reaches p -0.698 CSL GF, perplexity - = -0.542 on PHOENIX GF), reflecting their strength as direct uncertainty estimators since entropy and perplexity are tied token-level probabilities. Our grounding signal remains competitive, with Spearman correlations up to = -0.680 (CSL GF) and isotonic fits of 0.72-0.65 across datasets. Its counterfactual components exploit probability shifts between clean and perturbed video to provide an -bsed ranking that complements text baselines. The main observation that while grounding alone does not fully match text-only uncertainty measures, it contributes complementary cues: text features capture uncertainty within the language model, while grounding verifies whether predictions are visually supported. Together, as shown by the META (ours) variant (reaching م -0.736 736 on CSL GF and ISO up to 0.76), this synergy yields the most reliable hallucination detection across datasets and transfer settings.
Cross-dataset and cross-model generalization. A linear calibration trained on one dataset transfers to another one with only a small drop in regression performance, and the same holds across gloss- free ++ gloss-based models. Some loss is expected, as the calibration reflects inevitable dataset- and model-specifie biases. Nevertheless, reliability remains the strongest signal across transfers (the full transfer matrix is shown in the Appendix, Fig. 9.. In Fig. 4 Aа) we present cross-dataset transfer within the same model family (GF GB). remains close, as CSL-GF + PHOENIX GF, where the amount of hallucinations in both source and target might be similar and therefore the calibration signal learnable. As presented Fig. 4b), the transfer between gloss- based and gloss-free pronounced degradation in performance, indicating disparities in tokenization
models leads to granularity and a reduction in exposure to linguistically informative samples (as gloss-free models tend to exhibit higher hallucination rates): gloss-based systems operate mostly the word level to maintain gloss alignment, while gloss-fres systems use subword units (Kudo & Richardson 2018). Grounding signals align better with subunits, which clearly indicate which parts of words are guessed independently of video. Subunits level tokens often mark stems, subword boundaries which also correspond to visual units in sign language; glosses mostly represent content words, which are used as objects in CHAIR adaptation. This also suggests that transfer of the calibration to newer gloss-free architectures is more feasible, with smaller loss in terms of post-calibration performance, due to closer alignment with pretrained LLM subword tokenization. In Fig. 4 c), we also report the transfer gaps observed when transferring across both datasets and model types. The results confirm that the performance degradation is larger when reliability weights are transferred from gloss- gloss-based (GB) to gloss-free (GF) models, whereas the opposite direction (GF+GB) exhibits smaller gaps.
8

---

Published as a conference paper at ICLR 2026
GAP model
G4P(-
o9ss dMoset
(4P oles both
600
600
C7
-53
C3:07
-466
03:00
-00
04-46
-010
-42
-033
PHOOXXT
-030
-126
-4
RICOK 58
RICEM 60
Woter
marezzia
03:58
Hother
Protercia
(a) Cross- model
(b) Cross-dataset
(c) Cross-both
Figure 4: Gaps in transfer reliability (negative Spearman correlation, p): (a) Cross-mode within the same dataset, (b) Cross-datased within the same model family, (c) Cross-both (dataset and model
change). Blue= smaller drops, Orange = higher drops.
DESTRIBUTION: BAGEL INT CONFIDENCEE
DISTRIMUTION: GROUNDING.LOR MARGIALC
DISTRIMITION: HEDIENL WWW.HHHEOOONNNOTINNOOOOOOO SENNSIINTTTTAA
CAUGE
CAUST
PNCOMOGG) OZ
C5UG3 CUS
(HOOOEDGG I PHODMEN
CAUSE
GLU
PADKAN
03
Figure 5: Token-level distributions (normalized [0.1]) for confidence logo left), grounding log-
margin (log p( (Vvieeo) - log p(cf), middle), and hidden-state sensitivity angle (right).
Why gloss-free systems hallucinate more often. We compare token-level distributions of (i) grounding log-margin log p(video) - log (ii) hidden-state sensitivity angle, and (iii) confidence (-log p; near 0 means over-confident), as shown in Figure Log-margin: GF is left-shifted on both datasets (most prominently on CSL), while GB maintains longer high- high-margin tail, indicating that under counterfactual video, GF tokens remain plausible. Hidden-state sensitivity angle: GF concentrates at smaller angles (~ 0.20 - 0.35 vs. ~ 0.55 0.70 for GB), indicating weaker state updates when the visual evidence changes. mass sits near zero cover-confidence), whereas GB exhibits a heavier tail. The conjunction of low margin- + small angle + highly overconfident occurs far more frequently in GF (subword tokenization), exactly where the model should lean on the video but does not; GB (word-level) yields more high- margin, rideo-responsive tokens. Consequently. text-only uncertainty reflects the LM's internal belief but misses cases where the video fails to exert inffuence. Our grounding signal evaluates this evidence, and the META (ours) fusion outperforms others by integrating uncertainty with explicit visual support. From this observation, we formalize the claim that gloss-free training increases hallucinations in SLT by increasing the prevalence of weak visual use (see proof in Appendix C
-anfddence-based hallucination signal (-log p): GF
Reliability under visual degradation. To test whether reliability reflects visual usage, we sys- tematically degraded the input in ways: (i) by adding random noise video features, and (ii) by randomly dropping video frames. We then measured how the reliability measure (trained via regression on I-CRIIR) evolves under these perturbations. As shown in Figure 6 reliability consistently decreases as degradation intensity increases, tracking the rise in CHAIR and the decline in BLEU and chrF. The trend is not strictly monotonic, as raw regression outputs can fluctuate at higher degradation levels: this is expected since reliability is optimised to rank hallucination risk rather than to produce calibrated probabilities. Full degradation curves and numerical details are provided in Tables 3 and 4 in Appendix G
Prompt injection and language priors. We further analyze model behavior under adversarially injected start tokens. A comprehensive set of representative samples spanning multiple injected prompts and both decoding settings is presented in Appendix H.2 These qualitative cases reveal

---

Published as a conference paper at ICLR 2026
CSL FEATURE NOISE
PHOENIX FRAME DROPPING
Ralibility - CHAIR - BLEU - BLEU-1 - dvFF
Reliailiyy - CHAIR BLEU - BLEU-1 - dvF2
60
50
Score (%)
40
35
30
30
25
20
15
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
Degradation probability (p)
Degredation probability (p)
Figure 6: Reliability under visual degradation. Left: CSL with added feature noise. Right: PHOENIX with frame dropping. Reliability decreases with stronger degradation, with minor non-monotonic fluctuations, reflecting its role as regression-based ranking signal rather than calibrated probability.
consistent pattern: gloss-free (GF) models frequently continue the injected prefix and generate fluent continuations that follow linguistic priors, while gloss-based (GB) models trained with explicit visual supervision tend to ignore or rapidly override the injected cue and realign with the video evidence.
For instance, with the injected start "ab den' (from the), GF produces "ab den nachmitagsstunden lässt der regen oder schnee an den alpen nach ... (from the afternoon hours the rain snow in the Alps eases ...), faithfully extending the injected phrase but ignoring the signed content. In contrast, GB generates "ab den ein kräftiges tief über der nordsee bringt uns ab morgen früh schneefülle (a strong low over the North Sea brings us snowfall from tomorrow morning ...), which discards the misleading prefix and aligns more closely with the reference.
This contrast substantiates our claim that language priors dominate in gloss-free models. In the absence of intermediate gloss supervision, GF relies heavily on text-only linguistic expectations decoding, making injected prefixes act as strong attractors. contrast, GB benefits from explicit supervision that reinforces cross-modal alignment, resulting in outputs that are more faithful the signed input and less vulnerable to text-only biases.
5 CONCLUSION
Hallucination in multimodal generation cannot be fully explained by text-intrinsic signals such as confidence, entropy, or perplexity. While these signals capture uncertainty, they do not reveal whether outputs are actually grounded in external visual evidence. This limitation is especially critical in SLT, where the visual input is the true source language.
In this work, we propose a grounding usage based reliability measure as a new dimension for hallucination detection. Visual grounding usage quantifies how much each generated token is supported by visual input, and when combined with text-based signals (confidence, perplexity, or entropy), improves hallucination detection by distinguishing visually grounded from visually unsupported tokens. Our experiments show that incorporating grounding usage with established text-based baselines improves predictive performance. Our analysis further reveals a systematic imbalance: gloss-free models trained without explicit visual supervision tend to under-use the visual input, which explains their higher hallucination rates relative to gloss-based models. The latter benefit from intermediate gloss supervision, which enforces stronger visual grounding. In other words, hallucinations in gloss-free SLT largely arise when the model "guesses" based on linguistic priors instead of truly attending to visual evidence.
In summary, good multimodal model should strike the right balance between guessing (via language priors for fluency and grammaticality) and grounding (via visual evidence for factual correctness). Visual grounding alone insufficient, but explicitly disentangling its contribution complements existing text-only signals and provides more robust framework for hallucination detection.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was partially funded by the German ministry for education and research (BMBF) through project BIGEKO (grant number 16SV9093). KDC is supported by the Deutsche Forschungsge- meinschaft (DFG, German Research Foundation)- - SFB I 1102 Information Density and Linguistic Encoding. YG is supported by the German Federal Ministry of Research, Technology and Space (BMFTR) as part of the project TRAILS (01IW24005).
REFERENCES
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7784-77999 7793, 2018.
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transform- ers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10023-10033, 2020.
Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, and Chunhua Shen. PerturboLLaVA: Reducing multimodal hallucinations with perturbative visual training. In The Thirteenth International Conference on Learning Representations, 2025. URL https:://pen:se iew. net / forum?id= J1 JT.SS
Xiang Chen, Chenxi Wang. Yida Xue, Ningyu Zhang. Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Unified hallucination detection for multimodal large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 3235-3252, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.aad acw.o//o............1.o. 178. URL 178/]
http:/aaattttooooooooooooooooooe actp://2222.c.1loomol long
Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. Two-stream network for sign language recognition and translation. Advances in Neural Information Processing Systems, 35:17043-17056, 2022.
Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. Factorized learning assisted with large language model for gloss-free sign language translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 7071-7081, Torino, Italia, May 2024b.
ELRA and ICCL. URL https: http:::///alaaahholgg....ooo
2024 brec-main. 620/
Lcon Chlon, Sarah Rashidi, Zein Khamis, and Marc M. Awada. LLMs are Bayesian, in Expectation, not in Realization, 2025. hRRp::/t::///a// http://axivvvv. www.oor//////aooooooo//..../..ae
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in
large language models using semantic entropy. Nature, 630:625 630, 2024. URL https wtt//oorgssoooroooooooo
7/api. www.ssmtmticccccccc.oooom
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi, Oriol Nieto, Zeyu Jin, and Dinesh Manocha. Visual description grounding reduces hallucinations and boosts reasoning in LVLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL
https: http///ppweeeee.eeeeeeeeeee net 1d
Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. Llms are good sign lan- guage translators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18362-18372. June 2024.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the Thirty-Eighth AAAI Conference Artificial Intelligence and Thirty-Sive Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAT24/AATI Press, 2024. ISBN 978- 57735-887-9, doi: 10. www.///a.......1.....o URL https://doi
AAAI
org org/10. 100.0.660/aaaa... /ttp://aa.....11111.....

---

Published as a conference paper at ICLR 2026
Zhengsheng Guo, Zhiwei He, Wenxiang Jiao, Xing Wang, Rui Wang, Kehai Chen, Zhaopeng Tu, Yong Xu, and Min Zhang. Unsupervised sign language translation and generation. In Lun- Wen Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACI. 2024, pp. Linguistics. doi: 10. www.iings--aa..... URL htthttt:: //t:////c /acl anthology........ .rg/ 2024. www.fiindiggss acl. 831.
Bangkok, Thailand, August 2024. Association for Computational
Yasser Hamidullah, Josef van Genabith, and Cristina España-Bonet. Spatio-temporal sign language representation and translation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 977-982, 2022.
Yasser Hamidullah, Josef van Genabith, and Cristina Ispaña-Bonet, Sign language translation with sentence embedding supervision. In Len Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 425 434, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1220
URL https: ac http://///caaatooooo...o..oo org/
202 4. aaaaacllshoor 407]
Eui Jun Hwang, Sukmin Cho, Junmyeong Lee, and Jong C. Park. An efficient gloss-free sign language translation using spatial configurations and motion dynamics with LLMs. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nutions of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 3901-3920 Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8- 19--8888977---9-888 doi: 10. 10.18653/11/2025 naacl-long. 197. httLt:/s////alant
www.or////2...1111111rree1 loaeccllooog... 1977
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Sury. 55(12), March 2023. ISSN 0360-0300. doi: 10.1 10.1145/3571730 URL
https:://oi.... org.i.orgg/00/ wtt.o////............. 1111111111133
Taku Kudo and John Richardson. SentencePieces A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Com- putational Linguistics. doi: 11.///////DD1-2220
URL
ac http:///oo./a.a..oooo.o.ooo
D18 20122
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Confer- ence on Learning Representations, 2023. URL hhttps://openrete ew net. www.neefffrrrmm forum?id VD-AYEPOdve
Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation 08220
erasure, 2017.
Maximilian Li, Xander Davies, and Max Nadeau. Circuit breaking: Removing model behaviors with targeted ablation, 2024. htt._htttss//axx.v.orr////se
05973
Hui Liu and Xiaojun Wan. Models see hallucinations: Evaluating the factuality in video captioning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11807-11822 18833, Singapore, December 2023. Association for Computational Linguistics. doi: 186.186//vv1 h/20223.mmmmpmmmaii... 723. URL
.rw. wwwpooy..og//......ooo emnlp m00-maaa3 main. 7237
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT:
Zero-resource black-box hal- lucination detection for generative large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Lan- guage Processing, pp. Linguistics. doi: 11188653311/2223 hwt.::23..mmmmmm.......mmm URL https: : http///annooooooooooo.ooo. 2023. emn www.eemmmi..m551177
Singapore, December 2023. Association for Computational
12

---

Published as a conference paper at ICLR 2026
Kyungmin Min, Minbeom Kim, Kang-il Lee, Dongryeol Lee, and Kyomin Jung. Mitigating hal- lucinations in large aision-lanuuage models via summary-guides decoding. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACI, 2025, pp. 3183-4199, A Albuquerque, New Mexico, April 2025. Association for Computa- tional Linguistics. ISBN
195-7. doi: 10. (0.118663/1// 1 www.2222.....2.....oom maadingg-naall.335 URL httpss///ccaanhhoooo..oo//////// findings naacl 1.c.......... 235/
Oscar Obeso, Andy Arditi, Javier Ferrando, Joshua Freeman, Cameron Holmes, and Neel Nanda. Real- time detection of hallucinated entities in long-form generation. arXiv preprint arXiv:2509.0355
2025.
Kishore Papineni, Salim Roukos, Todd Ward. and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th amnual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Letitia Parcalabescu and Anette Frank. Do vision language decoders use images and text equally? how self-consistent are their explanations? In The Thirteenth International Conference on Learning Representations, 2025. URL https:://////m htttps:///ppnreviiiiie
Maja Popović. chrf: character n-gram f-score for automatic mt evaluation. Proceedings of the lenth workshop statistical machine translation, pp. 392-395, 2015.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallu- cination in image captioning. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4035-4045, Brussels, Belgium, October-Novemlle 2018. Association for Computational Lin- guistics. doi: 10.18653/v 1/D11- 1437. URL
Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgoz, and Jean Maillard. Towards privacy- aware sign language translation at scale. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Pupers). pp. 8624 8641, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. www.:///wo/222...lloooooo.oooo 4677
URL https: httpps////llanntoooo...o org/
Pritish Sahu, Karan Sikka, and Ajay Divakaran. Pelican: Correcting hallucination in vision-LLMs via claim decomposition and program of thought verification. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Meth- Miami, Florida, USA, November 2024. ods in Natural Language Processing, pp. 8228-822 8248,
Association for Computational Linguistics. doi: 10. 10.18663/11/224.420 http:///222...mmmaamimmm. emnlp-main.477.. org/ 2024. emn p ma n. 4707]
URL
Nasib Ullah and Partha Pratim Mohanta. Thinking hallucination for video captioning. In Proceedings of the Asian Conference on Computer Vision, pp. 3654-3671, 2022.
David Vilar, Jia Xu, D' H'ro Luis Fernando, and Hermann Ney. Error analysis of statistical machine translation output. In LREC, pp. 697-702, 2006.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui. Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(1)):200105, 2024.
Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back translation.... In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1316-1325, June 2021.
13

---

Published as a conference paper at ICLR 2026
LIMITATIONS
Our study evaluates on two datasets (PHOENIX-2015) and CSL-Daily) using two architectures (SpaMo and TwoStream-SLLI While these cover both gloss-free and gloss-based paradigms across two sign languages, generalization to other sign languages, domains beyond weather forecasts, and newer architectures remains to verified. The reliability framework requires three forward passes per sentence (clean, no-video, mismatched), tripling inference cost relative to standard decoding, which may limit applicability in real-time or resource-constraing settings, Although reliability can be applied without references at inference time, the fusion weights trained using CHAIR labels derived from reference translations. Fully unsupervised calibration remains an open problem. Our adaptation of CHAIR to SLT operates the content-wond level and may not capture all hallucination types, such as temporal ordering errors, spatial confusions between similar signs, or hallucinated function words that alter sentence meaning. Human evaluation of token-level reliability scores would provide stronger validation but was not conducted in this work. Finally, cross-dataset and cross-model transfer, while competitive, shows non-negi non-negligible performance drops, particularly when transferring from gloss-based based to gloss-free models. This suggests that reliability weights may need re-calibration when applied to new architectures or sign languages, which we leave for future work.
A LLM USAGE
We used LLMs solely for language polishing and minor phrasing edits.
B GROUNDING OR GUESSING? HYPOTHESES AND INTUITIONS
SLT has recently shifted from gloss- based pipelines CCaggoz et al. 2018 Chen et al.) 2022) to gloss-free models (Hamidullah et al. [022] Gong et al. 2024 Hamidullah et al. 2024 Hwang et al. 2025). While gloss annotations provide strong immediate supervision, they costly collect and limit scalability Carggoz et al. ) /Zhuu et al. (202T). Gloss-free approaches promise broader applicability. but they also introduce new challenges grounding the translation the visual signal
(uoo et al. 2024: Rust et al. 2024).
Encoder-decoder models face constant tension: tokens may be grounded in the source input or simply guessed from language priors. Function words and inflections can often be guessed predicted from context, but content words, numbers, names, entities require grounding in the encoder signal to avoid error.
Why gloss-free SLT drifts. In gloss-based pipelines, explicit sign annotations provide strong visual anchors that keep the decoder aligned with the video. Gloss-free systems, however, pair video encoders directly with LLM decoders under weak sentence-level supervision. In the absence of dense visual anchors, the decoder defaults to and leans heavily on its language prior, producing fluent but unsupported tokens (i.e., hallucinations). The result is fluent text that is not grounded in the video, i... hallucination.
Towards more hallucinations in SLT. Early SLT systems (Camgoz al.) 2018 2020) mostly failed by omission: missing vocabulary or unseen expressions due to scarce training data. Even so, their outputs remained visually constrained by the gloss signal. Gloss-free models backed by strong LLMs, in contrast, exhibit different failure mode: hallucinations dominate (Hwang al. (2025). Entire entities absent from the video are introduced solely because the linguistic bias of the decoder fills in the gaps. So we currently observe shift in the error profile: hallucinations dominate, where entities absent from the video are introduced by linguistic bias alone.
Reliability as grounding measure. We hypothesize that hallucinations arise precisely when decoding relies on guessing rather than grounding. To test this, we introduce reliability, a token- and sentence-leve measure of how much the decoder relies on the encoder signal of encoder usage. Figure 7 and Figure g show both mechanisms and the aggregate effect: Figure 7 at the token level, counterfactual signals highlight that content words (c.g.. freundlich,, im Nord) depend strongly on video, while function words remain largely language-driven; Figure 8 at the sentence level, lower
14

---

Published as a conference paper at ICLR 2026
loglitty video) luy(aly video))
COUNTERFACTUAL RELIABIUITY (ORANGE) (GRAY) REGIONS
LOU MARGRR (HSA)
Figure 7: Token-level reliability, Content words show strong video dependence, whereas function words rely more on language priors.
Reliability Hallucination Spearman -0.64
gioes-frew median
0.8
Rollabiity
0.6
0.2
0.0
Sentence hallucination rate (CHAIR)
Figure 8: Sentence-level reliability vs. hallucination (CHAIR). Lower reliability correlates with higher hallucination.
reliability correlates with higher hallucination rates. Thus, reliability provides an explicit, quantifiable estimate of grounding confirming reliability as a direct estimate grounding.
C SYMBOLIC PROOF: GLOSS-FREE SUPERVISION RAISES HALLUCINATIONS VIA WEAK VISUAL USE
We formalize the claim that gloss-free training increases hallucinations in SLT by increasing the prevalence of weak visual use.
Random variables. At decoding step t, let H E {0,1] 1] indicate hallucination, W E {0.1) indicate weak visual use (e.g., Tt. less than threshold 7), and GF {0, indicate gloss-free training.
Assumptions.
(A1) Pr(H==1) > 0 (non-triviality)
(A2) Pr(H=1 W=1) [[H=1|W-00 (weakness raises hallucination)
(A3) Pr(W=1 GF=1) > Pr(W=1 GF=0) (gloss-free raises weakness)
(A4) H Il. GF W (mediation through W)
15

---

Published as a conference paper at ICLR 2026
Proposition 1 (Transitivity via mediator). Under (AI) (A4),
Pr(H=1 GF=1) ( GF=0)
Prr(W=1| GF=1) - P P (6)
> 0.
Proof. By the law of total probability and (A4),
Pr(H=1 GF=g)=
Σ
Pr(H=1 W=w) Pr(W=w GF=g).
Taking the difference betweengg-ie and g=0 and using Pr(W=0] GF=g) 1- --rr(W=1 GF=g) yields
AH = (Pr(HI=1 P [Pr(W=1]GF== GF=1) - Pr(W-1 GF-
0
Continuous variant. Let R E [0, 1] be reliability. the mapping into probability m(r)
Pr(H=1 R=r) is non-increasing. R GF=1 is first-order stochastically dominated by R GF=0.
and 11 GF R, then ([)(R) GF=1] 2 E mm((R GF-0) with strict inequality under striet dominance.
D ALGORITHMS
We estimate reliability by comparing the model's predictions with and without visual input. Three passes are used: the true video (video), no visual encoder (no-video), and shuffled or unrelated one (mismatched-vidde) From these, we derive feature-based signals (changes in hidden states and attention) and counterfactual signals (differences in token probabilities and logits). These are fused through a small logistic layer obtain per-token reliabilities тı. Early pooling (mean, tail-q, harmonic, min) yields sentence-level scores R, indicating how much each translation relies on visual evidence.
E EXTRA PLOTS
In Figure present the complete matrix illustrating the cross cross-transse transfer gap datasets, models, and cross-both configurations. Overall, the performance drop remains relatively small; however, reliability weights trained gloss-based (GB) model outputs tend generalize less effectively, likely due to reduced exposure to more informative samples and a higher tendency toward hallucinations.
Figure 10 summarizes the correlation performance of our visual signals relative the strongest base- line for each metric. Similar to the detection results, our methods exhibit a stronger correlation with hallucinations when used as individual signals, and when combined with the text-based (uncertainty) baselines, they further enhance overall performance.
16

---

Published as a conference paper at ICLR 2026
Algorithm 1 RRELIABILTYYROOOIDDEE full computation of sentence-level reliability
Require: SLT model fe, video 2, teacher-forced prefix tokens C1, optional mismatched video x'
Ensure: Sentence-level pooled reliabilities R Rmean, Riai-gg Rharn Rmin and per-token 11.7 Signals and parameters:
Feature-based signals at step t: shid si
Counterfactual signals step t: St
Jog
8t logit ,. Agllan Ari
Fusion weights: Wib E RR, Wa
Wa E R bias beR (default or pre-trained)
Tail fraction qE (0, 1) 000.), numerical constant >>>
Forward passes (feacher-foreed):
(2 hiid.
(21.7.
A
vidd
fo(C1:2, use.encoder true)
(2i.r, h0 A...A 1:T)
f fe(C1:T. use_encoderr = false)
(2 (=1:2 11T) <
rrue)
x' batch-shuffled or external
Pvid f- softmax(=
Po softmax(2"), Pmii softmaxx222xx
for 1ort=11 to 7 do
Feature-based (internal sensitivity):
s <- x arccos (hyid /2) h vid IID)
Arid EH E. (2.h) A +
Al4i (C.h)
St Quant leSca le (AAnd)) QwwntileScaleeeeeeeeoe
Counterfactual (external evidence):
Yt arg max Pvid(w [t), Per ( (Ncc) Pmis (8te)
St
+- + P-llooiiia log Per(Ut); logit (V)) - logit (v)
st ( Actean Prid (o():;; Arth + (via - Pmia (Vt)
Token fusion:
h [shid atn g: +
log
St
logit
prob
Ackean clean
AmisT
ri + o W fb
h4
+ w w.1 +
end for
Early pooling to sentence-level reliabilities:
Rymean
Σ T
2i=1 Tbi Ruiltg Tom) Eielowes Pt
Riurm + Σi 7tt
:
Rmin < min;
return R [mmann Ruil-qu Brams Rmin and F1:T
GAP (-p)
CSL.GF
-0.1
C8
002
-033
GF
-0.4
-0.5
CSLGF
CSL-8
PHOENIX OF
PHOEHIX GB
Figure Full cross transfer gap.
17

---

Published as a conference paper at ICLR 2026
(-p norm.)
Spearman (-p norm.)
CSL
275
6.75
BDD
BLO
PHOENIX
CL PHOENIK Gi
CLL GF PHOENIEG
PHOENIX 3B
PHOENIX GB
PHOENIX 3B
Grounding (ours)
META (ours)
Best basaline (Cont/Pp/Entropy)
Figure 10: Radar visualization of correlation metrics (Pearson, Spearman, and Isotonic (ISO) between CHAIR and reliability scores (R)) normalized to the [0, 1] range.
F IMPLEMENTATION DETAILS
F.1 GENERICS
We compute clean, no-video, and mismatched runs jointly in single batched forward pass with shared kayuv caches, with mismatches generated by in-batch shuffling. Probabilities are accumulated in FP32 with 10 12 for logarithms, while incoder-decodee outputs are kept both raw and after sigmoid squashing. Cross-attention usage is normalized per sequence by mapping the 10/90% quantiles of pooling token-level reliabilities with mean, harmonic mean, tail-10%, exponential moving average (a 0.9), or minimum. Calibration is applied per dataset/model/pool using either linear regression (R aR B3 or isotonic regression against 1 CHAIR. Baselines (confidence, entropy, perplexity) are computed from the clean run; combined variants are also exported.
blocks use AMP/FP16. Regression
[0, 1]. Sentence scores obtained by
F.2 CHAIR COMPUTATION AND LANGUAGE-SPECIIN PREPROCESSING
To evaluate hallucinations we adopt the CHAIR metric (Rohrbach et al. 2018), which measures the proportion of hallucinated content words in the prediction relative to the reference. Since CHAIR relies on accurate identification content tokens, we applied different preprocessing pipelines for German (PHOENIX-2014T data) and Chinese (CSL-Daily data) to account for linguistic differences.
German (alphabetic languages). For German, we rely spaCy for tokenization and part-of- speech (PoS) tagging, keeping only tokens whose PoS belongs to a restricted content set (NOUN, PROPN, VERB, AUX, ADJ, ADV, NUM). Tokens matching stopword list (e.g., und, der. ist) discarded. The remaining tokens are normalized by lemmatization and optionally stemmed using the Snowbal 11 1 stemmer (German/Fnglish), ensuring that inflected forms (e.g., ginge, gegangen) map the same base form. This reduces sparsity and makes CHAIR counts more consistent.
Chinese (logographic language). For Chinese, we cannot rely on whitespace or inflectional morphology. Instead, we: (i) normalize the script with OpenCC (Traditional - Simplified) when needed; (ii) normalize spacing around Chinese punctuation to avoid spurious tokens; (iii) tokenize using jieba; and (iv) apply a Chinese stopword list to remove frequent grammatical particles and function words. Since Chinese lacks inflectional morphology, no stemming is applied.
CHAIR scoring. Given prediction and reference content tokens, we compute per-sentence halluci- nation as:
Instance-level CHAIR (CHAIR,): fraction predicted content tokens that are absent or over-predicted compared to the reference.
18

---

Published as a conference paper at ICLR 2026
The formulation from Rohrbach et al. (2018) is defined as:
# hallucinated object instances
CHAIR;
# all mentioned object instances
where the object here replaced by the content words.
This setup ensures that hallucination evaluation is linguistically adapted: German tokens are normal- ized via siemminggennmatization,, while Chinese tokens are segmented and filtered with script and stopword normalization. We use the instance-level CHAIR as label to train reliability linear weights.
G ABLATIONS
G.1 DEGRADATION UNDER VISUAL PERTURBATIONS
As presented earlier in Fig. G assess whether reliability reflects visual usage, we systematically degraded the video input by adding Gaussian noise and randomly dropping frames. The reliability trained via regression Reliability was rescaled to [0, 1] for visualization, though correlations remain unaffected by this linear transformation.
(1 CHAIR) was then evaluated under these perturbations.
Reliability consistently decreases as degradation increases, confirming its sensitivity to visual infor- mation, though slight mon-monot fluctuations occur because it is optimized to rank hallucination risk rather than to yield calibrated probabilities. CHAIR and anti-BLEU-I exhibit similar trends, with reliability following the same direction but with higher variance, consistent with its role anti-CHAIR ranking signal.
Overall results are presented in Tables 3 and
Reliability
CHAIR
BLEU
BLEU-1
chrF2
0.01
0.37
0.50
23.58
50.92
45.88
0.10
0.44
0.51
23.49
51.87
45.48
0.20
0.43
0.50
22.23
52.87
43.94
0.30
0.34
0.52
19.95
51.83
41.40
0.40
0.39
0.55
16.80
50.04
37.38
Table Degradation on PHOENIX under frame dropping. Values are means over runs. Reliability and CHAIR are reported in raw scale; BLEU, BLEU-1, and chrF2 are on their standard scales.
Reliability
CHAIR
BLEU
BLEU-1
chrF2
0.00
0.41
0.53
17.84
47.26
21.14
0.10
0.36
0.57
14.41
41.72
18.40
0.20
0.38
0.60
12.13
37.75
16.47
0.30
0.39
0.63
9.44
34.13
14.32
0.40
0.36
0.66
7.51
30.92
12.58
Table 4: Degradation on CSL under feature noise.
Values are means over runs. Reliability and CHAIR reported in raw scale; BLEU, BLEU-1, and chrF2 are their standard scales.
G.2 FEATURE IMPORTANCE AND POOLING
Feature weights. To better understand which reliability signals drive hallucination detection, we analyze the learned weights of our logistic regression head (Fig. IT). The model assigns the largest positive weight to the counterfactual log marginn confirming that probability differences between clean and perturbed video inputs the most informative cue. Smaller but consistent contributions are observed for logit-margin, prob-margin, and attention -iiftiinndiffeeecee whereas hidden-state sensitivity (hs prob) receives small negative weight. This suggests that hidden-state changes capture some
19

---

Published as a conference paper at ICLR 2026
FEATURE INFORTANC: (detection weights)
045
662
-142
marphn
plsh margin
mergin
mig
delte dlean
Figure 11: Learned feature weights for hallucination detection. Positive weights indicate stronger reliance of the classifier the corresponding reliability signal.
grounding information, but they are less predictive than counterfactual margins. Delta-based features adelta_clean delta_mis) contribute minimally once margin- signals are included, indicating redundancy.
Pooling strategies. We further investigate the effect of pooling strategies across tokens (Table 5). We compare mean pooling (Rmean)) and papoll-p 10), which focuses on the least reliable 10% tokens.
Across datasets and models, Reail-10 generally improves correlation with hallucination labels, espe- cially for gloss-free (GF) models where weak grounding makes local unreliability highly informative. Gloss-based (GB) models also benefit, though the gap is smaller, indicating that tail pooling is particularly effective when hallucinations arise from localized failures of visual grounding.
Metrics (Bmean B
Dataset
Model
Pearson
Spearman
ISO
CSL
GB
-0.439
-0.234
-0.426
-0.238
0.465
0.282
CSL
GF
-0.672
-0.682
-0.678
-0.644
0.718
0.704
PHOENIX
GB
-0.361
-0.458
-0.368
-0.340
0.453
0.493
PHOENIX
GF
-0.564
-0.623
-0.523
-0.590
0.596
0.650
Table 5: Comparison of pooling strategies. Each cell shows Rmean Rtail-10- yields stronger correlation with hallucinations, especially in gloss-free models.
Tail pooling often
G.3 FEATURE INTERACTIONS WITH RAW PROBABILITIES
REGRESSION WEIGHTS
PROBS
DETECTION WEIGHTS WITH WITHOUT RAW PROBS
6.8
6.00
455
830
Figure 12: Regression (left) and detection (right) feature weights with vs. without raw probabilities.
As shown in Figure 12 adding raw probabilities (v) yields very strong but highly correlated predictor, which reduces the relative weight of complementary cues such as hidden-state sensitiviry. This explains why grounding- reliability remains competitive when used alone: it captures
20

---

Published as a conference paper at ICLR 2026
visual reliance without being dominated by probability mass. In detection, hidden-state sensitivity contributes more strongly when considered in isolation without the raw probability, whereas in re- gression text-related signals dominate, highlighting complementarity rather than redundancy between probability-driven and state-driven evidence.
H QUALITATIVE ERROR ANALYSIS: GLOSS-FREE VS. GLOSS-BASED MODELS
We analyze few overlapping references with outputs from both gloss-free (GF) and gloss-based (GB) models.
H.1 GENERAL DISCUSSION
The qualitative analysis presented in Table outputs that differ lexically or stylistically from the reference may still be semantically correct, while fluent but misleading sentences can conceal omissions, additions, or substitutions that critically alter meaning. Across cases, GF tends to produce translations that stay structurally close to the reference. Its errors are typically subtle, involving small lexical shifts or added modifiers, yet these can still distort key meteorological facts such as temperatures, locations, weather conditions. GB, on the other hand, often generates more fluent and natural -ouundnng outputs. Omissions and hallucinated details are common, occasionally leading forecasts that polished in form yet misleading in content.
H.2 QUALITATIVE: PROMPT INJECTION (EXTENDED)
This section provides an extended qualitative analysis of model behavior under adversarially injected start tokens. includes a comprehensive set of representative samples covering multiple injected prompts and both decoding configurations. The examples complement the discussion in the main text, illustrating the systematic contrast between gloss-free (GF) presented in Table 7 and gloss- based (GB) models presented in Table g GF models often continue the injected prefix, producing linguistically fluent yet visually inconsistent continuations, whereas GB models, aided by explicit visual supervision, tend to suppress the injected bias and quickly realign with the visual evidence.
Column descriptions:
Prompt: the injected adversarial prefix;
Decoding: the generation strategy used (beam greedy);
Fooled: whether the model followed the injected prefix instead of the visual evidence;
Prediction: the model's ' generated translation;
Reference: the corresponding ground-truth sentence.
21

---

Published as a conference paper at ICLR 2026
Reference: am tag breiten sich die teilweise kräftigen schneefälle weiter aus.
GF: zum teil breitet sich kräftiger schnee aus.
GB: dann kommen
teil kräftige schneefälle auf.
Analysis: GF shortens and generalises: "kräftiger Schnee" instead "kräftige Schneefälle" (Incorrect word), and drops "weiter aus" (Omission). GB reformulates with "dann kommen auf", preserving semantics. Semantically correct.
Reference:
und es kann auch örtlich zum teil auch gefährlich glatt werden denn vor allen dingen in der mitte und auch im süden wird es frostig.
GF: sonst gibt auch eher gefährliche straßenglätte vor allen dingen in der mitte und im süden wird es deutlich kühler werden.
GB: und es besteht erhöhte glättegefahr vor allen dingen in der mitte und auch im süden.
Analysis: GF paraphrases with "Straßenglätte" and adds "deutlich kühler werden" (Addition), losing the "örtlich" aspect (Omission). GB compresses but retains core meaning "erhöhte Glättegefahr" + semantically adequate.
Reference: der wind weht schwach his mäßig aus südosten. GF: der wind weht schwach bis mäßig nord bis nordost.
GB: der wind weht schwach bis mäßig.
GF changes wind direction ("Südosten" - "Nord bis Nordost") = Incorrect word. GB omits wind direction entirely Omission. Both distort crucial meteorological info.
Analysis:
Reference: ein ausläufer des tiefs üher dem nordmeer lenkt allerdings von nordwesten wolkenfelder ins land.
GF: richtung westen machen sich tiefaustäufer mit wolken bemerkbar.
GB: ein kleines tief über dem nordwesten lenkt wolken ins land.
Analysis: GF simplifies but misplaces direction ("Richtung Westen" vs. "von Nordwesten") = Incorrect word. GB alters "Ausläufer des Tiefs über dem Nordmeer" + "kleines Tief über dem Nordwesten" (Incorrect word), still partially correct but factual distortion.
Reference: dabci das wetter aber nicht unfrcundlich höchstens mal im osten und auch im süden ein paar schauer.
GF: das milde und trockene wetter wird auch in der osthälfte und im süden ein bisschen schauer geben. GB: am freundlichsten bleibt es noch im osten und süden hier und da schauer.
Analysis: GF exaggerates with "mild und trocken" (Addition), partially shifts tone. GB preserves intended contrast (friendly with occasional showers) - semantically correct.
Reference: am tag dreizehn grad bei dauerregen und einundzwanzig grad am oberrhein.
GF: am tag dreizehn grad an der ostsee und bis zu elf grad in oberfranken.
GB: am tag dreizehu grad im osten und einundzwanzig grad am oberrhein.
Analysis: GF fabricates new regions ("Ostsee"" "Oberfranken") -> Incorrect word + Addition. GB faithful to source with minor rephrasing + semantically correct.
Reference: der wind weht schwach bis mäßig an der nordsee und im bergland auch frischer wind.
GF: der wind weht schwach bis mäßig an der nordsee und auf den bergen teilweise auch frisch. GB: der wind weht schwach bis mäßig an den küsten sowie aus süd bis südost.
Analysis: GF good paraphrase with minor lexical variation ("auf den Bergen" vs. "im Bergland") semantically correct. GB introduces new direction "aus Süd bis Südost" Addition Incorrect word.
Reference: auch im westen gibt es später wolkenlücken.
GF: im osten und westen mal dichtere wolken mal längere sonnenschein.
GB: das hoch das sich von westen his die mitte ausdehnt sorgt morgen verbreitet fiir sonnenschein. Analysis:
GF alters polarity: instead of "Wolkenlüicken" it adds "mal dichtere Walken" Incorrect word. GB hallucinates "Hocchruckgeebee"" and causal explanation Addition, but preserves sunny outcome + partially correct.
Table 6: Comparative qualitative analysis of GF vs. GB translations. Errors are categorized using taxonomy of Vilar et al. (2006): Omission, Addition, and Incorrect word. Correct near-equivalent paraphrases are marked as semantically correct.
22

---

Prompt
Dosoding
Fooled
Prediction
Reference
Published as conference paper at ICLR 2026
am tng
heam
Yes
am tag lässt der regen und schneefall an den alpen nach vor allem im osten und norden fillt noch ctwas regen sonst ist teilweise klar.
regen und schnee lassen an den alpen in der nacht nach im norden und nordosten fallen hier und da schauer sonst ist das klar.
("in the morning / daring the day")
(During the day, rain and snowfall over the Alps subside; mainly in the east and north bit of rain still falls, otherwise is partly clear.)
(Rain and snow subxide over the Alps during the night: in the north and northeast, occasional showers fall, otherwise is clear.)
am
beam
Yes
am tag regnet es an den kusten etwas sonst mal sonne mal wolken vor allem der mitte am freitag wechscllaftes wetter. (During the day, some rain on the coasts, other- wise sun and clouds alternating, especially in the center; on Friday, changeable weather.) am tng fliekt von narden kuhlere luft heran die für teilweise kräftige schneefälle und teilweise ergiebige regenfille sorgt.
am donnerstag, regen in der nordhälitee in der sudhalfte mal senne mal welken Shaliches wetter dann auch am freitag. (On Thursday. rain in the northern half: in the south, sun and clouds alternating, similar weather also on Friday.)
("in the morning / daringg the day")
am tng
heam
Yes
vom nordmeer rieht ein kriffiges tief heran und brings uns ab den morgenstunden heftige schnceftlle teil auch getriereaden degn
("in the morning / daring the day")
(During the day, cooler air flows in from the north, causing heavy snowfall and rain in places.)
moves in from the North Sea, bringing heavy snowfall and some freezing rain from the morning hours.)
(A strong
am
greody
Yes
am tag lässt der regen und schneefall an den alpen nach im osten und norden regnet es noch krättig sonst ist es willeiise klar.
regen und schnce lassen an den alpen in der nacht nach im norden und nordosten fallen hier und da schaucr sonst ist das klar.
("in the morning / during the day")
(During the day, rain and snow over the Alps subside; in the east and north it still rains heav- ily, otherwise partly clear.)
(Rain and snow subside over the Alps during the night; in the north and northeast, occasional showers, otherwise clear.)
23
tag
greody
Yes
tog regnet an den küsten ctwas sonst mal sonne mal wolken und nur vereinzelt ein pear tropten am freitg auch weniger.
donnerstag regen in der nordhälfte in der stichälite mal sonne mal wolken ahaliches wetter dana auch am freitag.
("in the morning / during the day")
(During the day, some rain on the coasts, other- wise sun and clouds with only a few drops; on Friday, less.)
(On Thursday, rain in the northern half: in the south. sun and clouds alternating, similar weather also on Friday.)
am tag
groody
Yes
am tag überquert norddeutschland cin tief mit kräftigem schneelall und welleise kräftigem regen.
vom nerdmeer zicht ein krästiges tief beran und beingt uns ab den morgenstunden heflige schnettälle teil auch gefrierenden regen.
("in the morning / during the day")
(During the day, low crosses northern Ger- many with heavy snowfall and rain in parts.)
strong low moves in from the North Sea, bringing heavy snowfall and some freezing rain from the morning hours.)
ab den
beam
Yes
ab den machnitagastundde lässt der regen oder schnce an den alpen nach osten und norden Falli noch elwas regen sonst ist es teilweise klar.
regen und schnce lassen an den alpen in der nacht nach im nonden und nordosten fallen und da schamer sonst ist das klar.
("from the ")
(From the afternoon hours, rain or snow over the Alps subside; in the east and north, some rain still falls, otherwise partly clear.) ab den donnerstags regnet
(Rain and snow subside over the Alps during the night; in the north and northeast, occasional showers, otherwise clear.)
ab den
beam
Yes
den küsten mal mall wolken varsicht wegen glitte.
donnerstag regen in der nordhältte in der südhäälte mal sonne mal wolken shnliches welter dann auch am freitag, (On Thursday, rain the northern half: the south, sun and clouds alternating, similar weather also on Friday.)
("from the ")
(From Thursday, rain on the coasts, alternating sun and clouds; caution due slipperiness.)
ab den
beam
No
ab den machmittugsstundeen erreicht uns daun ein tief mit teil-
vom nerdmeer zieht ein kräftiges tief beran und bringt uns den murgenstunden heflige schnetfälle zum teil auch gefrierenden regen.
weise krääftgee schneefall und teilweise gefrierenden regen.
("from the ")
(From the afternoon hours, a low reaches us bringing heavy snowfall and partly freezing rain.)
(A strong low moves in from the North Sea, bringing heavy snowfall and some freezing rain from the morning hours.)
Table 7: Nine representative prompt-injection cases for the GF (gloss-free) model with English translations (including prompt meaning) shown in italic below each example.

---

Prompt
Docoding
Fooled
Prediction
Referencs
Published as a conference paper at ICLR 2026
am tng
heam
No
am tag tiefer luftdrick bestimmtt unser wetter (During the day, low pressure determines our weather.)
tiefer luftdruck bestimmt in den nåchsten tagen unser wetter. (Low pressure will determine our weather the coming days.)
("in the morning / during the day")
am
heam
No
um dazu gibe es viele wolken die teilweise krilfige schamer und kräftige schaucr vereinzelt auch gewitter bringen. (During the day, there are many clouds bring- ing partly heavy showers and isolated thunder- storms.) )
das hedentet viele wolken und immer wieder zum teil kriffige schaner und gewitter.
("in the morning / ddrinng the day")
(That means many clouds and repeatedly partly heavy showers and thunderstorms.)
tag
No
am tag schwacher mäßiger an der teilweise ftischer wind aus sid bis südwest im medwestee slurke bis stürmische bieen bei gewittern sturmboen.
meist weht aur schwacher wind aus unterschiedlichen richtungen der bei schauern und gewiltern stark boig sein
("in the morning / during the day")
(During the day, weak to moderate, at the sea partly fresh wind from south to southwest; in the northwest, strong to stormy gusts and storm gusts in thunderstorms.)
(Mostly only weak wind blows from varying directions, which can become very gusty during showers and thunderstorms.)
am tag
"in the morning / during the day") groody
No
am tag tiefer luftdruck bestimmtt unser wetter (During the day, low pressure determines our weather.)
tiefer luftdruck bestimmt in den naschsten tagen unser wetter. (Low pressure will determine our weather in the coming days.)
am tag
gredd
No
am tag dazu gibt es viele wolken die trilweise kräftige schaner gewitter bringen.
das bedeutet viele wolken und immer wieder zum tell kriftige schawet und gewiteer. ('Tat means many clouds and repeatedly partly heavy showers and thunderstorms.)
kräftige schawer vereinzelt (During the day, there are many clouds bring- ing partly heavy showers and isolated thunder- storms.) )
("in the morning / during the day")
24
lag
greedy
No
tag der wind wehl schwach bis mälig der auch frisch aus sidd his slldwest im nordwesten bei gewittern sind stürmische büen möglich.
meist wehl schwacher wind unterschiedlichen richtungen der bei schauemn und gewittern stark boig sein kann.
("in the morning / durring the day")
(During the day, the wind blows weak to moder- ate, at the sea also fresh from south to southwest; in the northwest, stormy gusts possible in thun- derstorms.)
(Mostly only weak wind blows from varying directions, which can become very gusty during showers and thunderstorms.)
ab
ab regen und schneefälle tages nach im nordosten regnet es mitunter noch kräflig sonst zeigen sich auch steme..
die alpen laufe des
regen und schnee lassen alpen nacht nach nonden und nordusten fallen hier und da schawer sonsi ist das klar-
("from the ")
(During the day, rain and snowfall over the Alps subside; in the northeast it still rains heavily at times, otherwise the stars are visible.)
(Rain and snow subside over the Alps during the night; in the north and northeast, occasional showers, otherwise clear.
ab den
beam
Yes
ab den am dornerstag regnet es in der nordwesthällee gebi- etsweise anfangs an den küsten mal im wechsel gibt es dann am freitag thaliches we (From Thursday, rain in the northwestern half. partly initially on the coasts alternating clouds; similar weather continues ab den ein kräftiges tief üher der nonlsee bringt uns ab morgen fruh teilweise kriffiige schneefalle und gefrierenden regen.
um dounerstag regen in der northällte in der sädhälfte mal mal wolken äinllihee wetter dann auch freitag.
mal wolken und
("from the ")
(On Thursday, rain the northern half: the south, sun and clouds alternating, similar weather also on Friday.)
and Friday.)
ab den
beam
No
vom nordmeer zieht ein kriitigee tief beran und bringt uns den morgenstunden hefiie schneefhlle zum teil auch gefricrenden regen.
("from the ")
(From a strong low over the North Sea brings us from early tomorrow partly heavy snowfall and freezing rain.)
(A strong low moves in from the North Sea, bringing heavy snowfall and some freezing rain from the morning hours.)
Table 8: Nine representative prompt-injection cases for the GB (gloss-basser model with English translations (including prompt meaning) shown in italic below each example.


Reviews:
[{"review_id": "AjoPpltvSP", "reviewer": "Reviewer_zRXC", "summary": "The paper introduces a token-level reliability metric for hallucination detection in sign language translation (SLT), combining encoder sensitivity to input perturbations and internal state changes under counterfactual video masking. The metric is evaluated on PHOENIX-2014T and CSL-Daily using both gloss-based and gloss-free models.", "questions": "Performance Impact on Gloss-Free SLT:\n- The paper positions reliability as a diagnostic tool, yet provides no evidence that it leads to tangible improvements in gloss-free sign language translation. Can the authors demonstrate substantial gains (e.g., ≥1.0 BLEU)?  \n\nCross-Architecture Validation:\n- Claims of model-agnosticism are unsubstantiated. Can the authors apply the reliability score to ≥3 diverse gloss-free architectures (e.g., SignBT, STMC, BLT, or non-autoregressive variants) and show:  Consistent hallucination prediction (correlation, AUC)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  Precisely identifies hallucination in SLT—especially in gloss-free settings—as a critical, under-addressed issue tied to visual grounding weakness.\n\n- The token-level reliability score is architecture-independent, requiring no reference translations, making it broadly applicable across existing and future SLT systems.\n\n- Combines encoder sensitivity (input robustness) and counterfactual masking (internal consistency) into a well-justified, interpretable proxy for visual grounding.\n\n- Establishes a reusable evaluation tool that can become a standard metric in SLT research and development.", "weaknesses": "- Lacks testing on larger, more diverse, or continuous SLT benchmarks (e.g., How2Sign, OpenASL).  \n- Tested on a small set of architectures (primarily transformer-based). \n- Unclear whether the proposed reliability score outperforms or complements prior art.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S81UJfeU5w", "reviewer": "Reviewer_bLfN", "summary": "This manuscript aims to evaluate \"hallucination\" in sign language translation (SLT), defined as the extent to which the decoder generates token-level predictions without relying on visual information. To assess hallucination in SLT accurately, the manuscript proposes feature-wise signals, such as changes in feature orientation and intensity shifts in attention. Additionally, it leverages prediction discrepancies under noisy conditions to evaluate hallucination. Extensive experiments demonstrate that the proposed reliability indicator correlates with the \"CHAIR\" phenomenon in SLT, achieving impressive detection performance.", "questions": "- As mentioned in the weaknesses, I doubt the necessity of introducing the concept of hallucination in SLT, as it may mislead future research. However, I remain open to opinions and experimental evidence that could demonstrate the significance of hallucination in SLT.\n- I also encourage the author to improve the writing and organization of this manuscript, and provide more valuable take home messages.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- S1. This manuscript is well-motivated, addressing the crucial issue of evaluating and improving the utilization of visual information in sign language translation (SLT), a topic that has garnered significant attention in recent years. The manuscript aims to design an indicator to shed light on this challenge.\n- S2. The manuscript provides a comprehensive set of indicators for assessing the utilization of visual information, considering both the feature space and the output space.\n- S3. The manuscript conducts thoughtful and thorough experiments on hallucination evaluation and detection.", "weaknesses": "- W1. The definition of hallucination in SLT is unclear. While hallucination in visual-language models (VLMs) is introduced due to the ambiguity in input range and output space, SLT has clearly defined output results and visual-language correspondences. The errors in hallucination can be directly measured using metrics like word error rate (WER) or similar set prediction metrics. Therefore, the introduction of the hallucination concept in SLT seems unnecessary.\n- W2. As a translation task, many words are predicted based on contextual information. What we are truly concerned with are the words that can only be predicted by visual information. However, this manuscript does not address the distinction between different types of tokens in this context.\n- W3. The insufficient use of visual information is a well-known issue in the sign language understanding community, and addressing this challenge is critical. While this manuscript attempts to detect hallucination, it does not propose any solutions for resolving this problem.\n- W4. The practical applicability of the proposed method is limited. As shown in Figure 1, the proposed metric shows some correlation with \"CHAIR\" in SLT, which is used to demonstrate the effectiveness of the indicator. However, why not directly use \"CHAIR\" as the hallucination indicator? Furthermore, the high variation in the proposed metric suggests its limited practical value.\n- W5. While Table 1 demonstrates that the proposed method achieves high hallucination detection accuracy, the process for obtaining ground-truth labels for detection remains unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wGvfmIYzlM", "reviewer": "Reviewer_afii", "summary": "This work investigates hallucination in sign language translation. The authors introduce a novel metric named reliability to measure the hallucination in SLT. The propose metric is based on two parts, i.e., internal model changes and output probability shifts. By adopting the CHAIR as the annotation on sentence-level, the proposed metric shows encouraging performance based on two SLT models on both benchmarks, i.e., PHOENIX-2014T and CSL-Daily.", "questions": "1.The authors provide the results on two SLT models. I suggest that the authors provide results on more open-source models. Additionally, the current method only considers cases where the input is video. Could it measures the different inputs, like poses?\n\n2.All formulas in the manuscript lack recommended numbering.\n\n3.There are notations in the equations lack descriptions. \n\nIn Line 154, $\\pi^{-1}$ lack descriptions. Which part of the model does “hidden state” represent?\n\nIn Line 164, $scale$ lack descriptions. For $s_t^{attn}$, does a large result also indicate higher reliability? Are the set “video” and “masked” same?\n\n4.In the 4 note, the authors provide a brief description about the calculation method of CHAIR. I think the authors should provide a more detailed description.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "It is the first work to investigate hallucination in sign language. The authors provide a novel metric to measure the reliability for SLT based on the visual input and text output. They also demonstrate the proposed reliability surpasses text-only baselines in detecting hallucinations.", "weaknesses": "1.The authors did not provide a clear analysis of the relationships about the three key concepts, i.e., sign language translation performance, the rate of hallucination and the proposed metric.\n\n2.The key steps of the proposed metric is missing, such as how to calculate the weights. The appendix is also not completed, which is confusing. For specific issues, please refer to the descriptions in the Question section.\n\n3.The authors claim that the hallucinations significantly impact the performance of SLT. However, there lacks data showing how hallucinations hurt translation quality. There is only limited comparative examples are given in the appendix, which is insufficient to support the main contributions.\n\n4.Since the goal of mitigating hallucinations is to improve sign language translation performance, I think the authors should clarify the specific application directions of the proposed metric in current practical contexts. \n\n5.The authors attribute the main reason of hallucinations to the poor performance of visual encoder. I think this point has been observed in the previous study [1]. It enhances visual representation capabilities by incorporating small pre-trained language models. Please provide more details about how to distinguish it from existing research in the manuscript.\n\n[1] Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. 2024. Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7071–7081, Torino, Italia. ELRA and ICCL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dYyeGUUrW3", "reviewer": "Reviewer_U3pX", "summary": "This paper tackles the problem of hallucination in gloss-free sign language translation. SLT is a challenging task, which has been facilitated by the use of large language models (LLMs) and vision language models (VLMs) in recent years. However, these models tend to hallucinate. Gloss-level supervision can reduce hallucinations, by grounding the model with sign-level supervision, but glosses are expensive to annotate and label (note from reviewer: and introduce other problems). The authors of this paper propose a novel method to detect hallucinations and ground models in visual inputs, which reduces hallucinations and improves the results on gloss-free sign language translation.", "questions": "- If we remove the subclause from the first line of your abstract, it reads \"Hallucination is particularly critical in SLT.\" That is not the point you are trying to make, so please rephrase this.\n- I recommend adding a statement about the dangers of using glosses for sign language translation at the end of the first paragraph of your introduction. They should not be used for sign language translation purposes: see [1]. This will further strengthen your paper since you are tackling gloss-free SLT.\n- Does your system have merit in domains other than SLT to detect hallucinations?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is, for the most part, well-written and pleasant to read.\n- The paper is well situated in related work.\n- This is important work, as it improves the reliability of SLT models, especially gloss-free ones (which is critical because glosses can be considered harmful for SLT.)", "weaknesses": "- The paper contains acronyms that are not defined, e.g., AP.\n- The methodology is rather math heavy and can be challenging to follow at times. It would benefit from some intuitive explanations of the equations.\n- There is no statement on whether the signing community was consulted for this work. The authors should make a clear statement on their hearing/signing status.\n- Appendix C and D are empty (probably because they contain only figures.) This layout problem should be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)