Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation

## Abstract

Abstract Hallucination, where models generate fluent text unsupported by visual evidence, remains a major flaw in vision–language models and is particularly critical in sign language translation (SLT). In SLT, meaning depends on precise grounding in video, and gloss-free models are especially vulnerable because they map continuous signer movements directly into natural language without intermediate gloss supervision that serves as alignment. We argue that hallucinations arise when models rely on language priors rather than visual input.
To capture this, we propose a token-level reliability measure that quantifies how much the decoder uses visual information. Our method combines feature-based sensitivity, which measures internal changes when video is masked, with counterfactual signals, which capture probability differences between clean and altered video inputs. These signals are aggregated into a sentence-level reliability score, providing a compact and interpretable measure of visual grounding.
We evaluate the proposed measure on two SLT benchmarks (PHOENIX-2014T and CSL-Daily) with both gloss-based and gloss-free models. Our results show that reliability predicts hallucination rates, generalizes across datasets and architectures, and decreases under visual degradations. Beyond these quantitative trends, we also find that reliability distinguishes grounded tokens from guessed ones, allowing risk estimation without references; when combined with text-based signals (confidence, perplexity, or entropy), it further improves hallucination risk estimation. Qualitative analysis highlights why gloss-free models are more susceptible to hallucinations. Taken together, our findings establish reliability as a practical and reusable tool for diagnosing hallucinations in SLT, and lay the groundwork for more robust hallucination detection in multimodal generation.

## 1 Introduction

Despite being a video-to-text task, sign language translation (SLT) differs fundamentally from other video understanding problems. Whereas most multimodal approaches focus on recognizing scenes or describing visual events, SLT must process a complete natural language, namely sign language, with its own vocabulary and grammar. Early approaches relied on glosses, which are explicit human annotations that map video to textual sign labels (Camgoz et al., 2018; 2020). Glosses provide strong supervision but limit scalability, since most large-scale datasets are only weakly labeled and lack gloss annotations.

Recent advances in SLT increasingly favor gloss-free architectures that rely on large language model backbones (Gong et al., 2024; Chen et al., 2024b; Hwang et al., 2025). While this shift alleviates the costly gloss annotation bottleneck, it also changes the nature of mistakes these systems make.
We show that when visual representations are weak whether due to ambiguous signing or low video quality, the language model tends to dominate decoding. This results in translations that may appear fluent yet fail to reflect the signer’s actual message, resulting in hallucinations. Such behavior mirrors findings in large vision–language models (LVLMs), where hallucinations often arise from the model’s tendency to prioritize language patterns over visual evidence (Ghosh et al., 2025; Parcalabescu & Frank, 2025). In an ideal SLT system, strong visual representations ground the translation, while the language model complements them by ensuring grammaticality and fluency. Hallucinations emerge precisely when this balance is disrupted in favor of the language model.

Hallucination detection has been studied in other multimodal settings, particularly in LVLMs and text-to-image or video generation, where strong language priors can override weaker visual signals (Chen et al., 2025; Min et al., 2025). In these domains, approaches often focus on aligning model attention with visual content or using perturbation-based analyses to identify unreliable outputs. However, SLT presents a more challenging scenario because the visual modality is not auxiliary (more than just understanding scene, detecting objects and action) but constitutes the source language itself. Consequently, hallucinations in SLT are directly equivalent to translation errors, making the detection and quantification of visual grounding critical.

A crucial first step toward mitigating and reducing hallucinations is the ability to characterize and detect them, and to assign reliability scores that allow ranking candidate translations by their risk of hallucination. Text-based signals such as confidence, entropy, and perplexity capture useful linguistic uncertainty, but they miss a critical aspect in SLT: whether the output is actually supported by the signed input. We therefore introduce a complementary dimension of hallucination detection we refer to as grounding usage which quantifies the extent to which generated tokens rely on video evidence rather than language priors. By quantifying grounding usage and combining this signal with established text-only baseline measures, we can identify hallucinations arising when models guess from language model priors instead of leveraging the video stream. To this end, we define a reliability score that compares standard decoding with counterfactual runs where the video is masked or replaced. These perturbations yield two classes of evidence: (i) changes in internal states and routing (e.g., hidden-state angular shifts, cross-attention activations), and (ii) probability margins between clean and counterfactual outputs. A linear fusion of the two types produces token-level reliabilities, which are then pooled into sentence-level scores. Our experiments across datasets (PHOENIX-2014T (Camgoz et al., 2018) and CSL-Daily (Zhou et al., 2021)) and architectures (T5 (Hwang et al., 2025) and mBART (Chen et al., 2022)) demonstrate that reliability not only surpasses text-only baselines in predicting hallucinations but also generalizes robustly in cross-dataset and cross-architecture transfer. Specifically, it achieves an accuracy of 97% and maintains competitive regression performance when approximating the level of hallucination rate, with a Spearman correlation of $\rho=0.72$ (between the predicted hallucination rate and the computed hallucination rate). These findings further reveal that gloss-free models hallucinate more often because they systematically under-utilize visual information, defaulting instead to linguistic priors. Our contributions and findings can be summarized as follows:

- •
To the best of our knowledge, this is the first work to investigate hallucination in SLT.
- •
We introduce a novel metric based on *grounding usage* that quantifies how much generated tokens rely on visual input, providing a direct measure of whether SLT outputs are grounded in the source video.
- •
We design a reliability scoring framework using counterfactual perturbations, capturing both internal model changes and output probability shifts.
- •
We empirically demonstrate that reliability surpasses text-only baseline metrics in detecting hallucinations and provide insights into why gloss-free models are particularly prone to them.

## 2 Related work

### 2.1 Sign Language Translation

Camgoz et al. (2018) proposed an end-to-end approach for generating spoken language translations directly from sign language videos and introduced the PHOENIX-2014T dataset, which provides both gloss annotations and spoken language translations for German Sign Language (DGS) videos. SLT systems and datasets are typically categorized as either gloss-based or gloss-free. Traditional SLT approaches relied on glosses as an intermediate representation. However, creating gloss annotations is time-consuming, which has led to increased interest in gloss-free SLT models. Examples of such approaches include extending Transformer architectures to SLT (Camgoz et al., 2020), leveraging sentence embeddings instead of glosses (Hamidullah et al., 2024), and integrating spatial and motion visual feature extractors with large language models (LLMs) (Hwang et al., 2025). Alongside model development, datasets for various sign languages have emerged and gained popularity, such as CSL-Daily (Zhou et al., 2021) for Chinese Sign Language (CSL) and How2Sign (Duarte et al., 2021) for American Sign Language (ASL). Using LLMs as the backbone for SLT has largely solved the fluency and scalability challenges (Wong et al., 2024; Gong et al., 2024; Inan et al., 2025), but this progress comes at a price as it increases reliance on language priors and makes the models more prone to hallucinations.

### 2.2 Hallucination Detection

Hallucinations are often characterized as the generation of content that is logically unsound or that departs from the source; in other words, model outputs that assert facts, events, or details not supported, or even contradicted, by the available evidence or ground truth (Ji et al., 2023). In multimodal learning, hallucinations have been widely studied in image and video captioning, particularly with respect to object mentions. For example, Rohrbach et al. (2018) introduced the CHAIR metric to detect and quantify hallucinated objects in MSCOCO image captions, showing that captioning models frequently insert visually ungrounded entities. Similar patterns have been observed in video captioning, where generated descriptions may include unsupported objects, actions, or temporal details (Ullah & Mohanta, 2022; Liu & Wan, 2023).

With the advent of LVLMs, hallucination detection techniques have expanded significantly. Approaches now include reinforcement learning to align outputs with factual grounding (Gunjal et al., 2024), fact verification methods that leverage external knowledge bases or retrievers (Yin et al., 2024; Sahu et al., 2024), and tool-augmented strategies that integrate external APIs to assess factual consistency (Chen et al., 2024a). Another noteworthy case is SelfCheckGPT (Manakul et al., 2023), a sampling-based approach that detects hallucinations by leveraging response consistency: factual knowledge tends to yield stable outputs across multiple generations, whereas hallucinations produce divergent or contradictory answers.

## 3 From Grounding to Guessing: Reliability as a Lens on Visual Usage

#### Intuition.

At each decoding step, a multimodal model can either *ground* the next token in the video or
*guess* it from language priors. Our goal is a token-level score $r_{t}$ that increases
when the decision genuinely relies on the video signal. Prior work in hallucination detection has mostly relied on text-intrinsic proxies such as entropy or perplexity (Kuhn et al., 2023; Farquhar et al., 2024; Obeso et al., 2025), which are agnostic to visual input. While these capture uncertainty, they fail to tell whether a confident prediction is grounded in evidence or merely plausible under language priors. In multimodal generation tasks, where visual evidence is the *source* as in SLT, this distinction is critical (Chen et al., 2022; Guo et al., 2024). To capture visual grounding, we propose a reliability score. At each step, we (i) measure feature-based sensitivity, which captures how much the output hidden states and the cross-attention drift when the visual input is removed; and (ii) compare decoding with the correct input (which we call clean) to counterfactual runs in which the video is masked or mismatched.

Intuitively, feature-based signals reflect *internal reliance* on visual context, while counterfactual signals capture *external evidence* that the video actually helps the prediction. Together, these provide a robust, interpretable measure of visual grounding.

### 3.1 Feature-based signals: internal reliance

We compare the decoder’s hidden states and attention patterns with and without video input.

- •
Hidden state sensitivity.
If the decoder hidden state at step $t$ changes direction substantially when the encoder input is removed, the token is influenced by the video. We build on prior work (Li et al., 2017; 2024) by quantifying this directional change as the normalized angle between the hidden states with video ($h^{\mathrm{vid}}_{t}$) and with masked input ($h^{0}_{t}$):
$s^{\mathrm{hid}}_{t}\;=\;\pi^{-1}\arccos\!\left(\frac{\langle h^{\mathrm{vid}}_{t},\,h^{0}_{t}\rangle}{\|h^{\mathrm{vid}}_{t}\|\,\|h^{0}_{t}\|}\right),$
where $\langle\cdot,\cdot\rangle$ denotes the inner product and $\|\cdot\|$ the Euclidean norm.
A larger $s^{\mathrm{hid}}_{t}$ indicates a stronger change in orientation, and thus stronger reliance on the video input.
- •
Cross-attention usage.
Decoder cross-attention $A^{(\ell,h)}_{t\to i}$ at layer $\ell\in[1,L]$ and head $h\in[1,H]$
(from decoder token $t$ to encoder position $i$) is aggregated across all layers and heads.
We compute the mean attention mass directed to video encoder positions and subtract
the corresponding mass under a null (masked) encoder input, with per-sequence rescaling:
$s^{\mathrm{attn}}_{t}\;=\;\mathrm{scale}\!\left(\frac{1}{LH}\sum_{\ell=1}^{L}\sum_{h=1}^{H}\sum_{i\in\mathrm{video}}A^{(\ell,h)}_{t\to i}\;-\;\frac{1}{LH}\sum_{\ell=1}^{L}\sum_{h=1}^{H}\sum_{i\in\mathrm{masked}}A^{(\ell,h)}_{t\to i}\right)$

### 3.2 Counterfactual signals: external evidence

While feature-based signals measure how much the decoder state *changes* when video is removed, they do not yet capture whether the video actively *helps* the chosen prediction. To complement the hidden state sensitivity measure, we introduce counterfactual signals, which quantify the *probability advantage* of real video compared to wrong or no video.

#### Setup.

We run three parallel decoder passes, all conditioned on the same prefix $c_{t}$:
(i) with the true video $x$, giving $p_{\mathrm{vid}}(\cdot\mid c_{t},x)$,
(ii) with no video $\varnothing$ (cross-attention disabled), giving $p_{0}(\cdot\mid c_{t})$, and
(iii) with a mismatched video $x^{\prime}$, giving $p_{\mathrm{mis}}(\cdot\mid c_{t},x^{\prime})$.

Let $y_{t}$ be the token chosen under the clean pass with $x$. We then define the counterfactual(^1^11We take the maximum rather than the mean to avoid false positives: if either the no-video or the mismatched-video condition already explains $y_{t}$ nearly as well as the true video, then the token should be treated as ungrounded.) distribution as

$$ $p_{\mathrm{cf}}(y_{t}\mid c_{t})\;=\;\max\big\{p_{0}(y_{t}\mid c_{t}),\;p_{\mathrm{mis}}(y_{t}\mid c_{t},x^{\prime})\big\}.$ $$

This counterfactual approach is related to Chlon et al. (2025),
but whereas they consider multiple counterfactuals, we focus on the single strongest alternative.

#### Signals definitions.

From these passes we compute five complementary measures of video contribution (deltas and margins)(^2^22Margins capture relative strength across scales, while deltas provide absolute probability gains; using both gives complementary views of video contribution.):

$$ $\displaystyle s^{\mathrm{log}}_{t}=\log p_{\mathrm{vid}}(y_{t})-\log p_{\mathrm{cf}}(y_{t})$ $\displaystyle\text{(log-probability margin)};$ $\displaystyle s^{\mathrm{logit}}_{t}=\operatorname{logit}p_{\mathrm{vid}}(y_{t})-\operatorname{logit}p_{\mathrm{cf}}(y_{t})$ $\displaystyle\text{(scale-stable logit margin)};$ $\displaystyle s^{\mathrm{prob}}_{t}=\sigma(s^{\mathrm{log}}_{t})$ $\displaystyle\text{(normalized probability gain)};$ $\displaystyle\Delta^{\mathrm{clean}}_{t}=p_{\mathrm{vid}}(y_{t})-p_{0}(y_{t}),~\Delta^{\mathrm{mis}}_{t}=p_{\mathrm{vid}}(y_{t})-p_{\mathrm{mis}}(y_{t})$ $\displaystyle\text{(advantages vs.\ no-video / mismatched)};$ $\displaystyle p_{\mathrm{vid}}(y_{t}),\;p_{0}(y_{t})$ (raw probabilities, optional). $$

#### Interpretation.

All measures increase when the correct video makes the token more likely than either counterfactual.(^3^33Here, a counterfactual means either removing the video input or replacing it with mismatched visual content.)
The scores $s^{\mathrm{log}}$ and $s^{\mathrm{logit}}$ capture raw margins on log- and logit-scales, robust to small changes in scale.
$s^{\mathrm{prob}}$ maps these into a $[0,1]$ probability advantage, interpretable as a “confidence boost.”
$\Delta^{\mathrm{clean}}$ and $\Delta^{\mathrm{mis}}$ isolate the effect of removing vs. replacing the video, helping diagnose whether errors come from lack of signal or misleading signal.

#### Role in reliability.

The counterfactual signal captures whether a token depends on visual input.
Concretely, reliability is high when $p_{\mathrm{vid}}(y_{t})$ exceeds the corresponding counterfactual probabilities, and low otherwise, that is, when the token is no more likely under the correct video than under its counterfactual variants. This directly links hallucination risk to the model’s *under-use* of visual evidence.

### 3.3 Token-level and Sentence-level fusion of reliability signals

#### ∙ \bullet Token-level fusion:

For each token, we have two types of reliability signals:

- 1.
*feature-based* signals $\mathbf{h}_{t}=\big[s^{\mathrm{hid}}_{t},\,s^{\mathrm{attn}}_{t}\big]$, weighted by $\mathbf{w}_{\mathrm{fb}}$,
- 2.
*counterfactual* signals
$\mathbf{g}_{t}=\big[s^{\mathrm{log}}_{t},\,s^{\mathrm{logit}}_{t},\,s^{\mathrm{prob}}_{t},\,\Delta^{\mathrm{clean}}_{t},\,\Delta^{\mathrm{mis}}_{t},\,p_{\mathrm{vid}}(y_{t}),\,p_{0}(y_{t})\big]$, weighted by $\mathbf{w}_{\mathrm{cf}}$.

A token-level reliability score $r_{t}$ is then computed by a linear fusion with sigmoid activation:

$$ $r_{t}=\sigma\!\left(\mathbf{w}_{\mathrm{fb}}^{\top}\mathbf{h}_{t}\;+\;\mathbf{w}_{\mathrm{cf}}^{\top}\mathbf{g}_{t}\;+\;b\right).$ $$

#### ∙ \bullet Sentence-level fusion:

Since hallucination supervision is only available at the sentence level,
we aggregate token reliabilities $\{r_{t}\}_{t=1}^{T}$ into a fixed-size feature vector using *early pooling*.
For each sequence, we compute:

$$ $R_{\mathrm{tail}\text{-}q}=\frac{1}{\lceil qT\rceil}\sum_{t\in\text{lowest }q\%}r_{t}\;(\text{average over lowest $q$\% tokens})$ $$

We primarily rely on tail-10 pooling ($R_{\mathrm{tail}\text{-}10}$).
Other options include $R_{\mathrm{mean}}$ (average over tokens), $R_{\mathrm{harm}}$ (harmonic mean), or $R_{\mathrm{min}}$ (minimum token reliability).

The resulting sentence-level reliabilities $R_{\mathrm{tail}\text{-}q},\{R_{\mathrm{mean}},\dots\}$
serve as inputs to downstream classifiers or regressors, enabling consistent comparison across utterances of varying length without masking.

### 3.4 Text only baselines and combination

To compare reliabilities across datasets and models, we fit a monotone mapping from the reliability deficit to the hallucination rate CHAIR:(^4^44CHAIR (Caption Hallucination Assessment with Image Relevance) is a standard hallucination metric, here adapted to SLT by checking entity overlap between generated text and visual content (Rohrbach et al., 2018). Details can be found in Appendix [F.2](https://arxiv.org/html/2510.18439v1#A6.SS2).)

$$ $\text{CHAIR}\approx\text{ISO}\big(1-R_{\ast}\big),$ $$

where $R_{\ast}$ is a chosen pooled statistic (e.g., tail-10%),
and ISO denotes isotonic regression on a held-out split.

We benchmark reliability against text-only proxies computed from the clean run:
confidence $C_{t}=p_{\mathrm{vid}}(y_{t})$,
token entropy $H_{t}=-\sum_{w}p_{\mathrm{vid}}(w)\log p_{\mathrm{vid}}(w)$,
and self-perplexity $\mathrm{PPL}_{t}=1/p_{\mathrm{vid}}(y_{t})$ (perplexity under the model’s own predictive distribution),
pooled with the same pooling operator ($R_{\mathrm{tail}\text{-}10}$).
Finally, we also evaluate a combined META variant that concatenates grounding-based signals with these text-side baselines (confidence, entropy, perplexity) and fits a linear layer on the pooled statistics. This allows us to assess whether our visual grounding signal provides complementary information beyond existing text-only measures.

Figure: Figure 1: Correlation between reliability ($R_{\mathrm{tail}\text{-}10}$) and hallucination rates (CHAIR) on PHOENIX-2014T (PHOENIX) and CSL-Daily (CSL). Higher reliability aligns with fewer hallucinations for both gloss-free and gloss-based models.
Refer to caption: iclr2026/figures/corr_chair_R.png

## 4 Experiments and Results

### 4.1 Experimental Design

Experiments are done on two datasets PHOENIX-2014T (DGS$\to$DE) (Camgoz et al., 2018) and CSL-Daily (CSL$\to$ZH) (Zhou et al., 2021), with two systems: SpaMo, a gloss-free (GF) SLT model (Hwang et al., 2025), and TwoStream-SLT, a gloss-based (GB) model (Chen et al., 2022).

First, reliability features are extracted following the methodology of Section [3](https://arxiv.org/html/2510.18439v1#S3), with the detailed algorithm provided in Appendix [D](https://arxiv.org/html/2510.18439v1#A4).

Next, the hallucination detection model is trained to identify hallucinated outputs, leveraging both our grounding-based reliability signals (visual features) and the baseline features described in Section [3.4](https://arxiv.org/html/2510.18439v1#S3.SS4), all integrated within the META (ours) framework. Accuracy is computed at a 0.5 threshold, while isotonic regression is employed to predict both $\mathrm{CHAIR}$ and its complement $(1-\mathrm{CHAIR})$. Evaluation metrics include AUROC (AUC), AUPRC (AP), and ACC@0.5 (ACC). The resulting weights are further examined for their transferability across datasets and model families.

Finally, the behavior of reliability is analyzed under both visual and linguistic degradations. Stress tests cover Gaussian noise, temporal downsampling, and random frame dropping on the visual side, as well as wrong-start prompt prefixes to assess language-model dominance during decoding. In addition to CHAIR, the analysis tracks how BLEU (Papineni et al., 2002) an $n$-gram precision metric and chrF (Popović, 2015) a character-$n$-gram F-score sensitive to morphological variation, evolve under these perturbations and how their trends correlate with reliability.

Evaluation includes correlations between reliability and $\mathrm{CHAIR}$ (Pearson, Spearman), regression fits ($R$ against $1-\mathrm{CHAIR}$), calibration robustness under cross-dataset/model transfer, and ablations comparing feature vs. counterfactual signals and pooling operators (mean vs. tail-10%). Implementation details appear in Appendix [F](https://arxiv.org/html/2510.18439v1#A6).

### 4.2 Results

#### Low visual grounding as hallucination signal.

We begin by asking whether our reliability signal is meaningfully aligned with hallucination. Figure [1](https://arxiv.org/html/2510.18439v1#S3.F1) shows a clear negative correlation between reliability ($R_{\mathrm{tail}\text{-}10}$) and CHAIR across datasets and model types: as grounding usage (reliability) increases, hallucination decreases. This confirms that our measure captures an essential property of hallucination detection. Interestingly, the correlation is consistently stronger in gloss-free models compared to gloss-based models. We attribute this gap to the higher hallucination rates in gloss-free systems: more frequent errors provide richer supervision for the reliability signal, making the negative slope sharper. In contrast, gloss-based systems hallucinate less often, yielding a weaker but still consistent trend.

These results establish the foundation for our subsequent analyses. Grounding usage is not just an auxiliary statistic, it already shows strong correlation with hallucination rates, especially in the more challenging gloss-free setting where visual grounding is the only safeguard against hallucinations. This validates its relevance as a hallucination signal and motivates the deeper evaluations that follow, including detection, regression, transfer, and stress tests.

**Table 1: Summary of performance across datasets (CSL-Daily (CSL) and PHOENIX-2014T (PHOENIX)) and models (Gloss-based (GB) and Gloss-free (GF)). Top block reports detection metrics (AUC / AP / ACC), bottom block reports regression metrics (Pearson / Spearman / ISO). Best and second best values per metric are bold-faced.**
|  | CSL GB | CSL GF | PHOENIX GB | PHOENIX GF |
| --- | --- | --- | --- | --- |
| Detection (AUC / AP / ACC) |  |  |  |  |
| Grounding (ours) | 0.803 / 0.991 / 0.963 | 0.951 / 0.998 / 0.970 | 0.827 / 0.954 / 0.899 | 0.918 / 0.986 / 0.938 |
| META (ours) | 0.865 / 0.994 / 0.753 | 0.963 / 0.999 / 0.883 | 0.899 / 0.978 / 0.864 | 0.940 / 0.992 / 0.872 |
| Confidence | 0.647 / 0.976 / 0.508 | 0.953 / 0.998 / 0.868 | 0.618 / 0.887 / 0.645 | 0.926 / 0.989 / 0.860 |
| Perplexity | 0.609 / 0.980 / 0.610 | 0.952 / 0.998 / 0.842 | 0.595 / 0.892 / 0.561 | 0.888 / 0.979 / 0.713 |
| Token entropy | 0.729 / 0.985 / 0.665 | 0.958 / 0.998 / 0.866 | 0.639 / 0.900 / 0.581 | 0.871 / 0.975 / 0.731 |
| Regression (Pearson / Spearman / ISO) |  |  |  |  |
| Grounding (ours) | -0.439 / -0.426 / 0.465 | -0.682 / -0.680 / 0.722 | -0.458 / -0.400 / 0.493 | -0.623 / -0.590 / 0.650 |
| META (ours) | -0.518 / -0.515 / 0.543 | -0.736 / -0.705 / 0.755 | -0.610 / -0.574 / 0.637 | -0.650 / -0.613 / 0.675 |
| Confidence | -0.253 / -0.269 / 0.294 | -0.685 / -0.657 / 0.714 | -0.344 / -0.308 / 0.430 | -0.612 / -0.578 / 0.637 |
| Perplexity | -0.079 / -0.082 / 0.120 | -0.594 / -0.661 / 0.719 | -0.189 / -0.229 / 0.296 | -0.400 / -0.542 / 0.609 |
| Token entropy | -0.158 / -0.143 / 0.195 | -0.670 / -0.698 / 0.748 | -0.275 / -0.273 / 0.342 | -0.468 / -0.544 / 0.625 |

Figure: Figure 2: Visualization of detection metrics (AUC, AP, ACC) across datasets and models. We compare Grounding (ours), META (ours), and the best baseline (Confidence/Perplexity/Entropy) as seen in Table [1](https://arxiv.org/html/2510.18439v1#S4.T1).
Refer to caption: iclr2026/figures/radar_detection_metrics.png

#### Hallucination detection (binary classification).

Table [1](https://arxiv.org/html/2510.18439v1#S4.T1) upper block shows that our grounding-based reliability consistently improves over text-only baselines on hallucination detection. Across datasets and models, we gain up to +0.20 AUC and +0.30 ACC compared to entropy and perplexity, reaching near-perfect AP ($\approx 0.99$). We present the detailed gain over the other baselines in Table [2](https://arxiv.org/html/2510.18439v1#S4.T2). The META (ours) variant, which fuses grounding with text-side uncertainty signals, achieves the strongest overall detection performance, demonstrating that grounding contributes complementary information beyond text probability-based baselines. At the same time, pure Grounding (ours) is slightly better in terms of accuracy, as highlighted in the Figure [2](https://arxiv.org/html/2510.18439v1#S4.F2), showing that visual evidence can sharpen decision boundaries in ways text-only signals cannot. This advantage is especially critical in sign language translation, and even more so in the gloss-free case, where no intermediate gloss supervision is available and reliable grounding in the continuous video stream is the only safeguard against hallucinations. More broadly, this underscores the need in multimodal generation to go beyond text-only uncertainty: hallucination detection benefits from explicitly checking whether outputs are supported by the input modality.

**(a) Detection: $\Delta$(Grounding–baseline). A=AUC, P=AP, C=ACC.**
| Detection<br>(A / P / C) | CSL<br>GB | CSL<br>GF | PHOENIX<br>GB | PHOENIX<br>GF |
| --- | --- | --- | --- | --- |
| Confidence | A:+0.155<br>P:+0.015<br>C:+0.455 | A:-0.002<br>P:-0.000<br>C:+0.102 | A:+0.209<br>P:+0.067<br>C:+0.254 | A:-0.008<br>P:-0.003<br>C:+0.078 |
| Perplexity | A:+0.194<br>P:+0.010<br>C:+0.353 | A:-0.001<br>P:-0.000<br>C:+0.128 | A:+0.232<br>P:+0.062<br>C:+0.338 | A:+0.029<br>P:+0.008<br>C:+0.224 |
| Token entropy | A:+0.073<br>P:+0.005<br>C:+0.298 | A:-0.007<br>P:-0.000<br>C:+0.105 | A:+0.189<br>P:+0.054<br>C:+0.318 | A:+0.047<br>P:+0.011<br>C:+0.207 |

#### Predicting CHAIR (regression).

On the regression task (see Table  [1](https://arxiv.org/html/2510.18439v1#S4.T1) bottom block), where CHAIR is approximated as a continuous target, text-based baselines achieve slightly higher than our the correlations obtained using only our Grounding (ours) (e.g., token entropy reaches $\rho=-0.698$ on CSL GF, perplexity $\rho=-0.542$ on PHOENIX GF), reflecting their strength as direct uncertainty estimators since entropy and perplexity are tied to token-level probabilities. Our grounding signal remains competitive, with Spearman correlations up to $\rho=-0.680$ (CSL GF) and isotonic fits of $0.72$–$0.65$ across datasets. Its counterfactual components exploit probability shifts between clean and perturbed video to provide an energy-based ranking that complements text baselines. The main observation is that while grounding alone does not fully match text-only uncertainty measures, it contributes complementary cues: text features capture uncertainty within the language model, while grounding verifies whether predictions are visually supported. Together, as shown by the META (ours) variant (reaching $\rho=-0.736$ on CSL GF and ISO up to $0.76$), this synergy yields the most reliable hallucination detection across datasets and transfer settings.

#### Cross-dataset and cross-model generalization.

A linear calibration trained on one dataset transfers to another one with only a small drop in regression performance, and the same holds across gloss-free $\leftrightarrow$ gloss-based models. Some loss is expected, as the calibration reflects inevitable dataset- and model-specific biases.
Nevertheless, reliability remains the strongest signal across transfers (the full transfer matrix is shown in the Appendix, Fig. [8](https://arxiv.org/html/2510.18439v1#A5.F8)).
In Fig. [3](https://arxiv.org/html/2510.18439v1#S4.F3)(a) we present cross-dataset transfer within the same model family (GF vs GB). It remains close, as in CSL-GF $\rightarrow$ PHOENIX-GF, where the amount of hallucinations in both in source and target might be similar and therefore the calibration signal learnable.
We present the transfer between gloss-based and gloss-free models in Fig. [3](https://arxiv.org/html/2510.18439v1#S4.F3)(b) leads to a more pronounced degradation in performance, indicating disparities in tokenization granularity and a reduction in exposure to linguistically informative samples (as gloss-free models tend to exhibit higher hallucination rates): gloss-based systems operate mostly at the word level to maintain gloss alignment, while gloss-free systems use subunits like (Kudo & Richardson, 2018). Grounding signals align better with subunits, which clearly indicate which parts of words are guessed independently of video.
Subunits level tokens often marks stems, subword boundaries which also correspond to visual units in sign language; glosses mostly represent content words, which are used as objects in CHAIR adaptation.
This also suggests that transfer of the calibration to newer gloss-free architectures is more feasible, with smaller loss in terms of post-calibration performance, due to closer alignment with pretrained LLM subword tokenization.
n Fig. [3](https://arxiv.org/html/2510.18439v1#S4.F3)(c), we also report the transfer gaps observed when transferring across both datasets and model types. The results confirm that the performance degradation is larger when reliability weights are transferred from gloss-based (GB) to gloss-free (GF) models, whereas the opposite direction (GF→GB) exhibits comparatively smaller gaps.

Figure: (a)
Refer to caption: iclr2026/figures/transfer_spearman_strict_iclr_cross-model_gap_4.png

#### Why gloss-free systems hallucinate more often.

Figure: Figure 4: Token-level distributions (normalized [0,1]) for confidence ($-\log p$, left), grounding log-margin ($\log p(\text{video})-\log p(\text{cf})$, middle), and hidden-state sensitivity angle (right).
Refer to caption: iclr2026/figures/dist_tokens_baseline_confidence_norm.png

We compare token-level distributions of (i) *grounding log-margin* $\log p(\text{video})-\log p(\text{cf})$, (ii) *hidden-state sensitivity angle*, and (iii) *confidence* ($-\log p$; near 0 means over-confident), as shown in Figure [4](https://arxiv.org/html/2510.18439v1#S4.F4).
Log-margin: GF is strongly left-shifted on both datasets (most prominently on CSL), while GB maintains a longer high-margin tail, indicating that under counterfactual video, GF tokens often remain plausible.
Hidden-state sensitivity angle: GF concentrates at smaller angles ($\sim 0.20\!-\!0.35$ vs. $\sim 0.55\!-\!0.70$ for GB), indicating weaker state updates when the visual evidence changes. Confidence-based hallucination signal (-$\log p$): GF mass sits near zero (over-confidence), whereas GB exhibits a heavier tail.
The conjunction of *low margin + small angle + highly overconfident* occurs far more frequently in GF (subword tokenization), exactly where the model should lean on the video but does not; GB (word-level) yields more high-margin, video-responsive tokens. Consequently, text-only uncertainty reflects the LM’s internal belief but misses cases where the *video fails to exert influence*. Our grounding signal directly evaluates this evidence, and the *META (ours)* fusion outperforms others by integrating generic uncertainty with explicit visual support. From this observation, we formalize the claim that *gloss-free* training increases hallucinations in SLT by increasing the prevalence of weak visual use (see proof in Appendix  [C](https://arxiv.org/html/2510.18439v1#A3)).

#### Reliability under visual degradation.

To test whether reliability reflects visual usage, we systematically degraded the input in two ways:
(i) by adding random noise to video features (see Fig.  [5](https://arxiv.org/html/2510.18439v1#S4.F5) (left)), and (ii) by randomly dropping video frames (see Fig.  [5](https://arxiv.org/html/2510.18439v1#S4.F5) (right)).
We then measured how the reliability measure (trained via regression on $1\!-\!\text{CHAIR}$)
evolves under these perturbations. For visualization, we rescale reliability to $[0,1]$ and subsequently
to percentages; this scaling does not affect correlations, as they are invariant to linear transformations.

Reliability generally decreases as degradation increases, consistent with its role as a proxy for visual usage. The trend is not strictly monotonic: raw regression outputs can fluctuate upward at higher degradation levels. This behavior is expected, since reliability is optimized to *rank* hallucination risk by approximating $1\!-\!\text{CHAIR}$, rather than to produce calibrated probabilities. In contrast, the detection task employs a logistic-regression head to yield probability-like outputs.

The Fig. [5](https://arxiv.org/html/2510.18439v1#S4.F5) shows that CHAIR trends closely to anti-BLEU-1, the unigram version of BLEU,
as both measure ratios over words, although CHAIR is more selective. Reliability follows the same direction but with more noise, reflecting its design as an anti-CHAIR signal for ranking. More details and the concrete numbers are provided in Appendix [G](https://arxiv.org/html/2510.18439v1#A7).

Notice that our objective here is not exact matching of CHAIR but to demonstrate that reliability tracks its variation and effectively captures degradation-induced hallucinations.

Figure: Figure 5: Reliability under visual degradation. Left: CSL with added feature noise. Right: PHOENIX with frame dropping. Reliability decreases with stronger degradation, with minor non-monotonic fluctuations, reflecting its role as a regression-based ranking signal rather than a calibrated probability.
Refer to caption: iclr2026/figures/degradation_noise_csl.png

#### Prompt injection and language priors.

We further analyze model behavior under adversarially injected start tokens.
A comprehensive set of representative samples spanning multiple injected prompts and both decoding settings is presented in Appendix [H.2](https://arxiv.org/html/2510.18439v1#A8.SS2).
These qualitative cases reveal a consistent pattern: gloss-free (GF) models
frequently continue the injected prefix and generate fluent continuations that follow
linguistic priors, while gloss-based (GB) models trained with explicit visual
supervision tend to ignore or rapidly override the injected cue and realign with the video evidence.

For instance, with the injected start “*ab den*” (*from the*), GF produces
*“ab den nachmittagsstunden lässt der regen oder schnee an den alpen nach …”*
(*from the afternoon hours the rain or snow in the Alps eases …*), faithfully extending
the injected phrase but ignoring the signed content. In contrast, GB generates
*“ab den ein kräftiges tief über der nordsee bringt uns ab morgen früh schneefälle …”*
(*a strong low over the North Sea brings us snowfall from tomorrow morning …*),
which discards the misleading prefix and aligns more closely with the reference.

This contrast substantiates our claim that *language priors dominate in gloss-free models*.
In the absence of intermediate gloss supervision, GF relies heavily on text-only linguistic expectations at decoding, making injected prefixes act as strong attractors. In contrast, GB benefits from explicit supervision that reinforces cross-modal alignment,
resulting in outputs that are more faithful to the signed input and less vulnerable to text-only biases.

## 5 Conclusion

Hallucination in multimodal generation cannot be fully explained by text-intrinsic signals such as confidence, entropy, or perplexity. While these signals capture uncertainty, they do not reveal whether outputs are actually grounded in external visual evidence. This limitation is especially critical in SLT, where the visual input is the true source language.

In this work, we propose grounding usage as a new dimension for hallucination detection. Grounding usage quantifies how much each generated token is supported by visual input, and when combined with text-based signals (confidence, perplexity, or entropy), it improves hallucination detection by distinguishing visually grounded from visually unsupported tokens. Our experiments show that incorporating grounding usage with established text-based baselines improves predictive performance.
Our analysis further reveals a systematic imbalance: gloss-free models trained without explicit visual supervision tend to under-use the visual input, which explains their higher hallucination rates relative to gloss-based models.
The latter benefit from intermediate gloss supervision, which enforces stronger visual grounding. In other words, hallucinations in gloss-free SLT largely arise when the model “guesses” based on linguistic priors instead of truly attending to visual evidence.

In summary, a good multimodal model should strike the right balance between guessing (via language priors for fluency and grammaticality) and grounding (via visual evidence for factual correctness). Visual grounding alone is insufficient, but explicitly disentangling its contribution complements existing text-only signals and provides a more robust framework for hallucination detection.

## References

- Camgoz et al. (2018)
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden.
Neural sign language translation.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 7784–7793, 2018.
- Camgoz et al. (2020)
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden.
Sign language transformers: Joint end-to-end sign language recognition and translation.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 10023–10033, 2020.
- Chen et al. (2025)
Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, and Chunhua Shen.
PerturboLLaVA: Reducing multimodal hallucinations with perturbative visual training.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=j4LITBSUjs](https://openreview.net/forum?id=j4LITBSUjs).
- Chen et al. (2024a)
Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen.
Unified hallucination detection for multimodal large language models.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 3235–3252, Bangkok, Thailand, August 2024a. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.178.
URL [https://aclanthology.org/2024.acl-long.178/](https://aclanthology.org/2024.acl-long.178/).
- Chen et al. (2022)
Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak.
Two-stream network for sign language recognition and translation.
*Advances in Neural Information Processing Systems*, 35:17043–17056, 2022.
- Chen et al. (2024b)
Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao.
Factorized learning assisted with large language model for gloss-free sign language translation.
In *Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)*, pp. 7071–7081, Torino, Italia, May 2024b. ELRA and ICCL.
URL [https://aclanthology.org/2024.lrec-main.620/](https://aclanthology.org/2024.lrec-main.620/).
- Chlon et al. (2025)
Leon Chlon, Sarah Rashidi, Zein Khamis, and MarcAntonio M. Awada.
LLMs are Bayesian, in Expectation, not in Realization, 2025.
URL [https://arxiv.org/abs/2507.11768](https://arxiv.org/abs/2507.11768).
- Duarte et al. (2021)
Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto.
How2sign: a large-scale multimodal dataset for continuous american sign language.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 2735–2744, 2021.
- Farquhar et al. (2024)
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal.
Detecting hallucinations in large language models using semantic entropy.
*Nature*, 630:625 – 630, 2024.
URL [https://api.semanticscholar.org/CorpusID:270615909](https://api.semanticscholar.org/CorpusID:270615909).
- Ghosh et al. (2025)
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi, Oriol Nieto, Zeyu Jin, and Dinesh Manocha.
Visual description grounding reduces hallucinations and boosts reasoning in LVLMs.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=3PRvlT8b1R](https://openreview.net/forum?id=3PRvlT8b1R).
- Gong et al. (2024)
Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu.
Llms are good sign language translators.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 18362–18372, June 2024.
- Gunjal et al. (2024)
Anisha Gunjal, Jihan Yin, and Erhan Bas.
Detecting and preventing hallucinations in large vision language models.
In *Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence*, AAAI’24/IAAI’24/EAAI’24. AAAI Press, 2024.
ISBN 978-1-57735-887-9.
doi: 10.1609/aaai.v38i16.29771.
URL [https://doi.org/10.1609/aaai.v38i16.29771](https://doi.org/10.1609/aaai.v38i16.29771).
- Guo et al. (2024)
Zhengsheng Guo, Zhiwei He, Wenxiang Jiao, Xing Wang, Rui Wang, Kehai Chen, Zhaopeng Tu, Yong Xu, and Min Zhang.
Unsupervised sign language translation and generation.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Findings of the Association for Computational Linguistics: ACL 2024*, pp. 14041–14055, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-acl.835.
URL [https://aclanthology.org/2024.findings-acl.835/](https://aclanthology.org/2024.findings-acl.835/).
- Hamidullah et al. (2022)
Yasser Hamidullah, Josef van Genabith, and Cristina España-Bonet.
Spatio-temporal sign language representation and translation.
In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pp. 977–982, 2022.
- Hamidullah et al. (2024)
Yasser Hamidullah, Josef van Genabith, and Cristina España-Bonet.
Sign language translation with sentence embedding supervision.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pp. 425–434, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-short.40.
URL [https://aclanthology.org/2024.acl-short.40/](https://aclanthology.org/2024.acl-short.40/).
- Hwang et al. (2025)
Eui Jun Hwang, Sukmin Cho, Junmyeong Lee, and Jong C. Park.
An efficient gloss-free sign language translation using spatial configurations and motion dynamics with LLMs.
In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 3901–3920, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
ISBN 979-8-89176-189-6.
doi: 10.18653/v1/2025.naacl-long.197.
URL [https://aclanthology.org/2025.naacl-long.197/](https://aclanthology.org/2025.naacl-long.197/).
- Inan et al. (2025)
Mert Inan, Anthony Sicilia, and Malihe Alikhani.
SignAlignLM: Integrating multimodal sign language processing into large language models.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Findings of the Association for Computational Linguistics: ACL 2025*, pp. 3691–3706, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-256-5.
doi: 10.18653/v1/2025.findings-acl.190.
URL [https://aclanthology.org/2025.findings-acl.190/](https://aclanthology.org/2025.findings-acl.190/).
- Ji et al. (2023)
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung.
Survey of hallucination in natural language generation.
*ACM Comput. Surv.*, 55(12), March 2023.
ISSN 0360-0300.
doi: 10.1145/3571730.
URL [https://doi.org/10.1145/3571730](https://doi.org/10.1145/3571730).
- Kudo & Richardson (2018)
Taku Kudo and John Richardson.
SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.
In Eduardo Blanco and Wei Lu (eds.), *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pp. 66–71, Brussels, Belgium, November 2018. Association for Computational Linguistics.
doi: 10.18653/v1/D18-2012.
URL [https://aclanthology.org/D18-2012/](https://aclanthology.org/D18-2012/).
- Kuhn et al. (2023)
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar.
Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=VD-AYtP0dve](https://openreview.net/forum?id=VD-AYtP0dve).
- Li et al. (2017)
Jiwei Li, Will Monroe, and Dan Jurafsky.
Understanding neural networks through representation erasure, 2017.
URL [https://arxiv.org/abs/1612.08220](https://arxiv.org/abs/1612.08220).
- Li et al. (2024)
Maximilian Li, Xander Davies, and Max Nadeau.
Circuit breaking: Removing model behaviors with targeted ablation, 2024.
URL [https://arxiv.org/abs/2309.05973](https://arxiv.org/abs/2309.05973).
- Liu & Wan (2023)
Hui Liu and Xiaojun Wan.
Models see hallucinations: Evaluating the factuality in video captioning.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 11807–11823, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.723.
URL [https://aclanthology.org/2023.emnlp-main.723/](https://aclanthology.org/2023.emnlp-main.723/).
- Manakul et al. (2023)
Potsawee Manakul, Adian Liusie, and Mark Gales.
SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 9004–9017, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.557.
URL [https://aclanthology.org/2023.emnlp-main.557/](https://aclanthology.org/2023.emnlp-main.557/).
- Min et al. (2025)
Kyungmin Min, Minbeom Kim, Kang-il Lee, Dongryeol Lee, and Kyomin Jung.
Mitigating hallucinations in large vision-language models via summary-guided decoding.
In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), *Findings of the Association for Computational Linguistics: NAACL 2025*, pp. 4183–4198, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
ISBN 979-8-89176-195-7.
doi: 10.18653/v1/2025.findings-naacl.235.
URL [https://aclanthology.org/2025.findings-naacl.235/](https://aclanthology.org/2025.findings-naacl.235/).
- Obeso et al. (2025)
Oscar Obeso, Andy Arditi, Javier Ferrando, Joshua Freeman, Cameron Holmes, and Neel Nanda.
Real-time detection of hallucinated entities in long-form generation.
*arXiv preprint arXiv:2509.03531*, 2025.
- Papineni et al. (2002)
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation.
In *Proceedings of the 40th annual meeting of the Association for Computational Linguistics*, pp. 311–318, 2002.
- Parcalabescu & Frank (2025)
Letitia Parcalabescu and Anette Frank.
Do vision & language decoders use images and text equally? how self-consistent are their explanations?
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=lCasyP21Bf](https://openreview.net/forum?id=lCasyP21Bf).
- Popović (2015)
Maja Popović.
chrf: character n-gram f-score for automatic mt evaluation.
In *Proceedings of the tenth workshop on statistical machine translation*, pp. 392–395, 2015.
- Rohrbach et al. (2018)
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko.
Object hallucination in image captioning.
In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pp. 4035–4045, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
doi: 10.18653/v1/D18-1437.
URL [https://aclanthology.org/D18-1437/](https://aclanthology.org/D18-1437/).
- Rust et al. (2024)
Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgoz, and Jean Maillard.
Towards privacy-aware sign language translation at scale.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 8624–8641, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.467.
URL [https://aclanthology.org/2024.acl-long.467/](https://aclanthology.org/2024.acl-long.467/).
- Sahu et al. (2024)
Pritish Sahu, Karan Sikka, and Ajay Divakaran.
Pelican: Correcting hallucination in vision-LLMs via claim decomposition and program of thought verification.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 8228–8248, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.470.
URL [https://aclanthology.org/2024.emnlp-main.470/](https://aclanthology.org/2024.emnlp-main.470/).
- Ullah & Mohanta (2022)
Nasib Ullah and Partha Pratim Mohanta.
Thinking hallucination for video captioning.
In *Proceedings of the Asian Conference on Computer Vision*, pp. 3654–3671, 2022.
- Vilar et al. (2006)
David Vilar, Jia Xu, D’Haro Luis Fernando, and Hermann Ney.
Error analysis of statistical machine translation output.
In *LREC*, pp. 697–702, 2006.
- Wong et al. (2024)
Ryan Wong, Necati Cihan Camgoz, and Richard Bowden.
Sign2GPT: Leveraging large language models for gloss-free sign language translation.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=LqaEEs3UxU](https://openreview.net/forum?id=LqaEEs3UxU).
- Yin et al. (2024)
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen.
Woodpecker: Hallucination correction for multimodal large language models.
*Science China Information Sciences*, 67(12):220105, 2024.
- Zhou et al. (2021)
Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li.
Improving sign language translation with monolingual data by sign back-translation.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 1316–1325, June 2021.

## Appendix A LLM Usage

We used LLMs solely for language polishing and minor phrasing edits.

## Appendix B Grounding or Guessing? Hypotheses and Intuitions

SLT has recently shifted from gloss-based pipelines (Camgoz et al., 2018; Chen et al., 2022) to gloss-free models (Hamidullah et al., 2022; Gong et al., 2024; Hamidullah et al., 2024; Hwang et al., 2025). While gloss annotations provide strong immediate supervision, they are costly to collect and limit scalability Camgoz et al. (2018); Zhou et al. (2021). Gloss-free approaches promise broader applicability, but they also introduce new challenges in grounding the translation in the visual signal (Guo et al., 2024; Rust et al., 2024).

Encoder–decoder models face a constant tension: tokens may be *grounded* in the source input or simply *guessed* from language priors. Function words and inflections can often be guessed predicted from context, but content words, numbers, names, entities require grounding in the encoder signal to avoid error.

#### Why gloss-free SLT drifts.

In gloss-based pipelines, explicit sign annotations provide strong visual anchors that keep the decoder aligned with the video. Gloss-free systems, however, pair video encoders directly with LLM decoders under weak sentence-level supervision. In the absence of dense visual anchors, the decoder defaults to and leans heavily on its language prior, producing fluent but unsupported tokens (i.e., hallucinations). The result is fluent text that is not grounded in the video, i.e., hallucination.

#### Towards more hallucinations in SLT.

Early SLT systems (Camgoz et al., 2018; 2020) mostly failed by omission: missing vocabulary or unseen expressions due to scarce training data. Even so, their outputs remained visually constrained by the gloss signal. Gloss-free models backed by strong LLMs, in contrast, exhibit a different failure mode: hallucinations dominate (Hwang et al., 2025). Entire entities absent from the video are introduced solely because the linguistic bias of the decoder fills in the gaps. So we currently observe a shift in the error profile: hallucinations dominate, where entities absent from the video are introduced by linguistic bias alone.

#### Reliability as grounding measure.

We hypothesize that hallucinations arise precisely when decoding relies on guessing rather than grounding. To test this, we introduce *reliability*, a token- and sentence-level measure of how much the decoder relies on the encoder signal of encoder usage. Figure [6](https://arxiv.org/html/2510.18439v1#A2.F6) and Figure  [7](https://arxiv.org/html/2510.18439v1#A2.F7) show both mechanisms and the aggregate effect: Figure [6](https://arxiv.org/html/2510.18439v1#A2.F6) at the token level, counterfactual signals highlight that content words (e.g., *freundlich*, *im Nord*) depend strongly on video, while function words remain largely language-driven; Figure [7](https://arxiv.org/html/2510.18439v1#A2.F7) at the sentence level, lower reliability correlates with higher hallucination rates. Thus, reliability provides an explicit, quantifiable estimate of grounding confirming reliability as a direct estimate of grounding.

Figure: Figure 6: Token-level reliability. Content words show strong video dependence, whereas function words rely more on language priors.
Refer to caption: iclr2026/figures/token_reliability.jpg

Figure: Figure 7: Sentence-level reliability vs. hallucination (CHAIR). Lower reliability correlates with higher hallucination.
Refer to caption: iclr2026/figures/overview_scatter_centered_phoenix.png

## Appendix C Symbolic proof: gloss-free supervision raises hallucinations via weak visual use

We formalize the claim that *gloss-free* training increases hallucinations in SLT by increasing the prevalence of weak visual use.

#### Random variables.

At a decoding step $t$, let $H\in\{0,1\}$ indicate hallucination, $W\in\{0,1\}$ indicate weak visual use (e.g., $r_{t}$ less than a threshold $\tau$), and $GF\in\{0,1\}$ indicate gloss-free training.

#### Assumptions.

- (A1)
$\Pr(H{=}1)>0$ (non-triviality)
- (A2)
$\Pr(H{=}1\mid W{=}1)>\Pr(H{=}1\mid W{=}0)$ (weakness raises hallucination)
- (A3)
$\Pr(W{=}1\mid GF{=}1)>\Pr(W{=}1\mid GF{=}0)$ (gloss-free raises weakness)
- (A4)
$H\perp\!\!\!\perp GF\mid W$ (mediation through $W$)

###### Proposition 1 (Transitivity via a mediator) .

Under (A1)–(A4),

$$ $\displaystyle\Pr(H{=}1\mid GF{=}1)-\Pr(H{=}1\mid GF{=}0)$ $\displaystyle=\Big(\Pr(H{=}1\mid W{=}1)-\Pr(H{=}1\mid W{=}0)\Big)$ (1) $\displaystyle\quad\times\Big(\Pr(W{=}1\mid GF{=}1)-\Pr(W{=}1\mid GF{=}0)\Big)$ $\displaystyle>0.$ $$

###### Proof.

By the law of total probability and (A4),

$$ $\displaystyle\Pr(H{=}1\mid GF{=}g)$ $\displaystyle=\sum_{w\in\{0,1\}}\Pr(H{=}1\mid W{=}w)\,\Pr(W{=}w\mid GF{=}g).$ $$

Taking the difference between $g{=}1$ and $g{=}0$ and using $\Pr(W{=}0\mid GF{=}g)=1-\Pr(W{=}1\mid GF{=}g)$ yields

$$ $\displaystyle\Delta_{H}$ $\displaystyle=\Big(\Pr(H{=}1\mid W{=}1)-\Pr(H{=}1\mid W{=}0)\Big)\,\Big(\Pr(W{=}1\mid GF{=}1)-\Pr(W{=}1\mid GF{=}0)\Big)\;>\;0.$ $$

∎

###### Proposition 1 (Transitivity via a mediator) .

###### Proof.

#### Continuous variant.

Let $R\in[0,1]$ be *reliability*. If the mapping into probability $m(r):=\Pr(H{=}1\mid R{=}r)$ is non-increasing, $R\mid GF{=}1$ is first-order stochastically dominated by $R\mid GF{=}0$, and $H\perp\!\!\!\perp GF\mid R$, then $\mathbb{E}[m(R)\mid GF{=}1]\geq\mathbb{E}[m(R)\mid GF{=}0]$ with strict inequality under strict dominance.

## Appendix D Algorithms

We estimate reliability by comparing the model’s predictions with and without visual input.
Three passes are used: the true video (video), no visual encoder (no-video), and a shuffled or unrelated one (mismatched-video).
From these, we derive feature-based signals (changes in hidden states and attention) and
counterfactual signals (differences in token probabilities and logits).
These are fused through a small logistic layer to obtain per-token reliabilities $r_{t}$. Early pooling (mean, tail-$q$, harmonic, min) yields sentence-level scores $\mathcal{R}$, indicating how much each translation relies on visual evidence.

Figure: Algorithm 1 ReliabilityFromVideo: full computation of sentence-level reliability

## Appendix E Extra plots

In Figure [8](https://arxiv.org/html/2510.18439v1#A5.F8), we present the complete matrix illustrating the cross-transfer gap across datasets, models, and cross-both configurations. Overall, the performance drop remains relatively small; however, reliability weights trained gloss-based (GB) model outputs tend to generalize less effectively, likely due to reduced exposure to more informative samples and a higher tendency toward hallucinations.

Figure: Figure 8: Full cross transfer gap.
Refer to caption: iclr2026/figures/transfer_spearman_strict_iclr_gap_4.png

Figure [9](https://arxiv.org/html/2510.18439v1#A5.F9) summarizes the correlation performance of our visual signals relative to the strongest baseline for each metric. Similar to the detection results, our methods exhibit a stronger correlation with hallucinations when used as individual signals, and when combined with the text-based (uncertainty) baselines, they further enhance overall performance.

Figure: Figure 9: Radar visualization of correlation metrics (Pearson, Spearman, and Isotonic (ISO)) between CHAIR and reliability scores ($R_{\mathrm{tail}\text{-}10}$), normalized to the $[0,1]$ range.
Refer to caption: iclr2026/figures/radar_correlation_metrics.png

## Appendix F Implementation details

### F.1 Generics

We compute clean, no-video, and mismatched runs jointly in a single batched forward pass with shared key–value caches, with mismatches generated by in-batch shuffling. Probabilities are accumulated in FP32 with $\varepsilon=10^{-12}$ for logarithms, while encoder–decoder blocks use AMP/FP16. Regression outputs are kept both raw and after sigmoid squashing. Cross-attention usage is normalized per sequence by mapping the 10/90% quantiles of mass to $[0,1]$. Sentence scores are obtained by pooling token-level reliabilities with mean, harmonic mean, tail-10%, exponential moving average ($\alpha=0.9$), or minimum. Calibration is applied per dataset/model/pool using either linear regression ($\hat{R}=\alpha R+\beta$) or isotonic regression against $1-\mathrm{CHAIR}$. Baselines (confidence, entropy, perplexity) are computed from the clean run; combined variants are also exported.

### F.2 CHAIR Computation and Language-Specific Preprocessing

To evaluate hallucinations we adopt the CHAIR metric (Rohrbach et al., 2018),
which measures the proportion of hallucinated content words in the prediction
relative to the reference. Since CHAIR relies on accurate identification of
content tokens, we applied different preprocessing pipelines for
German (PHOENIX-2014T data) and Chinese (CSL-Daily data) to account for linguistic differences.

#### German (alphabetic languages).

For German, we rely on spaCy for tokenization and
part-of-speech (PoS) tagging, keeping only tokens whose PoS belongs to a restricted
content set (*NOUN, PROPN, VERB, AUX, ADJ, ADV, NUM*).
Tokens matching a stopword list (e.g., *und, der, ist*) are discarded.
The remaining tokens are normalized by lemmatization and optionally stemmed
using the Snowball stemmer (German/English), ensuring that inflected
forms (e.g., *ginge*, *gegangen*) map to the same base form.
This reduces sparsity and makes CHAIR counts more consistent.

#### Chinese (logographic language).

For Chinese, we cannot rely on whitespace or inflectional morphology.
Instead, we: (i) normalize the script with OpenCC
(Traditional $\to$ Simplified) when needed;
(ii) normalize spacing around Chinese punctuation to avoid spurious tokens;
(iii) tokenize using jieba; and
(iv) apply a Chinese stopword list to remove frequent grammatical particles
and function words.
Since Chinese lacks inflectional morphology, no stemming is applied.
If spaCy’s pipeline is used, we additionally filter by PoS tags
(same content set as above).

#### CHAIR scoring.

Given prediction and reference content tokens, we compute per-sentence hallucination as:

- •
Instance-level CHAIR ($\text{CHAIR}_{I}$): fraction of predicted
content tokens that are absent or over-predicted compared to the reference.

The formulation from Rohrbach et al. (2018) is defined as:

$$ $\text{CHAIR}_{i}=\frac{\#\,\text{hallucinated object instances}}{\#\,\text{all mentioned object instances}}$ $$

where the object is here replaced by the content words.

This setup ensures that hallucination evaluation is
*linguistically adapted*: German tokens are normalized via
stemming/lemmatization, while Chinese tokens are segmented and filtered
with script and stopword normalization.
We use the instance-level CHAIR as label to train reliability linear weights.

## Appendix G Ablations

### G.1 Degradation under visual perturbations

As presented earlier in Fig. [5](https://arxiv.org/html/2510.18439v1#S4.F5), to assess whether reliability reflects visual usage, we systematically degraded the video input by adding Gaussian noise and randomly dropping frames. The reliability measure—trained via regression on $(1-\text{CHAIR})$—was then evaluated under these perturbations. Reliability was rescaled to $[0,1]$ for visualization, though correlations remain unaffected by this linear transformation.

Reliability consistently decreases as degradation increases, confirming its sensitivity to visual information, though slight non-monotonic fluctuations occur because it is optimized to *rank* hallucination risk rather than to yield calibrated probabilities. CHAIR and anti-BLEU-1 exhibit similar trends, with reliability following the same direction but with higher variance, consistent with its role as an anti-CHAIR ranking signal.

Overall results is presented in Tables [3](https://arxiv.org/html/2510.18439v1#A7.T3) and [4](https://arxiv.org/html/2510.18439v1#A7.T4).

**Table 3: Degradation on PHOENIX under frame dropping. Values are means over runs. Reliability and CHAIR are reported in raw scale; BLEU, BLEU-1, and chrF2 are on their standard scales.**
| $p$ | Reliability | CHAIR | BLEU | BLEU-1 | chrF2 |
| --- | --- | --- | --- | --- | --- |
| 0.01 | 0.37 | 0.50 | 23.58 | 50.92 | 45.88 |
| 0.10 | 0.44 | 0.51 | 23.49 | 51.87 | 45.48 |
| 0.20 | 0.43 | 0.50 | 22.23 | 52.87 | 43.94 |
| 0.30 | 0.34 | 0.52 | 19.95 | 51.83 | 41.40 |
| 0.40 | 0.39 | 0.55 | 16.80 | 50.04 | 37.38 |

**Table 4: Degradation on CSL under feature noise. Values are means over runs. Reliability and CHAIR are reported in raw scale; BLEU, BLEU-1, and chrF2 are on their standard scales.**
| $p$ | Reliability | CHAIR | BLEU | BLEU-1 | chrF2 |
| --- | --- | --- | --- | --- | --- |
| 0.00 | 0.41 | 0.53 | 17.84 | 47.26 | 21.14 |
| 0.10 | 0.36 | 0.57 | 14.41 | 41.72 | 18.40 |
| 0.20 | 0.38 | 0.60 | 12.13 | 37.75 | 16.47 |
| 0.30 | 0.39 | 0.63 | 9.44 | 34.13 | 14.32 |
| 0.40 | 0.36 | 0.66 | 7.51 | 30.92 | 12.58 |

### G.2 Feature importance and pooling

#### Feature weights.

To better understand which reliability signals drive hallucination detection, we analyze the learned weights of our logistic regression head (Fig. [10](https://arxiv.org/html/2510.18439v1#A7.F10)). The model assigns the largest positive weight to the counterfactual *log_margin*, confirming that probability differences between clean and perturbed video inputs are the most informative cue. Smaller but consistent contributions are observed for *logit-margin*, *prob-margin*, and *attention-difference*, whereas *hidden-state sensitivity (hs_prob)* receives a small negative weight. This suggests that hidden-state changes capture some grounding information, but they are less predictive than counterfactual margins. Delta-based features (*delta_clean*, *delta_mis*) contribute minimally once margin-based signals are included, indicating redundancy.

Figure: Figure 10: Learned feature weights for hallucination detection. Positive weights indicate stronger reliance of the classifier on the corresponding reliability signal.
Refer to caption: iclr2026/figures/detection_weight.png

#### Pooling strategies.

We further investigate the effect of *pooling* strategies across tokens (Table [5](https://arxiv.org/html/2510.18439v1#A7.T5)). We compare mean pooling ($R_{mean}$),
and tail10-pooling ($R_{\mathrm{tail}\text{-}10}$,
which focuses on the least reliable 10% tokens.

Across datasets and models, $R_{\mathrm{tail}\text{-}10}$ generally improves correlation with hallucination labels, especially for gloss-free (GF) models where weak grounding makes local unreliability highly informative. Gloss-based (GB) models also benefit, though the gap is smaller, indicating that tail pooling is particularly effective when hallucinations arise from localized failures of visual grounding.

**Table 5: Comparison of pooling strategies. Each cell shows $R_{mean}$ $|$ $R_{\mathrm{tail}\text{-}10}$. Tail pooling often yields stronger correlation with hallucinations, especially in gloss-free models.**
| Dataset | Model | Metrics ($R_{mean}$ $|$ $R_{\mathrm{tail}\text{-}10}$) |  |  |
| --- | --- | --- | --- | --- |
| Pearson | Spearman | ISO |  |  |
| CSL | GB | -0.439 $|$ -0.234 | -0.426 $|$ -0.238 | 0.465 $|$ 0.282 |
| CSL | GF | -0.672 $|$ -0.682 | -0.678 $|$ -0.644 | 0.718 $|$ 0.704 |
| PHOENIX | GB | -0.361 $|$ -0.458 | -0.368 $|$ -0.340 | 0.453 $|$ 0.493 |
| PHOENIX | GF | -0.564 $|$ -0.623 | -0.523 $|$ -0.590 | 0.596 $|$ 0.650 |

### G.3 Feature interactions with raw probabilities

Figure: Figure 11: Regression (left) and detection (right) feature weights with vs. without raw probabilities.
Refer to caption: iclr2026/figures/regression_only_plot_2.png

As shown in Figure [11](https://arxiv.org/html/2510.18439v1#A7.F11), adding raw probabilities $p_{\text{vid}}(y)$ yields a very strong but highly correlated predictor, which reduces the relative weight of complementary cues such as *hidden-state sensitivity*. This explains why grounding-based reliability remains competitive when used alone: it captures visual reliance without being dominated by probability mass. In detection, hidden-state sensitivity contributes more strongly when considered in isolation without the raw probability, whereas in regression text-related signals dominate, highlighting complementarity rather than redundancy between probability-driven and state-driven evidence.

## Appendix H Qualitative Error Analysis: Gloss-Free vs. Gloss-Based Models

We analyze a few overlapping references with outputs from both gloss-free (GF) and gloss-based (GB) models.

**Table 6: Comparative qualitative analysis of GF vs. GB translations. Errors are categorized using taxonomy of Vilar et al. (2006): Omission, Addition, and Incorrect word. Correct or near-equivalent paraphrases are marked as semantically correct.**
| Reference: am tag breiten sich die teilweise kräftigen schneefälle weiter aus. |
| --- |
| GF: zum teil breitet sich kräftiger schnee aus. |
| GB: dann kommen zum teil kräftige schneefälle auf. |
| Analysis:<br>GF shortens and generalises: “kräftiger Schnee” instead of “kräftige Schneefälle” (Incorrect word),<br>and drops “weiter aus” (Omission).<br>GB reformulates with “dann kommen … auf”, preserving semantics. Semantically correct. |
| Reference: und es kann auch örtlich zum teil auch gefährlich glatt werden denn vor allen dingen in der mitte und auch im süden wird es frostig. |
| GF: sonst gibt es auch eher gefährliche straßenglätte vor allen dingen in der mitte und im süden wird es deutlich kühler werden. |
| GB: und es besteht erhöhte glättegefahr vor allen dingen in der mitte und auch im süden. |
| Analysis:<br>GF paraphrases with “Straßenglätte” and adds “deutlich kühler werden” (Addition), losing the “örtlich” aspect (Omission).<br>GB compresses but retains core meaning “erhöhte Glättegefahr” $\rightarrow$ semantically adequate. |
| Reference: der wind weht schwach bis mäßig aus südosten. |
| GF: der wind weht schwach bis mäßig aus nord bis nordost. |
| GB: der wind weht schwach bis mäßig. |
| Analysis:<br>GF changes wind direction (“Südosten” $\rightarrow$ “Nord bis Nordost”) = Incorrect word.<br>GB omits wind direction entirely = Omission. Both distort crucial meteorological info. |
| Reference: ein ausläufer des tiefs über dem nordmeer lenkt allerdings von nordwesten wolkenfelder ins land. |
| GF: richtung westen machen sich tiefausläufer mit wolken bemerkbar. |
| GB: ein kleines tief über dem nordwesten lenkt wolken ins land. |
| Analysis:<br>GF simplifies but misplaces direction (“Richtung Westen” vs. “von Nordwesten”) = Incorrect word.<br>GB alters “Ausläufer des Tiefs über dem Nordmeer” $\rightarrow$ “kleines Tief über dem Nordwesten” (Incorrect word),<br>still partially correct but factual distortion. |
| Reference: dabei das wetter aber nicht unfreundlich höchstens mal im osten und auch im süden ein paar schauer. |
| GF: das milde und trockene wetter wird auch in der osthälfte und im süden ein bisschen schauer geben. |
| GB: am freundlichsten bleibt es noch im osten und süden hier und da schauer. |
| Analysis:<br>GF exaggerates with “mild und trocken” (Addition), partially shifts tone.<br>GB preserves intended contrast (friendly with occasional showers) $\rightarrow$ semantically correct. |
| Reference: am tag dreizehn grad bei dauerregen und einundzwanzig grad am oberrhein. |
| GF: am tag dreizehn grad an der ostsee und bis zu elf grad in oberfranken. |
| GB: am tag dreizehn grad im osten und einundzwanzig grad am oberrhein. |
| Analysis:<br>GF fabricates new regions (“Ostsee”, “Oberfranken”) $\rightarrow$ Incorrect word + Addition.<br>GB faithful to source with minor rephrasing $\rightarrow$ semantically correct. |
| Reference: der wind weht schwach bis mäßig an der nordsee und im bergland auch frischer wind. |
| GF: der wind weht schwach bis mäßig an der nordsee und auf den bergen teilweise auch frisch. |
| GB: der wind weht schwach bis mäßig an den küsten sowie aus süd bis südost. |
| Analysis:<br>GF good paraphrase with minor lexical variation (“auf den Bergen” vs. “im Bergland”) $\rightarrow$ semantically correct.<br>GB introduces new direction “aus Süd bis Südost” Addition + Incorrect word. |
| Reference: auch im westen gibt es später wolkenlücken. |
| GF: im osten und westen mal dichtere wolken mal längere sonnenschein. |
| GB: das hoch das sich von westen bis in die mitte ausdehnt sorgt morgen verbreitet für sonnenschein. |
| Analysis:<br>GF alters polarity: instead of “Wolkenlücken” it adds “mal dichtere Wolken” Incorrect word.<br>GB hallucinates “Hochdruckgebiet” and causal explanation Addition,<br>but preserves sunny outcome $\rightarrow$ partially correct. |

### H.1 General Discussion

The qualitative analysis presented in Table [6](https://arxiv.org/html/2510.18439v1#A8.T6) shows outputs that differ lexically or stylistically from the reference may still be semantically correct, while fluent
but misleading sentences can conceal omissions, additions, or substitutions that critically alter meaning. Across cases, GF tends to produce translations that stay structurally close to the reference. Its errors are
typically subtle, involving small lexical shifts or added modifiers, yet these can still distort key meteorological
facts such as temperatures, locations, or weather conditions. GB, on the other hand, often generates more fluent and natural-sounding outputs. Omissions and hallucinated details are
common, occasionally leading to forecasts that are polished in form yet misleading in content.

### H.2 Qualitative: Prompt injection (extended)

This section provides an extended qualitative analysis of model behavior under adversarially injected start tokens. It includes a comprehensive set of representative samples covering multiple injected prompts and both decoding configurations. The examples complement the discussion in the main text, illustrating the systematic contrast between gloss-free (GF) presented in Table [7](https://arxiv.org/html/2510.18439v1#A8.T7) and gloss-based (GB) models presented in Table [8](https://arxiv.org/html/2510.18439v1#A8.T8): GF models often continue the injected prefix, producing linguistically fluent yet visually inconsistent continuations, whereas GB models, aided by explicit visual supervision, tend to suppress the injected bias and quickly realign with the visual evidence.

Column descriptions:

- •
Prompt: the injected adversarial prefix;
- •
Decoding: the generation strategy used (beam or greedy);
- •
Fooled: whether the model followed the injected prefix instead of the visual evidence;
- •
Prediction: the model’s generated translation;
- •
Reference: the corresponding ground-truth sentence.

**Table 7: Nine representative prompt-injection cases for the GF (gloss-free) model with English translations (including prompt meaning) shown in italic below each example.**
| Prompt | Decoding | Fooled | Prediction | Reference |
| --- | --- | --- | --- | --- |
| am tag | beam | Yes | am tag lässt der regen und schneefall an den alpen nach vor allem im osten und norden fällt noch etwas regen sonst ist es teilweise klar. | regen und schnee lassen an den alpen in der nacht nach im norden und nordosten fallen hier und da schauer sonst ist das klar. |
| (“in the morning / during the day”) |  |  | (During the day, rain and snowfall over the Alps subside; mainly in the east and north a bit of rain still falls, otherwise it is partly clear.) | (Rain and snow subside over the Alps during the night; in the north and northeast, occasional showers fall, otherwise it is clear.) |
| am tag | beam | Yes | am tag regnet es an den küsten etwas sonst mal sonne mal wolken vor allem in der mitte am freitag wechselhaftes wetter. | am donnerstag regen in der nordhälfte in der südhälfte mal sonne mal wolken ähnliches wetter dann auch am freitag. |
| (“in the morning / during the day”) |  |  | (During the day, some rain on the coasts, otherwise sun and clouds alternating, especially in the center; on Friday, changeable weather.) | (On Thursday, rain in the northern half; in the south, sun and clouds alternating, similar weather also on Friday.) |
| am tag | beam | Yes | am tag fließt von norden kühlere luft heran die für teilweise kräftige schneefälle und teilweise ergiebige regenfälle sorgt. | vom nordmeer zieht ein kräftiges tief heran und bringt uns ab den morgenstunden heftige schneefälle zum teil auch gefrierenden regen. |
| (“in the morning / during the day”) |  |  | (During the day, cooler air flows in from the north, causing heavy snowfall and rain in places.) | (A strong low moves in from the North Sea, bringing heavy snowfall and some freezing rain from the morning hours.) |
| am tag | greedy | Yes | am tag lässt der regen und schneefall an den alpen nach im osten und norden regnet es noch kräftig sonst ist es teilweise klar. | regen und schnee lassen an den alpen in der nacht nach im norden und nordosten fallen hier und da schauer sonst ist das klar. |
| (“in the morning / during the day”) |  |  | (During the day, rain and snow over the Alps subside; in the east and north it still rains heavily, otherwise partly clear.) | (Rain and snow subside over the Alps during the night; in the north and northeast, occasional showers, otherwise clear.) |
| am tag | greedy | Yes | am tag regnet es an den küsten etwas sonst mal sonne mal wolken und nur vereinzelt ein paar tropfen am freitag auch mal weniger. | am donnerstag regen in der nordhälfte in der südhälfte mal sonne mal wolken ähnliches wetter dann auch am freitag. |
| (“in the morning / during the day”) |  |  | (During the day, some rain on the coasts, otherwise sun and clouds with only a few drops; on Friday, less.) | (On Thursday, rain in the northern half; in the south, sun and clouds alternating, similar weather also on Friday.) |
| am tag | greedy | Yes | am tag überquert norddeutschland ein tief mit kräftigem schneefall und teilweise kräftigem regen. | vom nordmeer zieht ein kräftiges tief heran und bringt uns ab den morgenstunden heftige schneefälle zum teil auch gefrierenden regen. |
| (“in the morning / during the day”) |  |  | (During the day, a low crosses northern Germany with heavy snowfall and rain in parts.) | (A strong low moves in from the North Sea, bringing heavy snowfall and some freezing rain from the morning hours.) |
| ab den | beam | Yes | ab den nachmittagsstunden lässt der regen oder schnee an den alpen nach im osten und norden fällt noch etwas regen sonst ist es teilweise klar. | regen und schnee lassen an den alpen in der nacht nach im norden und nordosten fallen hier und da schauer sonst ist das klar. |
| (“from the ”) |  |  | (From the afternoon hours, rain or snow over the Alps subside; in the east and north, some rain still falls, otherwise partly clear.) | (Rain and snow subside over the Alps during the night; in the north and northeast, occasional showers, otherwise clear.) |
| ab den | beam | Yes | ab den donnerstags regnet es an den küsten mal sonne mal wolken vorsicht wegen glätte. | am donnerstag regen in der nordhälfte in der südhälfte mal sonne mal wolken ähnliches wetter dann auch am freitag. |
| (“from the ”) |  |  | (From Thursday, rain on the coasts, alternating sun and clouds; caution due to slipperiness.) | (On Thursday, rain in the northern half; in the south, sun and clouds alternating, similar weather also on Friday.) |
| ab den | beam | No | ab den nachmittagsstunden erreicht uns dann ein tief mit teilweise kräftigem schneefall und teilweise gefrierenden regen. | vom nordmeer zieht ein kräftiges tief heran und bringt uns ab den morgenstunden heftige schneefälle zum teil auch gefrierenden regen. |
| (“from the ”) |  |  | (From the afternoon hours, a low reaches us bringing heavy snowfall and partly freezing rain.) | (A strong low moves in from the North Sea, bringing heavy snowfall and some freezing rain from the morning hours.) |

**Table 8: Nine representative prompt-injection cases for the GB (gloss-based) model with English translations (including prompt meaning) shown in italic below each example.**
| Prompt | Decoding | Fooled | Prediction | Reference |
| --- | --- | --- | --- | --- |
| am tag | beam | No | am tag tiefer luftdruck bestimmt unser wetter . | tiefer luftdruck bestimmt in den nächsten tagen unser wetter . |
| (“in the morning / during the day”) |  |  | (During the day, low pressure determines our weather.) | (Low pressure will determine our weather in the coming days.) |
| am tag | beam | No | am tag dazu gibt es viele wolken die teilweise kräftige schauer und kräftige schauer vereinzelt auch gewitter bringen . | das bedeutet viele wolken und immer wieder zum teil kräftige schauer und gewitter . |
| (“in the morning / during the day”) |  |  | (During the day, there are many clouds bringing partly heavy showers and isolated thunderstorms.) | (That means many clouds and repeatedly partly heavy showers and thunderstorms.) |
| am tag | beam | No | am tag schwacher bis mäßiger an der see teilweise frischer wind aus süd bis südwest im nordwesten starke bis stürmische böen bei gewittern sturmböen . | meist weht nur ein schwacher wind aus unterschiedlichen richtungen der bei schauern und gewittern stark böig sein kann . |
| (“in the morning / during the day”) |  |  | (During the day, weak to moderate, at the sea partly fresh wind from south to southwest; in the northwest, strong to stormy gusts and storm gusts in thunderstorms.) | (Mostly only a weak wind blows from varying directions, which can become very gusty during showers and thunderstorms.) |
| am tag | greedy | No | am tag tiefer luftdruck bestimmt unser wetter . | tiefer luftdruck bestimmt in den nächsten tagen unser wetter . |
| (“in the morning / during the day”) |  |  | (During the day, low pressure determines our weather.) | (Low pressure will determine our weather in the coming days.) |
| am tag | greedy | No | am tag dazu gibt es viele wolken die teilweise kräftige schauer und kräftige schauer vereinzelt auch gewitter bringen . | das bedeutet viele wolken und immer wieder zum teil kräftige schauer und gewitter . |
| (“in the morning / during the day”) |  |  | (During the day, there are many clouds bringing partly heavy showers and isolated thunderstorms.) | (That means many clouds and repeatedly partly heavy showers and thunderstorms.) |
| am tag | greedy | No | am tag der wind weht schwach bis mäßig an der see auch frisch aus süd bis südwest im nordwesten bei gewittern sind stürmische böen möglich . | meist weht nur ein schwacher wind aus unterschiedlichen richtungen der bei schauern und gewittern stark böig sein kann . |
| (“in the morning / during the day”) |  |  | (During the day, the wind blows weak to moderate, at the sea also fresh from south to southwest; in the northwest, stormy gusts possible in thunderstorms.) | (Mostly only a weak wind blows from varying directions, which can become very gusty during showers and thunderstorms.) |
| ab den | beam | Yes | ab den regen und schneefälle lassen die alpen im laufe des tages nach im nordosten regnet es mitunter noch kräftig sonst zeigen sich auch die sterne . | regen und schnee lassen an den alpen in der nacht nach im norden und nordosten fallen hier und da schauer sonst ist das klar . |
| (“from the ”) |  |  | (During the day, rain and snowfall over the Alps subside; in the northeast it still rains heavily at times, otherwise the stars are visible.) | (Rain and snow subside over the Alps during the night; in the north and northeast, occasional showers, otherwise clear.) |
| ab den | beam | Yes | ab den am donnerstag regnet es in der nordwesthälfte gebietsweise anfangs an den küsten mal sonne mal wolken und im wechsel gibt es dann am freitag ähnliches we | am donnerstag regen in der nordhälfte in der südhälfte mal sonne mal wolken ähnliches wetter dann auch am freitag . |
| (“from the ”) |  |  | (From Thursday, rain in the northwestern half, partly initially on the coasts alternating sun and clouds; similar weather continues on Friday.) | (On Thursday, rain in the northern half; in the south, sun and clouds alternating, similar weather also on Friday.) |
| ab den | beam | No | ab den ein kräftiges tief über der nordsee bringt uns ab morgen früh teilweise kräftige schneefälle und gefrierenden regen . | vom nordmeer zieht ein kräftiges tief heran und bringt uns ab den morgenstunden heftige schneefälle zum teil auch gefrierenden regen . |
| (“from the ”) |  |  | (From a strong low over the North Sea brings us from early tomorrow partly heavy snowfall and freezing rain.) | (A strong low moves in from the North Sea, bringing heavy snowfall and some freezing rain from the morning hours.) |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Performance Impact on Gloss-Free SLT:
- The paper positions reliability as a diagnostic tool, yet provides no evidence that it leads to tangible improvements in gloss-free sign language translation. Can the authors demonstrate substantial gains (e.g., ≥1.0 BLEU)?  

Cross-Architecture Validation:
- Claims of model-agnosticism are unsubstantiated. Can the authors apply the reliability score to ≥3 diverse gloss-free architectures (e.g., SignBT, STMC, BLT, or non-autoregressive variants) and show:  Consistent hallucination prediction (correlation, AUC)?
soundness: 3
strengths: -  Precisely identifies hallucination in SLT—especially in gloss-free settings—as a critical, under-addressed issue tied to visual grounding weakness.

- The token-level reliability score is architecture-independent, requiring no reference translations, making it broadly applicable across existing and future SLT systems.

- Combines encoder sensitivity (input robustness) and counterfactual masking (internal consistency) into a well-justified, interpretable proxy for visual grounding.

- Establishes a reusable evaluation tool that can become a standard metric in SLT research and development.
summary: The paper introduces a token-level reliability metric for hallucination detection in sign language translation (SLT), combining encoder sensitivity to input perturbations and internal state changes under counterfactual video masking. The metric is evaluated on PHOENIX-2014T and CSL-Daily using both gloss-based and gloss-free models.
weaknesses: - Lacks testing on larger, more diverse, or continuous SLT benchmarks (e.g., How2Sign, OpenASL).  
- Tested on a small set of architectures (primarily transformer-based). 
- Unclear whether the proposed reliability score outperforms or complements prior art.

---

confidence: 5
contribution: 2
initial_rating: 2
presentation: 2
questions: - As mentioned in the weaknesses, I doubt the necessity of introducing the concept of hallucination in SLT, as it may mislead future research. However, I remain open to opinions and experimental evidence that could demonstrate the significance of hallucination in SLT.
- I also encourage the author to improve the writing and organization of this manuscript, and provide more valuable take home messages.
soundness: 1
strengths: - S1. This manuscript is well-motivated, addressing the crucial issue of evaluating and improving the utilization of visual information in sign language translation (SLT), a topic that has garnered significant attention in recent years. The manuscript aims to design an indicator to shed light on this challenge.
- S2. The manuscript provides a comprehensive set of indicators for assessing the utilization of visual information, considering both the feature space and the output space.
- S3. The manuscript conducts thoughtful and thorough experiments on hallucination evaluation and detection.
summary: This manuscript aims to evaluate "hallucination" in sign language translation (SLT), defined as the extent to which the decoder generates token-level predictions without relying on visual information. To assess hallucination in SLT accurately, the manuscript proposes feature-wise signals, such as changes in feature orientation and intensity shifts in attention. Additionally, it leverages prediction discrepancies under noisy conditions to evaluate hallucination. Extensive experiments demonstrate that the proposed reliability indicator correlates with the "CHAIR" phenomenon in SLT, achieving impressive detection performance.
weaknesses: - W1. The definition of hallucination in SLT is unclear. While hallucination in visual-language models (VLMs) is introduced due to the ambiguity in input range and output space, SLT has clearly defined output results and visual-language correspondences. The errors in hallucination can be directly measured using metrics like word error rate (WER) or similar set prediction metrics. Therefore, the introduction of the hallucination concept in SLT seems unnecessary.
- W2. As a translation task, many words are predicted based on contextual information. What we are truly concerned with are the words that can only be predicted by visual information. However, this manuscript does not address the distinction between different types of tokens in this context.
- W3. The insufficient use of visual information is a well-known issue in the sign language understanding community, and addressing this challenge is critical. While this manuscript attempts to detect hallucination, it does not propose any solutions for resolving this problem.
- W4. The practical applicability of the proposed method is limited. As shown in Figure 1, the proposed metric shows some correlation with "CHAIR" in SLT, which is used to demonstrate the effectiveness of the indicator. However, why not directly use "CHAIR" as the hallucination indicator? Furthermore, the high variation in the proposed metric suggests its limited practical value.
- W5. While Table 1 demonstrates that the proposed method achieves high hallucination detection accuracy, the process for obtaining ground-truth labels for detection remains unclear.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1.The authors provide the results on two SLT models. I suggest that the authors provide results on more open-source models. Additionally, the current method only considers cases where the input is video. Could it measures the different inputs, like poses?

2.All formulas in the manuscript lack recommended numbering.

3.There are notations in the equations lack descriptions. 

In Line 154, $\pi^{-1}$ lack descriptions. Which part of the model does “hidden state” represent?

In Line 164, $scale$ lack descriptions. For $s_t^{attn}$, does a large result also indicate higher reliability? Are the set “video” and “masked” same?

4.In the 4 note, the authors provide a brief description about the calculation method of CHAIR. I think the authors should provide a more detailed description.
soundness: 3
strengths: It is the first work to investigate hallucination in sign language. The authors provide a novel metric to measure the reliability for SLT based on the visual input and text output. They also demonstrate the proposed reliability surpasses text-only baselines in detecting hallucinations.
summary: This work investigates hallucination in sign language translation. The authors introduce a novel metric named reliability to measure the hallucination in SLT. The propose metric is based on two parts, i.e., internal model changes and output probability shifts. By adopting the CHAIR as the annotation on sentence-level, the proposed metric shows encouraging performance based on two SLT models on both benchmarks, i.e., PHOENIX-2014T and CSL-Daily.
weaknesses: 1.The authors did not provide a clear analysis of the relationships about the three key concepts, i.e., sign language translation performance, the rate of hallucination and the proposed metric.

2.The key steps of the proposed metric is missing, such as how to calculate the weights. The appendix is also not completed, which is confusing. For specific issues, please refer to the descriptions in the Question section.

3.The authors claim that the hallucinations significantly impact the performance of SLT. However, there lacks data showing how hallucinations hurt translation quality. There is only limited comparative examples are given in the appendix, which is insufficient to support the main contributions.

4.Since the goal of mitigating hallucinations is to improve sign language translation performance, I think the authors should clarify the specific application directions of the proposed metric in current practical contexts. 

5.The authors attribute the main reason of hallucinations to the poor performance of visual encoder. I think this point has been observed in the previous study [1]. It enhances visual representation capabilities by incorporating small pre-trained language models. Please provide more details about how to distinguish it from existing research in the manuscript.

[1] Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. 2024. Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7071–7081, Torino, Italia. ELRA and ICCL.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: - If we remove the subclause from the first line of your abstract, it reads "Hallucination is particularly critical in SLT." That is not the point you are trying to make, so please rephrase this.
- I recommend adding a statement about the dangers of using glosses for sign language translation at the end of the first paragraph of your introduction. They should not be used for sign language translation purposes: see [1]. This will further strengthen your paper since you are tackling gloss-free SLT.
- Does your system have merit in domains other than SLT to detect hallucinations?
soundness: 3
strengths: - The paper is, for the most part, well-written and pleasant to read.
- The paper is well situated in related work.
- This is important work, as it improves the reliability of SLT models, especially gloss-free ones (which is critical because glosses can be considered harmful for SLT.)
summary: This paper tackles the problem of hallucination in gloss-free sign language translation. SLT is a challenging task, which has been facilitated by the use of large language models (LLMs) and vision language models (VLMs) in recent years. However, these models tend to hallucinate. Gloss-level supervision can reduce hallucinations, by grounding the model with sign-level supervision, but glosses are expensive to annotate and label (note from reviewer: and introduce other problems). The authors of this paper propose a novel method to detect hallucinations and ground models in visual inputs, which reduces hallucinations and improves the results on gloss-free sign language translation.
weaknesses: - The paper contains acronyms that are not defined, e.g., AP.
- The methodology is rather math heavy and can be challenging to follow at times. It would benefit from some intuitive explanations of the equations.
- There is no statement on whether the signing community was consulted for this work. The authors should make a clear statement on their hearing/signing status.
- Appendix C and D are empty (probably because they contain only figures.) This layout problem should be addressed.


Decision: Accept (Poster)