ANHALTEN: Cross-Lingual Transfer for German Token-Level Reference-Free Hallucination Detection (2024)

Abstract: Research on token-level reference-free hallucination detection has predominantly focused on English, primarily due to the scarcity of robust datasets in other languages. This has hindered systematic investigations into the effectiveness of cross-lingual transfer for this important NLP application. To address this gap, we introduce ANHALTEN, a new evaluation dataset that extends the English hallucination detection dataset to German. To the best of our knowledge, this is the first work that explores crosslingual transfer for token-level reference-free hallucination detection. ANHALTEN contains gold annotations in German that are parallel (i.e., directly comparable to the original English instances). We benchmark several prominent cross-lingual transfer approaches, demonstrating that larger context length leads to better hallucination detection in German, even without succeeding context. Importantly, we show that the sample-efficient few-shot transfer is the most effective approach in most setups. This highlights the practical benefits of minimal annotation effort in the target language for reference-free hallucination detection. Aiming to catalyze future research on cross-lingual token-level reference-free hallucination detection, we make ANHALTEN publicly available: https://github.com/janekh24/anhalten

heading: 1 Introduction
text: Detecting hallucinations in large pretrained language models (e.g., Brown et al., 2020; Jiang et al., 2024) is critical for ensuring their reliability in realworld applications. Most existing hallucination detection benchmarks focus on reference-based tasks (e.g., summarization, machine translation, question answering) (Maynez et al., 2020; Rebuffel et al., 2022; Sadat et al., 2023), comparing model generated text against provided references. However, reference-based hallucination detection is not appropriate for free-form text generation, where
obtaining ground-truth references in real-time demands sufficient and accurate preceding retrieval step. To address these challenges, reference-free hallucination detection approaches have been introduced (Liu et al., 2022; Su et al., 2024), focusing on identifying inconsistencies within the generated context itself to effectively detect hallucinations in real-time. Besides, most research in hallucination detection has concentrated on sentence or passage-level (Dhingra et al., 2019; Manakul et al., 2023; Zhang et al., 2023), which is inadequate for real-time applications that require immediate feedback during text generation. Fine-grained, tokenlevel reference-free hallucination detection benchmark is necessary for this purpose. However, research in this area has focused on English (Liu et al., 2022), primarily due to the lack of robust evaluation datasets in other languages. Creating token-level hallucination detection datasets for new languages (from scratch or using machine translation) is significantly more expensive and timeconsuming than for most other NLP tasks, due to the need for accurate translation and adaptation of nuanced contexts and token-level annotations. The lack of multilingual evaluation benchmarks hinders the investigation of cross-lingual transfer approaches for token-level reference-free hallucination detection.
In this work, we target this gap and introduce ANHALTEN (germAN HALucinaTion dEtectioN), a new benchmark derived from the English tokenlevel reference-free hallucination detection dataset HADES (Liu et al., 2022). ANHALTEN is: (1) reliable – with complete texts and hallucination spans (i.e., labels) manually translated, and (2) parallel – the same set of texts and labels have been translated to German, enabling direct comparison of multilingual models and cross-lingual transfer approaches.
We then use ANHALTEN to benchmark a range of cross-lingual transfer approaches and simulate the real-world applications in multiple setups. Our
186
results show that (i) hallucination detection works comparably well even without succeeding texts, indicating that larger context length helps detect hallucinations in German, thus supporting proactive hallucination prevention on-the-fly during text generation, and (ii) few-shot transfer methods achieve high performance with minimal annotated data, highlighting the practical benefits of inexpensive annotation of a handful of target-language hallucination instances for training detection models.

heading: 2 Methodology

heading: 2.1 Dataset Creation
text: We translate the full development set and 10% of the training set of English HADES dataset (Liu et al., 2022) in German, with 1,000 and 876 instances, respectively.1 Each instance includes a TEXT, MARKED WORD SPANS, POSITION OF MARKED WORD SPANS, and LABEL to indicate whether the MARKED WORD SPANS causes hallucination. Examples compared to the original English HADES dataset are shown in Table 1.
1Since the original test set labels were not published, we rely on training and development sets throughout our experiments. We also ensure the subsample of the training set retains the original label ratio of the training data.
Following the well-established practice (Hung et al., 2022; Senel et al., 2024), we carried out a two-phase translation process: (1) we started with an automatic translation – followed by (2) the manual post-editing of the translations. We first automatically translate the development and training set portions for both TEXT and MARKED WORD SPANS relying on DeepL Translator. We then incorporate native speaker with University degree and fluent in English, to post-edit the automatic translations to ensure the correctness of the translation – especially the directly preceding and succeeding context, and the correct determination of the MARKED WORD SPANS. Common errors identified in machine-translated texts include incorrect translations, missing words, grammatical mistakes, or contextual inaccuracies. Examples comparing the original English HADES with the automatically translated and manually post-edited texts are shown in Table 2. Besides, as the position of MARKED WORD SPANS changes in the German text2, the
2German and English, both Germanic languages, differ in ways that impact dataset design. In German, compound words are written as single words, whereas in English, they are separated by spaces, affecting MARKED WORD SPANS. Additionally, German commonly uses particle verbs, where MARKED WORD SPANS are split by other parts of the sentence.
POSITION OF MARKED WORD SPANS is adjusted accordingly.
Additionally, to conduct Translate-Train experiment for cross-lingual transfer, full training set (8,754 instances) are automatically translated using DeepL Translator, without post-editing. However, only 6,344 instances (72.5%) remain, since the discarded ones contain incorrect MARKED WORD SPANS.

heading: 2.2 Downstream Cross-Lingual Transfer
text: The parallel nature and substantial size of ANHALTEN facilitate benchmarking of cross-lingual transfer methods for hallucination detection tasks. We investigate three common methods for downstream cross-lingual transfer (XLT) (Ebing and Glavaš, 2023; Senel et al., 2024): (1) Zero-Shot Transfer, where we assume the absence of labeled task instances in the target language. The model is trained exclusively in English and is expected to perform the task directly in German without prior exposure to German labeled data. This method relies on the model’s capability to generalize knowledge from English to German. (2) Few-Shot Transfer, where a limited number of labeled instances in the target language exist with the majority of training data in the source language. The model is trained on abundant English data and a small amount of German data jointly,3 helping it adapt to the specific nuances of the German language with limited annotated data. (3) Translate-Train, where training instances in source language are automatically translated (i.e., noisy) to target language leveraging the state-of-the-art machine translation model. While this approach relies on the quality of translation, it benefits from creating a substantial amount of training data in German, closely approximating a fully supervised learning scenario.
To facilitate modular and efficient XLT, adapterbased approach is proposed to learn specialized task and language adapters for high portability and parameter-efficient transfer to various tasks and languages (Pfeiffer et al., 2020b). For downstream XLT, a task adapter is stacked on the pre-trained source language adapter, where the parameters are only updated for the task adapter. During evaluation, the source language adapter is replaced by the
In such cases, only the conjugable main part of the verb is marked, while the particle is ignored.
3Compared to sequential fine-tuning (Lauscher et al., 2020; Hung et al., 2022), joint fine-tuning (Schmidt et al., 2022) on instances in both source and target language can achieve better performances with higher stability.
pre-trained target language adapter. In our setup, the task adapter is trained by (1) the English-only data for Zero-Shot Transfer; (2) a joint training of English and a small portion of German data for Few-Shot Transfer; or (3) the machine-translated English-to-German data for Translate-Train. The adapter-based approach ensures that the model can efficiently adapt to new tasks with minimal parameter updates, maintaining the balance between performance and computational efficiency.

heading: 3 Experimental Setup

heading: 3.1 Evaluation Tasks and Measures
text: We evaluate multilingual pre-trained language models (PLMs) in XLT methods (§ 2.2) for token-level reference-free hallucination detection tasks. To simulate real-world applications, we evaluate on two sub-tasks: offline and online (Liu et al., 2022). In the offline setting, the model accesses both preceding and succeeding contexts of the MARKED WORD SPANS, suitable for detecting hallucinations in pre-generated texts. In the online setting, the model considers only the preceding context, enabling proactive prevention of hallucinations during on-the-fly text generation.
We follow Liu et al. (2022) and evaluate the XLT capabilities utilizing multilingual PLMs on hallucination detection tasks. The evaluation metrics include accuracy, precision, recall, F1, Area Under Curve (AUC), G-Mean (Espíndola and Ebecken, 2005), and Brier Score (BS) (Brier, 1950). These metrics provide a comprehensive evaluation of model performance, balancing correctness, and the ability to handle imbalanced classes.

heading: 3.2 Models and Experimental Setup
text: Experiments are conducted on multilingual PLMs, namely multilingual BERT (mBERT) (Devlin et al., 2019) and XLMR (Conneau et al., 2020),4 using language adapters5 proposed by Pfeiffer et al. (2020b) to facilitate modular and efficient XLT.
To evaluate downstream XLT, the experiments are conducted with 5 runs in both offline and online settings, with a fixed context window of 200 tokens. In the online setting, the context includes the 200 tokens preceding the MARKED WORD SPANS. In the offline setting, it includes 100 tokens before and
4The weights of PLMs are loaded from HuggingFace: multilingual-bert-base-cased and xlm-roberta-base.
5The pre-trained adapters are selected from AdapterHub (Pfeiffer et al., 2020a) for English (en-wiki@ukp) and German (de-wiki@ukp).
Not Hallucination Hallucination

heading: Model Setting Acc. ↑ G-Mean ↑ BS ↓ AUC ↑ P ↑ R ↑ F1 ↑ P ↑ R ↑ F1 ↑
text: mBERT offline 61.00 56.12 26.49 69.66 71.04 42.92 50.68 57.89 80.04 66.54 XLMR offline 62.82 59.38 25.51 72.90 74.89 41.87 53.37 58.18 84.89 68.96
mBERT online 60.44 55.34 26.71 67.81 73.47 36.69 48.10 56.33 85.46 67.76 XLMR online 63.70 62.01 24.14 72.44 71.51 48.85 57.76 59.72 81.14 68.02
Table 4: Zero-shot transfer results (%) averaged over 5 runs. Reference English performance of XLMR for accuracy: 70.40% (offline), 68.80% (online).
after the MARKED WORD SPANS.6 During training, the instances are randomly split into a 70/30 train and validation split, while the original label ratio of training data is retained for the split. We train for 10 epochs in batches of 8 instances, with learning rate 5 · 10−3, and a dropout ratio 0.2 is set to avoid overfitting.

heading: 4 Results and Discussions
text: We present and discuss the downstream XLT results on ANHALTEN for the token-level reference-free hallucination detection task across three XLT setups (§ 2.2): zero-shot transfer, few-shot transfer, and translate-train.
Zero-Shot Transfer. The results summarized in Table 4 highlight the performance of zero-shot transfer. Notably, XLMR consistently outperforms mBERT across most metrics, indicating that XLMR is better suited for zero-shot transfer. Minimal performance differences between the online and offline settings suggest that the selection of
6Liu et al. (2022) observed that model performance for English HADES dataset stabilizes around 80 tokens, with minimal performance differences between offline and online settings regarding context length. Thus, using 200 tokens would not limit performance, and increasing the context is unlikely to improve results.
large context windows does not significantly impact performance, aligning with findings from Liu et al. (2022). Having only preceding text with larger context lengths aids in detecting hallucinations, which is valuable for real-world applications, especially for proactively preventing hallucinations during on-the-fly generation. Compared with reference English performance, the zero-shot transfer results show significantly lower accuracy for both online and offline settings, with drops exceeding 5% points. These substantial performance declines underscore the inherent challenges in achieving reliable zero-shot XLT, which is consistent with the findings from prior work (Lauscher et al., 2020; Pfeiffer et al., 2020b).

heading: Few-Shot Transfer and Translate-Train. As
text: detailed in Table 3, few-shot transfer results for XLMR show remarkable improvements as the number of annotated German instances increases. With 10% of the English HADES training set (i.e., 876 annotated instances), accuracy improves by 4.9% points (offline) and 3.6% points (online) compared to zero-shot transfer. The corresponding G-Mean score increases by 8.2% points (offline) and 5.1% points (online). Notably, with only 100 annotated instances, accuracy improves by 2.3% points (offline) and 1.2% points (online), and the G-Mean score improves by 4.5% points (offline) and 1.9% points (online). This demonstrates the substantial impact of incorporating minimal annotated data on enhancing XLT performance. The translate-train approach, which involves translating a large corpus of 6,344 instances, yields accuracy gains of 3.6% points (offline) and 4.0% points (online) compared to zero-shot transfer. While beneficial, the marginal gains compared to few-shot transfer highlight the
Not Hallucination Hallucination

heading: POS Accuracy ↑ G-Mean ↑ BS ↓ AUC ↑ P ↑ R ↑ F1 ↑ P ↑ R ↑ F1 ↑
text: Adjectives 65.80 64.65 22.20 72.61 71.07 53.55 61.06 62.66 78.06 69.52 Nouns 58.42 56.31 25.37 63.99 61.97 43.97 51.15 56.64 72.88 63.62 Verbs 52.16 37.20 28.13 59.69 51.24 88.65 64.93 58.78 15.68 24.64
Table 5: Part-of-Speech (POS) results of XLMR (%) in the online setting averaged over 5 runs. We only consider instances with MARKED WORD SPANS containing particular types of POS in the German language: adjectives, nouns, verbs.
practical efficiency of using smaller amounts of high-quality annotated data. Based on our findings, few-shot transfer emerges as a highly viable strategy for cross-lingual transfer of reference-free hallucination detection, offering robust performance gains over zero-shot transfer without the extensive resource required by the translate-train approach. This re-emphasizes the well-documented practical benefits of few-shot cross-lingual transfer (Lauscher et al., 2020; Schmidt et al., 2022), here for reference-free hallucination detection.
Analysis. According to Liu et al. (2022), nouns and verbs are the most frequently occurring partof-speech (POS) in the MARKED WORD SPANS of the HADES dataset. The majority of instances with nouns (62.4%) and adjectives (74.0%) in the MARKED WORD SPANS belong to the hallucination class, while the majority of instances with verbs belong to the non-hallucination class (62.8%). This indicates a significant imbalance in label distribution. To assess the impact of this imbalance on cross-lingual transfer performance, we classify the validation set of ANHALTEN based on the selected POS (nouns, verbs, adjectives) in the MARKED WORD SPANS. Instances with MARKED WORD SPANS containing multiple words from different POS are excluded. To ensure an equal number of labels for each POS, we randomly remove instances from the more frequent class. This process results in 292 noun instances, 222 verb instances, and 62 adjective instances.
We then analyze the XLT results of XLMR in the online setting. The POS results in Table 5 show that adjectives are significantly more effective in detecting hallucinations compared to nouns and verbs in German. While the effectiveness of adjectives is notable, the imbalanced distribution of instances across different part-of-speech tags, as highlighted by Liu et al. (2022), warrants further investigation and consideration. Addressing these imbalances is crucial for improving the overall robustness and
accuracy of hallucination detection models. We further conduct morphological analysis (detailed in Appendix B) and demonstrate that preceding words indicate grammatical gender in German impact model performance, underscoring the importance of linguistic context. These findings emphasize the need to address imbalances and encourage future work to enhance model performance concerning diverse linguistic features for tokenlevel reference-free hallucination detection.

heading: 5 Conclusions
text: Token-level reference-free hallucination detection has predominantly focused on English, primarily due to the lack of robust benchmarks in other languages, hindering investigation into cross-lingual transfer approaches for this important task. To address this gap, we have presented ANHALTEN, an extension of the English HADES containing gold hallucination annotations in German, allowing for reliable and comparable cross-lingual estimates for token-level reference-free hallucination detection tasks. We utilized a modular adapter-based approach to facilitate the cross-lingual transfer, demonstrating the effectiveness of sample-efficient few-shot transfer. We believe that our dataset and findings advance the understanding of hallucination detection in cross-lingual transfer setups and contribute towards multilingual hallucination detection and real-time hallucination prevention in free-form text generation.

heading: Acknowledgements
text: This work was supported by the Alcatel-Lucent Stiftung and Deutsches Stiftungszentrum through the grant “Equitably Fair and Trustworthy Language Technology” (EQUIFAIR, Grant Nr. T0067/43110/23).

heading: Limitations
text: Despite the contributions of this research, several limitations are acknowledged, which present opportunities for future enhancement. Currently, ANHALTEN extends hallucination detection to German, broadening the scope beyond English but still covering only two languages. Expanding this research to include additional languages could further increase the global applicability of our findings. Besides, incorporating data from sources other than Wikipedia could enrich the diversity and complexity of the dataset. Additionally, extending the re-
search to include other types of hallucinations (e.g., subjective hallucinations) would provide a more comprehensive understanding of hallucination detection in various text types. We experimented on encoder-only multilingual PLMs, while decoderbased PLMs (e.g., Le Scao et al., 2022; Jiang et al., 2023; Abdin et al., 2024) warrants exploration. We hope that future research builds on top of our findings and extends the research toward more domains, more languages, and specifically with the efficiency and effective concerns of hallucination detection in different languages.

heading: Ethics Statement
text: This research addresses the critical need for nonEnglish language datasets in hallucination detection by introducing ANHALTEN. The ethical considerations of this work are multifaceted. By extending hallucination detection to German, the research promotes linguistic diversity and inclusivity in AI systems. This inclusivity helps to mitigate biases and misinformation that can arise from language restrictions, fostering more equitable applications. The study also aims to facilitate the recognition of potential hallucinated content produced by large-scale pretrained models in free-form generation – could be useful in both offline and online settings. Additionally, the research outcome emphasizes the importance of resource-efficient approaches, reducing the reliance on extensive annotated data and promoting more sustainable development.

heading: A Dataset
text: The HADES dataset, introduced by Liu et al. (2022), is designed for reference-free token-level hallucination detection tasks in English. It is sourced from English Wikipedia (Guo et al., 2020), with extracted text segments that are first perturbed and then verified by crowd-sourced annotators to determine if the marked word spans in the text cause hallucination. The dataset is available under an open-source MIT License and contains a total of 10954 instances, divided into train, development, and test sets with sizes of 8754, 1000, 1200 respectively. Within the dataset, 54.5% of the instances are classified as Hallucination, while 45.5% of the instances are classified as Not Hallucination. Since the original labels of the test set were not published, we primarily rely on the training and development sets throughout our experiments.
To further facilitate research on cross-lingual transfer in German hallucination detection tasks, we propose ANHALTEN in this work. We manually annotate 876 training instances and the entire development set of 1000 instances, which are machine-translated and further post-edited. The proposed ANHALTEN compared with the original Wiki and perturbed HADES, is shown in Table 6.

heading: B Additional Experiments
text: B.1 Cross-Lingual Transfer Results of mBERT

heading: B.2 Morphological Analysis
text: In English, grammatical gender is not distinguished, whereas German has three grammatical genders that influence articles, pronouns, and adjectives. Words indicating gender often lie outside the MARKED WORD SPANS used for hallucination detection. Our experiment selects instances where gender-indicating words (articles, possessive pronouns, demonstrative pronouns) precede nouns in the MARKED WORD SPANS from both the German and English datasets. This dataset includes 64 instances per language, with an equal distribution of labels.
Testing with XLMR in the online setting, the goal is to determine if contextual gender information influences hallucination detection results. The additional gender information might help classify nonhallucination instances but could mislead models if the original, correct word has a different gender.
Results in Table 8 show a performance drop in accuracy and G-Mean when gender-indicating words are included in the MARKED WORD SPANS, particularly for English instances. However, AUC improves, suggesting that the extended spans do not hinder the model’s ability to distinguish between classes. The models tend to assign more instances to the hallucination class, reducing the F1 score for the nonhallucination class. This performance drop may result from a lack of such gender-indicating contexts in the fine-tuning dataset, indicating potential issues with handling longer MARKED WORD SPANS.

heading: Future Work
text: Despite the contributions of this research, several limitations are acknowledged, which present opportunities for future enhancement. Currently, ANHALTEN extends hallucination detection to German, broadening the scope beyond English but still covering only two languages. Expanding this research to include additional languages could further increase the global applicability of our findings. Besides, incorporating data from sources other than Wikipedia could enrich the diversity and complexity of the dataset. Additionally, extending the re-
search to include other types of hallucinations (e. g. , subjective hallucinations) would provide a more comprehensive understanding of hallucination detection in various text types. We experimented on encoder-only multilingual PLMs, while decoderbased PLMs (e. g. , Le Scao et al. , 2022; Jiang et al. , 2023; Abdin et al. , 2024) warrants exploration. We hope that future research builds on top of our findings and extends the research toward more domains, more languages, and specifically with the efficiency and effective concerns of hallucination detection in different languages