HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation
Authors: David Dale
Elena Voita
Janice Lam
Prangthip Hansanti
Christophe Ropers
Elahe Kalbassi
Cynthia Gao
Loic Barrault
Marta R. Costa-jussà
Venue: EMNLP 2023 Main (2023)
Keywords: Hallucination
Omissions
Dataset
Machine Translation
TL;DR: We release an annotated dataset for the hallucination and omission phenomena covering 18 translation directions with varying resource levels and scripts.

Abstract: Hallucinations in machine translation are translations that contain information completely unrelated to the input. Omissions are translations that do not include some of the input information. While both cases tend to be catastrophic errors undermining user trust, annotated data with these types of pathologies is extremely scarce and is limited to a few high-resource languages. In this work, we release an annotated dataset for the hallucination and omission phenomena covering 18 translation directions with varying resource levels and scripts. Our annotation covers different levels of partial and full hallucinations as well as omissions both at the sentence and at the word level. 
Additionally, we revisit previous methods for hallucination and omission detection, show that conclusions made based on a single language pair largely do not hold for a large-scale evaluation, and establish new solid baselines.

HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation
David Dale, Elena Voita, Janice Lam, Prangthip Hansanti, Christophe Ropers,
Elahe Kalbassi, Cynthia Gao, Loic Barrault, Marta R. Costa-jussà
FAIR, Meta
chrisropers ekalbassi cynthhaaga lot www.baorccaaala... costajussa oospra/esooaooarraaaoooooossssaa
lenavoiia, janilam, wwwngtttipaaanaannininnnnnnnnnne htwp://aaaaanaaaaaaaaaaaaaaaaaaa
com
Abstract
18 lanuuage aairs
-150 sentercee
Professional annotation
eacrs
Hallucinations machine translation are trans- lations that contain information completely related to the input. Omissions are translations that do not include some the input informa- tion. While both cases tend to be catastrophic errors undermining user trust, annotated data with these types of pathologies is extremely scarce and is limited to a few high-resource lan- guages. In this work, we release an annotated dataset for the hallucination and omission phe- nomena covering 18 translation directions with varying resource levels and scripts. Our anno- tation covers different levels of partial and full hallucinations as well as omissions both at the sentence and the word level. Additionally, we revisit previous methods for ballucination and omission detection, show that conclusions made based on single language pair largely do not hold for a large-scale evaluation, and establish new solid baselines.
Transtations
Guidelines and qualfication tests linguists
Hin' resource
Ruscar
8167
anmocattion
Engiish
Spanish
German
Pastprocemsing
Chinese
De4h
Loweresource
Kashmir
Dataset
Engish
Yorube
Maripuri
Hallucinarioos Omtisiions
Zere-shot
Yoruca
Annotations
Sentence level Word- -evel-ivvll
Number of exammles
Hallucination, full
Haluciiation, partial
Omiision
Other
Figure 1: Dataset summary.
ham et al. (2013), MQM (Lommel et al., 2014), or XSTS (Licht et al., 2022) do not seem suitable since they do not distinguish between hallucina- tions, omissions and other translation errors. In this work, we aim to address this limitation.
1 Introduction
Ideally, an evaluation dataset for hallucina- tion/omission detection should satisfy several con- ditions: (i) data has to cover a broad range of lan- guages with varying resource levels and scripts, (ii) translations should be generated naturally, (iii) the models that produced the translations have to be available, and (iv) considered model- ing paradigms have to cover several approaches (i.e., encoder-decoder vs decoder-only, single lan- guage pair vs multilingual, etc.). The first point is important because the best-performing detectors are different for high- and low-resource settings, and general conclusions cannot be made based on a single language pair (Section 4). Secondly, transla- tions have to be generated naturally as opposed to using specifically developed perturbations of mod- els and/or data because conclusions for the latter might not transfer for detection of natural hallucina- tions (Section 6). Thirdly, the corresponding model should be released along with the translations to allow evaluating "internal" detection methods. Fi-
With neural machine translation systems reaching an overall satisfactory quality, alleviating those rare but severe translation pathologies that undermine user trust becomes very important. These patholo- gies include hallucinations (translations containing information completely unrelated to the input) and omissions (translations that do not include some of the information of the input). While under- standing hallucinations is receiving increasing at- tention (Raunak et al., 2021; Müller and Sennrich, 2021; Zhou et al., 2021; Guerreiro et al., 2023; Dale et al., 2023; Guerreiro et al., 2022), progress in this direction is hampered by the lack of annotated data. To the best of our knowledge, previous datasets are limited to German-Engie English data with sentence-level annotations of hallucinations and omissions (Guer- reiro al., 2023) and Chinese-Engllic data with token-level hallucination labels (Zhou et al., 2021). Previously available ceneral-purpont quality as- sessments, such as direct assessment (DA) Gra-

---

nally, in the most ideal setting, various models are needed test whether the detection methods trans- fer between modeling approaches.
(ii) attention is very fragile when used to judge translation quality, among other observations;
introduce word-level pathology detection tasks along with the baselines.
While satisfying all the desiderata is very chal- lenging, we can satisfy all but last by focus- ing on the ofate-of-thh-ar multilingual NLLB-200 model (NLLB Team et al., 2022). In addition to covering a broad range of languages and being pub- licly available along with its training data, NL.LB is widely recognized' and is likely to stay the state- of-the-art for the foreseeable future. For this model, we choose 18 language pairs that include high- and low-resource languages, as well as zero-shot pair (Figure 1). We develop rigorous annotation guidelines for identifying full and partial halluci- nations and omissions and use these guidelines for manual anmotation of translations in all 18 direc- tions. The resulting dataset contains fine-gratd fineggraieed sentence-level and teven-level annotations.
2 Dataset Creation
The steps to create the dataset were (i) choosing the language pairs, (ii) gathering data for anno- tation, (iii) developing annotation guidelines and qualification sets, (iv) manual annotation, (v) post- processing. Here, we explain these steps.
2.1 Selection of Languages
We optimized the language selection in order to cover (i) different resource levels and (ii) variety of language families and scripts. Among the lan- guages available NL.LB-200, we include 5 high- resource language pairs (Arabic, Mandarin Chi- nese, German, Russian, and Spanish paired with Linlish), 3 low-resource language pairs (Kashmiri, Manipuri, and Yoruba paired with English) and a zero-shot pair YSpunish-Yoruba). We consider all language pairs in both directions which gives 18 translation directions summarized in Figure 1.
We highlight the importance of our dataset by making several valuable observations that would not be possible otherwise. For example, we find that for low-resource directions, internal methods perform much better than external methods that substantially fail. When analyzing performance of a range of recently introduced pathology detection methods, we see that some of the previous results do not transfer across languages. As another ex- ample, we show that relying attention to make conclusions about translation quality is very fragile. Finally, introduce some detection tasks (e.g., token-level omission detection) that became possi- ble only with our data. We believe our work opens the door for reliable and accessible rescarch on detecting and analyzing translation pathologies as well as understanding their causes. Overall, we:
2.2 Gathering Data for Annotation
Since strong NLLB models rarely generate halluci- nations and omissions, getting translations that are likely to contain these types errors is not straight- forward. To gather these translations, we developed a multi-step procedure where we first choose data to generate translations and then choose a subset of the generated translations for annotation.
Choosing data for translation. Since we expect that the NLLB model will not hallucinate much when handling high-resource languages, in addi- tion to clean in-domain data, we use noisier out- of-domain sources. Overall, the data we use to generate translations is as follows:
release dataset with fine-grained profes- sional annotations of hallucinations and omis- sions for 18 language pairs?;
in-domain: FLORES-200 development set (NLLB Team al., 2022);
analyze previous sentence-level detectors and find that e.g. (i) for low-resource set- tings, model internal characteristics work best,
out-of-domain: Jigsaw toxicity detection com- petition corpora (Jigsaw, 2020)* for En- glish, Russian and Spanish; comments from Wikipedia discussion pages$ for Chinese,
OOnly 4-months after launching NLLB-200. Wikimedia reported that this was the third most used machine transla- tion engine accounting for 3.8% of all published translations. Scientific impact is also prominent:
model has been used as standard to compare with other MT paradigms such as prompting with large language models (Zhu et al.. 2023).
"In the NLLB training dataset. Spanish and Yoruba sen- tences were paired to English but each other.
"ht tpt: htts://wwww /ww. kaggle. com/ compe ti tions// janss/jigswwnn ingualltoxice wawiic-comettteeee classii wwwsssliiaasiiiiiiiiiiiiiiii fication/ FFrom public dumps: https: / //dumps. wi wikimedia. org/.
'The data and code are availahle at https: / /ii thub. com Tracebookresearce httpapesss/sss/////eeeecss tetoop//treeeeee teeeeeeeeeeeeee wwweemaan/dem/o/ee/eeeeee hwwiin//em///ommm.ii.. /halomi

---

Definitions
Hallucination mistranslation
Severity levels
To decicl whethera source token "corresponds" to an erroneous target token, answer:
Dependligg the number of words ahe
Hallucinations: contain
hellucinatedom/tte, a patholoogy can
Information completely unrelated imput.
Does this source word fall into the common meaning category as this target word?
level-eeell only words,
Omissions: some Information contained the source is not present in the translation.
Does this source word have semantic connection with this target word?
Partial: least words, not
all except maybe 1-2 words.
Can you to this source word is associated with this target word?
with reasonable theory how
The severity levels do not overlap halluciaaal full).
It "no" to all, then hallucination.
Figure 2: Overview of some parts of the annotation guidelines.
Arabic and German. The Jigsaw corpora were extracted from Wikipedia talk pages, the distributions of these texts rather similar.
2.3 Guidelines and Qualification Tests
To ensure annotation quality, guidelines and qualifi- cation tests were prepared by professional linguists.
We translated these texts with the 600M distilled NLLB model following the standard setting (beam size 5, forbidden generation of the <UNK> token, forbidden repetition of 4-grams, limiting the trans- lation length to 3-len(source) +-5 tokens.
Annotation guidelines. These guidelines define:
the notion of hallucinations and omissions;
the hallucination vs mistranslation distinction;
hallucinatiion/omissoon severity levels.
Choosing translations for annotation. To find potentially pathological translations, scored sen- tence pairs by multiple metries that were used as hallucination detectors in previous works. Specifi- cally, we used some methods from Guerreiro et al. (2023): ChrI++ (Popović, 2017), reference-based COMET and referenceless COMET-QE (Rei et al., 2020), and Seq-Logprob (their best detec- tor). We also used some methods introduced in Dale al. (2023): cosine similarity coming from LASER3 (Heffernan al., 2022) and LaBSE (Feng al., 2022), bidirectional XNLI score, and ALTI+ source contributions (Ferrando et al., 2022). For each translation direction and data source, we selected sentence pairs with 3 strategies:
Figure summarizes the resulting guidelines. Note that distinguishing hallucinations from other trans- lation errors is one of the known difficulties when dealing with hallucinations (Raunak et al., 2021; Guerreiro et al., 2023). In our guidelines, a token is referred to as hallucinated if there is no corre- sponding token in the source (Figure 2).
For all pathologies, linguísts provide positive and negative examples in diverse languages. Addi- tionally, we ask the annotators to mark a trans- lation is incomprehensible, i.e. whether the text is garbled or in another language. These translations are then discarded."
Qualification tests and postprocessing. For notation, we choose professional translators (2 for each language) who are allowed annotate our data only after passing specifically developed qualification test. More details on this test and postprocessing steps can be found in Appendix A.
Sample uniformly to preserve data diversity and hon-haalueinated examples;
Sample favoring potentially pathological translations (with the probabilities propor- tional to the quantiles of the detectors);
3 Dataset Description
Pick the worst according the detectors to increase the chance of hallucinations.
Annotation format. The resulting data contains the source text and its translation, along with the
"We believe that incomprehensible texts should consid- ered separately for two reasons. From the user perspective, hallucinations and omissions are mostly fluent, which can mislead the user into trusting the translation: differently, in- comprehensible texts are clearly bad sentences and thus do not mislead the user. From the detection perspective, incompre- hensible sentences can he recognized regardless of the source. while hallucinations and omissions can he judged as such only in relation to the souree sentence.
Appendix A describes the amount of data selected by these strategies for all directions.
"We selected the smallest model from NLLBB family as most popular one, and potentially the one generating most hallucinations and omissions.
"We applied the reference-hacce hased metrics only to the FL.O- RES data.

---

English-Arabic
English- Chineee
German-English
Because of their success with submarincs, after war Germans aren't trusted to have many of them.
He had 2 gonts and assists Washingren's the Adanta Thrashers.
Selostangabe) nur geprifte
Input
over
Output
Joe y ou my clael jell اlاااا of panlai wow
fil S 28 01 e Adf (8
that no not what I'm saylng. Full
that
اارر et in 200 ماي المول oe Jom
PR PR 40 NEEEE
Omission degree
Word-level
Partial
Omitted word span annotation
Because of their success with submarinees after the war Germans aren't eestrusted top have many of them.
He nad ex goals and 2 assists in Weshinston's 30o0 eecover the Atlanta ce<Tirashers>>>.
cx-Selbstangabel nur geprüftte Dokumentexoo,
Hall, degree
No hallucination
Partial
Full
Hallucinated word
Husst bny الااا اoاودl of panloi vew الع الال ei on البر عاي الuون oe (ركر
<eeit's I'm sure what I'm saylng>>>.
span annotation
326)
Figure 3: Annotated examples from our dataset.
word-level and sentence-level annotations of omis- sions and hallucinations. Figure shows examples of annotated translations from our dataset.
machine-genennn translation is first passed to quality estimation system and then, if needed, is corrected by human translators.
Overall statisties. Figure shows the propor- tions of hallucinations and omissions in the data (translations with both hallucinations and omis- sions are referred to hallucinations). Overall, all directions have at least 3% translations with hallucinations (I % full) and 17% with omissions (5% full). Most of the full hallucinations are also labelled as full omissions, and vice versa.
Detection tasks. For our dataset, we define three sentence-level detection tasks:
hallucination detection: same in previous work mentioned above;
omission detection: detecting translations
with omissions on hallucination-fres subset. ure 1 we see that, not surprisingly, high-resource language pairs hallucinate less than low-resource. A less anticipated difference between high- and low-resource settings seen when looking within each language pair. In high-resource settings, trans- lating to English leads to more hallucinations than translating from English. Differently, for low- resource pairs, translation from English has higher hallucinatory rates than translation to English for the same language pair. This might inspire future work to analyze the role of English data in the mul- tilingual NLLB model. Finally, while for the zero- shot pair one might expect more pathologies, this is not what we see: results for the zero-shot pair are comparable to those for low-resource languages.
Differences between resource levels. From Fig-
The latter is to disentangle omissions from more severe hallucination pathology;
pathology detection: detecting translations that are either hallucinations or omissions.
Evaluation methodology. We evaluate the abil- ity of detector rank more severe pathologies higher (e.g.. full hallucinations higher than partial, any hallucinations higher than non-hallucinations, etc). For this, we use an adaptation of the binary ROC AUC score for several classes. Formally, we subtract from the perfect score, i.e. 1, the percent- age of incorrectly ranked pairs of sentences with different labels. For two classes, this metric is equivalent the ROC AUC score.
We compute the metrics for each translation di- rection separately.
4 Sentence-Level Detection
Detecting pathologies at the sentence level is the task of flagging a whole translation as pathological or not. This is the standard definition of e.g. the hal- lucination detection task (Lee et al., 2019; Müller al., 2020; Raunak al., 2021; Guerreiro al., 2023; Dale et al., 2023; Guerreiro et al., 2022: Xu et al., 2023). Such sentence-level pathology de- tection (instcad of flagging individual erroncous tokens) is an integral part of hybrid pipelines when
4.1 Detection Methods
Detection metrics can be either internal, i.e. rely- ing only the information from the model that generated the inspected translation, or external, i.e. using external models. We use the best detectors from several recent works, along with some of their modifications we propose in this work. The metrics are summarized in Figure 4.

---

Internal methods
Gray
External methods
Sequence pee-rrbabilltt
contributions
Attention
quality estimation system
Sentence similarity measures
generated translations
abnormality
Seq-Logprob
ALTI
ALTI SOUrDD
Attn-OT
COMET-QE
LASER
LaBSE
XNLI
BLASER
source
OT-distance between attention patterns: "good" and the current
Multilingual similarity between the source translation
how mucha a prediction is based on the source
how much each source token is used
Figure 4: Summary of the sentence-level detection methods.
Internal methods. For internal models, we use the best method from Guerreiro et al. (2023) (sequence (og-probability) and the best internal method from (Dale et al., 2023), ALTI, ALTI (Fer- rando et al., 2022) is an attribution method that evaluates token contributions to generated transla- tions. For hallucination detection, Dale al. (2023) evaluate how, on average, the prediction of each target token is based on the source. Here, mostly to detect omissions, we propose a different vari- ant ALTT' that computes how much, on average, each source token was used to generate the transla- tion. Intuitively, if many source tokens are not used during generation, the translation is likely to not contain some information. The difference between the two versions of ALTI is illustrated in Figure 4.
4.2 Experimental Results
The detection scores for hallucinations and omis- sions are shown in Figure The scores for detect- ing all pathologies are given in Appendix C.
High-resource: much easier handle. We can see that it is much easier detect pathologies in high-resource settings: the gap between the best scores for high- and low-resource directions is rather big (e.g., for halucinations, 0.89 vs 0.79). Note also that for high-resource language pairs, both internal and external methods perform quite well (e.g., .., Seq-Logprob and LaBSE for hallucina- tions; XNLI, LaBSE and ALTIT for omissions).
Low-resource: internal methods take the lead. In low-resource settings, external methods drop substantially with some of them losing sensibil- ity. For example, high-performing XNLI drops close to chance for all pathologies. Overall, most external models (with the exception of massively multilingual BLASER) are unlikely to be compe- tent for low-resoure, directions as they do not ob- serve enough relevant data during training. While previous work already expressed this concern and advocated focusing on internal methods (Dale al., 2023), without our dataset, verifying this was not possible.
External methods. For external methods, use the state-of-the-art quality estimation system COMET-QE (Rei et al., 2020) and sentence sim- ilarity measures proposed Dale et al. (2023). The latter are cosine similarities coming from LASER3 (Heffernan et al., 2022), LaBSE (Feng et al., 2022), and a bidirectional XNLI score. Fi- nally, we evaluate translation quality estima- tion method from Seamless Communication et al. (2023), BLASER 2.0-QE, built on top of SONAR sentence embeddings (Duquenne et al., 2023).
Hallucinations: Seq-Logprob the most stable. After turned out that the standard sequence log- probability is more informative for hallucination de- tection than the heuristics introduced earlier (Guer- reiro et al., 2023), a few recent works reported im- provements over Seq-Logprob: ALTI and LaBSE in Dale et al. (2023) and Attn-OT in Guerreiro al. (2022). We see, however, that on average, Seq- Logprob is still the most robust accross translation directions. This discrepancy comes from the fact that those works made conclusions based on a sin-
Gray-area method. Finally, also use a recent optimal transport-based measure evaluating the ab- normality of the attention distribution compared to those of good translations (Guerreiro et al., 2022). While this method uses internal characteristics, requires external data, i.c. large collection of at- tention maps for "good" translations, which can be hard to obtain for low-resource settings.
WWe use the hest variant of the original method (Guerreiro ct al., 2022). For more details, see Appendix B.

---

Internal
External
Internal
External
Seq Loge Regpad
COMET-QE
BLASER-QE
COMET-QE
BLASER-QE
Sec Loggo
ALTI
ALTIT
Attn-OT OT
LaBSE
LASER
XNLI
ALTI
ALTIT
Attn-OT
LaBSE
LASER
XNL
Arglish-Arabic
0.89
0.78
0.54
0.36
0.84
0.84
0.89
0.83
0.90
0.63
0.76
0.64
0.61
0.79
0.77
0.76
0.85
Arabic-English
0.83
0.75
0.72
0.51
0.80
0.88
0.83
0.87
0.94
0.57
0.49
0.82
0.72
0.68
0.83
0.62
0.86
0.77
English-Russan
0.95
0.36
0.37
0.61
0.99
0.86
0.80
0.62
0.89
0.53
0.56
0.79
0.75
0.53
0.85
0.74
0.80
0.84
Russian-English
0.86
0.76
0.56
0.55
0.78
0.63
0.85
0.77
0.92
0.59
0.52
0.86
0.76
0.57
0.76
0.75
0.80
0.80
High- resource
English-Spanish
0.87
0.65
0.69
0.59
0.85
0.68
0.84
0.62
0.85
0.39
0.331
0.89
0.89
0.65
0.86
0.64
0.85
0.80
Spanish-English
0.92
0.89
0.67
0.37
0.86
0.94
0.88
0.89
0.87
0:51
0.50
0.59
0.66
0.69
0.73
0.62
0.70
0.71
English-Germar
0.85
0.97
0.69
0.55
0.88
0.97
0.88
0.94
0.87
0.50
0:46
0.77
0.74
0.54
0.66
0.57
0.56
0.85
German- Eniglish
0.90
0.60
0.59
0:65
0.92
0.95
0.86
0.68
0.97
0:18
0.38
0.82
0.83
0.62
0.70
0.64
0.77
0.70
English- Chinese
0.88
0.82
0.47
0.60
0.84
0.86
0.79
0.84
0.78
0.60
0.51
0.73
0.67
0.57
0.69
0.73
0.63
0.80
Chinesa-English
0.89
08.88
0.65
0.46
01.89
0.88
0.82
0.78
0.87
0.23
0.61
0.77
0.64
0.66
0.25
0.74
0.79
0.73
High-resource mean
0.88
10.79
0.99
0.52
0.86
0.89
0.84
00.81
0.899
0.56
0.48
0.78
0.73
0.61
0.76
0.08
0.75
0.79
English-Kashmir
0.68
0.71
0.24
0.54
0.64
10.76
0,70
0.68
0.81
0.50
0.52
(0900
0.81
0.80
0.76
0.55
0.60
0.77
Kaglisi-Englissh
0.59
0.67
0.65
0.65
0.69
0.58
0.59
00.47
0.73
0,36
0.50
0.64
0.63
0.61
0.59
0.44
0.53
0.45
Low-
Ynglish-Yoruba
0.68
0.81
0.49
0.54
0.57
0.80
0.76
0.335
0.83
0.48
0.52
0.80
0.73
0.51
0.82
0.68
0.64
0.80
resource
Englis-English
0,70
0.64
0.49
0.57
0.58
0.57
0.59
0.44
0.78
0.42
10:33
0.68
0.63
0.51
0.52
0:45
0.52
0.74
English Manipuri
0.77
0.74
0.59
0.65
0.68
0.56
0.57
0.44
0.79
0.77
0.50
0.70
0.72
0.59
0.65
0.41
0.54
0.67
Manipuri-Englissh
0.78
0.72
0.54
0.43
0.58
0.66
0.69
0.47
0.00
0.68
0.60
0.65
0.55
0.57
0.65
0.76
0.56
0.56
Low-resource mean
0.70
0.72
0.50
0.56
0.62
0.66
0.65
0.48
0.79
0.53
0.50
0.73
0.68
0.60
0.67
0.55
0.57
0.66
Zero-
Yoruba Spanish
0.60
0.65
0.47
0.44
0.54
0.56
0.60
0.51
0.50
0.62
0.47
0.25
0.69
0.40
0.62
0.63
00311
0.66
shot
Spanish-Yaruba
0.61
0.66
0.52
0.55
0.55
0.66
0.64
0.52
0.68
0.68
0.50
0.83
0.60
0.51
0.69
0.77
0.42
0.67
Overall mean
0.79
0:75
0.58
0.53
0.75
0.78
0.75
0.67
0.83
0.56
0.48
0.77
0.70
0.59
0.72 0.64
0.65
0.74
Hallucinations (among all translations)
Omissions (among non soluiaaluciinatonns)
Figure 5: Results for sentence-level detection of hallucinations (left) and omissions (right).
gle language pair. This highlights the importance of our dataset enabling large-scale evaluation over several language pairs.
this even further. This points to a larger debate on the distinction between attention and attribu- tion and the consequences of mistaking one for the other (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Bastings and Filippova, 2020). While Attn-OT was introduced as way to evaluate detachment from the source sequence (Guerreiro et al., 2022), we see that im- plementing this intuition with attention instead of attribution (as e.g. ALTI) leads to varying results: from high performance in Guerreiro et al. (2022) to near-random performance in our experiments.
MLASER-QER a SOTA hallucination detector. On average, BLASER-QU performs on par with the best hallucination detection methods high-resource directions, and outperforms them on low-resource directions. Apparently, fine-tuning massively mul- tilingual sentence encoders to predict semantic sim- ilarity is good recipe for hallucination detectors.
Attention-based method: close to chance. For hallucinations, Attn-OT detecting attention anomaly is outlier and performs close to chance. 1011 While previous work already showed that relying on attention patterns make conclu- sions about translation quality can be fragile (Guer- reiro et al., 2023), results with our dataset highlight
Omissions: internal ALTIT T performs best. For detecting omissions among non-halluee halllulalaainns,, the quality is generally worse than for hallucinations. The best method is ALTI' which confirms our in- tuition that if, according to token contributions, some source words are not used for translation, a translation is likely to omit relevant information. LaBSE, XNLI and BLASER QE also perform well for high-resource languages but, similar to halluci- nation detection, are worse than internal methods for low-resource. Finally, while Attn-OT does not seem to identify hallucinations, it is sensible for omissions.
We tried all the versions of the method from Guerreira al. (2022) as well some additional modifications improve its results. For the dataset from Guerreiro et (2022), we managed reproduce their results. For out dataset, we show the best method variant in the main text and the rest (along with the implementation details) in Appendix B.
"For NLLB, poor performance of this method could be attributed to the overall large attention to the EOS token. We tried removing this token from the optimal transport computa- tion, but this did not improve the results significantly.

---

Log probabiiity
Token contributions
two types of log-probability: the standard and its difference with the unconditioned log- poo-probayilityy for the same token (i.e., when conditioning on an empty source sentence). Intuitively, the latter one more way measuring whether the model uses the source or relies more on its language model. For ALTI, we use both the total source contribution and the maximum contribution among the source tokens. The latter is high when the model is "fo- cused" on specific source token this might be indicative of the prediction quality. For omissions, we use the same methods with swapped source and target sentences (i.e., ALTI turns into ALTI").
standard
contrastive
ALTI (sum)
AmTT(max)
P()l)..)
Scurce
Poilyen <<as>>
39
Figure 6: Token-level detection methods for hallucina-
tions. For omissions, we swap the source and the target.
5 Word-Level Detection
In contrast to sentence-level detection, detecting pathologies at the word level received much less attention. In terms of both available data and detec- tors, previous attempts were rather limited (Zhou et al., 2021; Vamvas and Sennrich, 2022). Here, want facilitate future research in this direction.
Combination of methods. Apart from the indi- vidual methods, we also consider their linear com- binations. We use logistic regression trained us- ing 3-fold group-wise cross-validation (with sen- tence ids as groups). We train the same feature combination for all languages by maximizing the detection score on the pooled data.
Detection tasks. We define two detection tasks:
hallucination detection: for each translation word, predict whether is hallucinated;
omission detection: for each source word, pre- dict whether omnitted from the translation.
5.2 Experimental Results
Segmentation. We segment texts using Sacre- BLEU tokenizer (Post, 2018). For Chinese, in- terprets each Chinese character as an individual word. For other languages, it applies regex- -ased tokenization based on spaces and punctuation.
The results are shown in Figure 7. Overall, the methods we proposed are reasonable and perform much better than the random baseline.
Token contributions perform best. We see that for both hallucinations and omissions, token con- tributions coming from ALTI (or ALTIT for omis- sions) perform better than the log-probabilityy com- ing from the model. Note that for hallucinations, this is different from the sentence-level results where Seq-Logprob outperformed ALTI.
Evaluation methodology. For these binary clas- sification tasks, we use the ROC AUC score. Since models operate at the token level, we make predic- tions for tokens and not words. If word is segmen- ted into several tokens, we assign the worst score among its tokens (i.e., if one of word's tokens is hallucinated, the entire word is hallucinated).
Contrastive vs standard log-probability. An- other interesting observation is that for hallucina- tion detection in the high-resource setting, con- trastive log- bog-probability gives better signal than the standard log- bog-probability. This indicates that comparing explicitly the changes when dropping some information can be useful. This is not surpris- ing: in a broad sense, our contrastive log-prot proobbbaaii lity is a variant of crasure- based interpretation ap- proaches (Zeiler and Fergus, 2014; Li et al., 2017; Kádár et al., 2017: Poerner et al., 2018; 8; Li et al.. 2019). In our case, the erased part is rather large, i.e. the whole source sentence. For such large segments, similar idea previously appeared in context-aware machine translation when dropping the entire context sentence (Fernandes al., 2021).
5.1 Detection Methods
To the best our knowledge, there are no publicly available models for token-level detection of hallu- cinations or omissions that could be easily adapted to handle the language pairs in our dataset. Apply- ing previous approach by Zhou al. (2021), i.e. training specialized model (or several language pair-specific models) on synthetic data, would be very demanding in terms of engineering and re- search effort to work well on diverse resource lev- els. Therefore, we suggest starting with internal methods and their combinations.
Internal methods. For internal methods, we rely on the same methods that were previously used: model log- proobabbayylty and ALTI (Figure 6). We use
The detectors are complementary. Finally, we see that log booo-rbbaaylity and token contributions are

---

Logprob
ALTI
Difficult to detect pathologies. Examples 4-6 show partial hallucinations and omissions that are difficult to detect, either because (in some sense) they resemble a correct translation, or because the translation indeed remains within the range of pos- sible translations despite having these pathologies. This raises question: what does it really mean to have a hallucinated translation? While our sen- tence-level labels are fine-grained, the severity of pathology is defined based on the number of omit- ted/hallucinated words rather than on the degree of semantic inadequacy the pathology (similarly to e.g. Guerreiro et al. (2023)). This agrees with previous work noting that defining severity of trans- lation errors is not straightforward (Graham et al., 2013; Licht et al., 2022; Guerreiro et al., 2023).
Rand
Shandl
Contr.
Sum
Comb.
High-rescurce
0.50
0.87
0.71
cow-fessurce mean
0.53
3.66
0.61
0.67
Hallucinations
Overall Eeeaall mean
0.5
0.70
0.70
0.70
0.05
0.54
High-resaurce mean
0.51
9.6
0.79
0.86
0.03
0.92
Low-tesaurceemen mean
0.52
0.1
0.65
0.69
0.65
0.74
Omisssoos
Overall
0.51
0.76
Figure 7: Word-level detection results.
complementary: for both hallucinations and omis- sions, combining the features leads noticeable improvement in detection quality.
6 Natural vs "Artificial" Pathologies
One of the difficulties when dealing with hallucina- tions (and, to a lesser extent, omissions) is that this is a rare phenomenon. Therefore, previous work often resorted to artificially amplifying the prob- lem by applying various perturbations (Lee et al. (2019); Raunak et al. (2021), among others). How- ever, it is not clear whether conclusions made based this synthetic data would transfer to pathologies generated by model in natural setting.
Correctly detected pathologies. Examples 7-11 show more severe hallucinations and omissions these are detected correctly by at least a few of the considered methods. Many of these pathologies are produced for out-of-distribun junction of several sentences, typos, non-sentence texts (such as dates), and very short or incomplete sentences. Note that e.g. short sentences are non- typical for the NLLB training data. As we see, for such inputs the model is often not confident even for correct translations (see e.g. examples 2 and 3 correct but short translations are flagged as patho- logical). This suggests that these errors might be alleviated by augmenting training data with similar (very short, multi-sentence, etc.) samples.
inputs: con-
In Appendix D, we compare performance of detection methods between two datasets: (i) our dataset with natural translations and (ii) translations generated with perturbed model. We find that data with translations generated under perturbation has to be used with caution, especially when evaluating pathology detection methods: the conclusions are likely not transfer to the natural setting.
7.2 Word-Level Detection
7 Discussion
In Appendix E.1, we also show examples of word- level detection and discuss the behavior of the de- tection methods. For example, we note that log- probability focuses on the beginnings of sentences while ALTI focuses on the endings.
We that some of the internal and external meth- ods can detect hallucinations and omissions with the quality that is much better than nothing, much worse than perfect. But what are the cases which methods perform well and what can be improved?
8 Additional Related Work
7.1 Sentence- Detection
Except for mentioned above work, previous work on hallucinations in machine translation largely avoided human annotation. To judge whether a translation is hallucinated, they relied on various heuristics or string- string-based based automatic evaluation met- rics (Lee et al. (2019); Berard al. (2019); Müller and Sennrich (2021); Raunak et al. (2021)). These however, were shown not be indicative of hallu- cinations (Guerreiro et al., 2023), which highlights the importance of our human- annotated data. For omissions, previous work mostly focused on empty
Figure 8 shows manually selected examples of false and true positive and negative detection results.
Flagging correct translations. Examples 1-3 are correct translations, but some of them are flagged pathological. Example 2 is flagged as hallu- cination and omission, probably because the input ":::Jeez." " slang and has wide range of potential meanings. Example is flagged by Seq-Logprob: for the model, this translation may be "unlikely" because is short and consists of a rare word.

---

Sea: Logerob
AL
ALTI
Output
gladd Greyecen sedioee Wollpprdia,,
of
alegeo dee Larry Grayeom
-0.74
0.44
0.56
0.98
Diss
Kernet
Kerntt
<2/70
0.69
0.09
1.00
seqeeecggee legpro
th group starts up., assates stor, we should looirat other advertiiing ceoaagg gusense and abou which ons we' the to exper ohw about cod hames, Aga
Slezte grupa com enta combanan deboriamos mitara a ctros "ooogranasen crassss"" publiiciarn yoe perser en ouéseccror ws ros gus guslaempler esccibrr sobre.
false negetive fullemmaaion
.0.7
0.60
0.57
0.86
Former U.S. Speaker of the House Neww Gingrich came comellieeen insector with percent.
Elex cresdemee dela Cérara coege Resprsem an aes de 'stettos Unidos,, Newt Grogrich muadd segundeollgggee con ciento.
Talso negative Ludiattt
-0.99
0.6
0.96
Most
wayoo
regeatiee omission
wrall.
-047
0.22
0.01
0.89
7 Good coepcinpion, secuuttoo acourn tharthey are the same coothingoon,
Buenc, supuss an mismoo.
postioe
0.77
8
Honevver the disccrvery of unmo in 1922 mudehima cwodaiity sedW io momery tombs the pae wire robboo, this tomo was left struual indisuoocoioo
Siventurrgo, of tles Jbriin ee antoo tumbbrer 1972 in conmint io acofobardda..
-0.66
0.56
0.29
0.79
posiiie ommtsioo
<ox PPr quérno 2ooc
-2.18
0.37
0.29
postteetivv cellecnntion
10
2010- 1-010450000
corSe
true postahe nell.cintion
cesemnll in ámibiio seguridad
2:40
0.23
0.44
0.09
11
Bye.
Heste luggo, excamigooos.
-2.58
0:58
1.02
0.63
true posiiive nallucinatin
Figure Examples of successful and failed sentence-level detection for translations from English to Spanish.
translations (Stahlberg and Byrne, 2019; Vijayaku- mar et al., 2016) with some work using artificially created undertranslations (Vamvas and Sennrich, 2022). As saw in Section 6, the latter is unlikely be helpful when evaluating detection methods.
to evaluate progress on hallucinations/omisiioos de- tection and mitigation. However, all the annotated translations were generated with a single model, and the generalization to other models is yet to be verified.
The dataset is rather small and may not cover all possible tallucinations/omissions cases.
9 Conclusions
We present the first dataset with human-annotated hallucinations and omissions that satisfics several conditions: (i) it covers a broad range of languages with varying resource levels and scripts, (ii) the translations are generated naturally (i.e., without artificial perturbations), (iii) the model producing the translations is publicly available. Through extensive experiments, we illustrate why each of these conditions is important. Additionally, we make several observations of individual importance. For example, for low-resource directions internal pathology detection methods perform better than most of the external ones, attention is very frag- ile when used to judge translation quality, among others. We believe work opens the door for reliable and accessible research detecting and analyzing translation pathologies.
Ethical considerations
The annotations were provided by professionals and they were all paid fair rate.
References
Jasmijn Bastings and Katja Filippova. 2020. The ele- phant in the interpretability room: Why use attention as explanation when we have saliency methods? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 149-155, Online. Association for Com- putational Linguistics.
Alexandre Berard, Ioan Calapodescu, and Claude Roux. 2019. Naver labs Europe' s systems the WMT19 machine translation robustness task. In Proceedings of the Fourth Conference (Volume 2: Shared Task Papers, Day 1), pages 526- 532, Florence, Italy. Association for Computational Linguistics.
Machine Translation
10 Limitations
David Dale, Elena Voita, Loic Barrault, and Marta R. Costa-jussà. 2023. Detecting and mitigating halluci- nations in machine translation: Model internal work-
Our experiments are reproducible, and our dataset together with the NLLB model can be widely used

---

ings alone do well, sentence similarity even better. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics, Toronto, Canada. Association for Computational Linguisties.
ciation for Computational Linguistics: Human Lan- guage Technologies. Volume (Long and Short Pa- pers), pages 3543-3556, Minneapolis, Minnesota. Association for Computational Linguisties.
Paul-Ambroise Duquenne, Holger Schwenk, and Benoit Sagot. 2023. SONAR: sentence-level multimodal and language-agnostic representations.
Jigsaw. 2020. Jigsaw multilingual toxic comment www.://agglle..o///ig.aa..
classification. hwwp:/uuuuuooii-ooooonneneeaaaaa cessed: 2023-03-27.
Ac-
Fangxiaoyu Feng, Yinfei Yang. Daniel Cer, Naveen Ari- vazhagan, and Wei Wang. 2022. Language-agnosse BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 878-891, Dublin, Ireland. Association for Computa- tional Linguistics.
Ákos Kádár, Grzegorz Chrupata, and Afra Alishahi. 2017. Representation of linguistic form and fune- tion in recurrent neural networks. Computational Linguistics, 43(4):761-780.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020, Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 7057-7075, Online. Association for Computa- tional Linguistics.
Patrick Fernandes, Kayo Yin, Graham Neubig, and An- dré F. T. Martins. 2021. Measuring and increasing context usage in context-awas
machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 6467-6478, Online. Association for Computational Linguistics.
Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fan- njiang, and David Sussillo. 2019. Hallucinations in neural machine translation.
Javier Ferrando, Gerard I. Gállego, Belen Alastruey, Carlos Escolano, and Marta R. Costa-jussà, 2022. Towards opening the black box of neural machine translation: Source and target interpretations of the transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. Online and Abu-Dhabi, UAE, Association for Computational Linguistics.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Un- derstanding neural networks through representation erasure.
Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019. On the word alignment from neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 1293-1303, Florence, Italy. Association for Computational Linguisties.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Pro- ceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33-41, Sofia, Bulgaria. Association for Computational Lin- guistics.
Daniel Licht, Cynthia Gao, Janice Lam, Francisco Guz- man, Mona Diab, and Philipp Koehn. 2022. Consis- tent human evaluation of machine translation across language pairs. In Proceedings of the 15th biennial conference of the Association for Machine Transla- tion in the Americas (Volume I: Research Track). pages 309-321, Orlando, USA. Association for Ma- chine Translation in the Americas.
Nuno M. Guerreiro, Pierre Colombo, Pablo Piantanida, and André F. T. Martins. 2022. Optimal transport for unsupervised hallucination detection in neural machine translation.
Arle Lommel, Aljoscha Burchardt, and Hans Uszkor- eit. 2014. Multidimensional quality metries (mqm): A framework for declaring and describing transla- tion quality metrics. Tradumàtica: tecnologies de la traducció, 0:455 463.
Nuno M. Guerreiro, Elena Voita, and André Martins. 2023. Looking for needle in a haystack: A com- prehensive study of hallucinations in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 1059- 1075, Dubrovnik, Croatia. Association for Computational Linguistics.
Mathias Müller, Annette Rios, and Rico Sennrich. 2020. Domain robustness in neural machine translation. In Proceedings of the 14th Conference of the Associa- tion for Muchine Translation the Americas (Volume I: Research Track), pages 1614 Virtual. Associa- tion for Machine Translation in the Americas.
Kevin Heffernan, Onur Çelebi. and Holger Schwenk. 2022. Bitext mining using distilled sentence repre- sentations for low-resource languages. arXiv preprint arXiv:2205.12654.
Mathias Müller and Rico Sennrich. 2021. Understand- ing the properties of minimum Bayes risk decoding in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 1thh International Joint
Sarthak Jain and Byron C. Wallace. 2019, Attention is not Explanation. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso-

---

Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 259- 272, Online. Asso- ciation for Computational Linguistics.
Howes, Bernie Huang, Min-Jae Hwang, Hirofumi In- aguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ili Kulikov, Janice Lam, Daniel Li, Xutai Ma, Rus- lan Mavlyutov, Benjamin Peloquin, Mohamed Ra- madan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Celebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee. Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Sehwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. 2023. Seamlessm41- massively multilingual & multimodal machine trans- lation.
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood. Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov. Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Kochn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Sofia Serrano and Noah A. Smith. 2019. Is attention in- terpretable? Procerdings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages
Florence, Italy, Association for Computational Linguistics.
Nina Poerner, Hinrich Schütze, and Benjamin Roth. 2018. Evaluating neural network explanation meth- ods using hybrid documents and morphosyntactic agreement. In Proceedings of the 56th Anmual Meet- ing of the Association for Compulational Linguis- tics (Volume I: Long Papers), pages 340-350, Mel- bourne. Australia. Association for Computational Linguistics.
Felix Stahlberg and Bill Byrne 2019. On NMT search ertors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP--JCNLPP) pages 3356- 3362. Hong Kong, China. Association for Computa- tional Linguisties.
Maja Popović. 2017. chrF++: words helping charac- ter n-grams. In Proceedings of the Second Confer- ence on Machine Translation, pages 612-618, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Jannis Vamvas and Rico Sennrich. 2022. As little possible, as much as necessary: Detecting over- and undertranslations with contrastive conditioning, In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 490-500, Dublin, Ireland. As- sociation for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Ashwin K Vijayakumar, Michael Cogswell, Ram- prasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural se- quence models. arXiv preprint 111v::6610002244.
Vikas Raunak, Arul Menezes, and Marcin Junczys- Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1172-1188 1183, Online. Association for Computational Linguisties.
Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 11 11-20, Hong Kong, China. Association for Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702 Online. Association for Computational I Linguistics.
Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Mari- anna Martindale, and Marine Carpuat. 2023. Un- derstanding and detecting hallucinations in neural machine translation via model introspection.
Seamless Communication, Loic Barrault, Yu-An Chung. Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar. Hongyu Gong, Kevin Hefferman, John Hoffman. Christopher Klaiber, Pengwei Li. Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guil- laume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ
Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Com- puter Vision ECCV 2014, pages 918-833, Cham. Springer International Publishing.
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan

---

Data source
FLORES
Comments
than once, in which case they are given an oppor- tunity receive detailed feedback and re-do the test. If they do not achieve a passing score of 96% at the second attempt, the vendor is required to find a replacement. Once two annotators are qualified for a given language, one annotator performs the annotations, which are then reviewed by the second annotator.
Selection
U
B
W
U
B
eng. Latn-arb_Aret
18
31
31
22
28
14
arb
Arab-eng Latn
19
31
31
21
31
23
eng Latn-rus_ccc Cyrl
19
31
31
20
32
13
rus Cyrl-eng_ Latn
18
31
30
22
33
24
eng_L.annspp__ar Lam
19
31
31
22
33
17
spa Latn-nng Lam
19
31
31
22
33
24
eng_latm-deu_Ja Latn
19
31
31
22
31
12
deu Latn-enn Latn
18
31
31
21
31
23
Our qualification tests were developed for each of the language directions, and contained 15 items to annotate: 3 full hallucinations, partial halluci- nations, 2 word-level hallucinations, 5 mistransla- tions, and I incomprehensible sentence. The tests were found effective in identifying annotation qual- ity issues before annotators annotate real data.
Lag_Latm Hans
19
31
31
22
33
24
zho Hans-eng Latn
19
31
31
22
33
23
eng Latn-kas Deva
38
36
19
36
32
kas Deva-eng Latn
40
54
57
0
0
eng Latn-yor_ Latn
39
36
27
40
29
yor yor_Latn-eng__1 Lam
.5
0
eng Latn-mni Beng
39
37
27
39
31
mni L_____ennne__naan
54
58
Lww_Latn-spa_Latt spa_Latn-yor__aan
54
58
53
58
Post Post-processing. For cach language, annotations were performed by one annotator and reviewed by another annotator. From the data, we discard the translations marked incomprehensible along with the data with some issues (c.g. unbalanced brackets in the word-level annotations of hallueina- tions or omissions; word-level annotations that sig- nificantly differ from the initial input/output texts). After this filtering, we were left with 144 to 197 annotated sentence pairs per direction.
Table I: Number of sentence pairs selected from each source with each method: uniform sampling U. biased sampling B, and selecting worst cases W.
Ghazvininejad, 2021. Detecting hallucinated content in conditional neural sequence generation. In Find- ings of the Association for Computational Linguis- tics: ACI-IJCNLP 2021, pages 1393-1404, Online. Association for Computational Linguistics.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. 2023. Multilingual machine translation with large language models: Empirical results and analy- sis.
B Attention-based anomaly detection
Reproducibility. The Attn-OT sentence-level de- tection method that we use in Section is our re- production of the Wass-Combo method from Guer- reiro et al. (2022). Their paper did not provide code and training data, so our implementation is not exact. For Wass-to-Unif, we obtained the same ROC AUC scores Guerreiro et al. (2022) on their test set, but for Wass-to- tDta and Wass- Cass-C the AUC scores are 2% lower than in the original pa- per, probably due to the differences in selecting the reference data.
A Dataset Creation
Selecting the data. The 3 sampling strategies described in Section 2.2 were applied in different proportions, depending on what kind of data we had for a particular translation direction. Our re- leased dataset has field with the sampling strategy labels. The resulting proportions are reported in Table 1.
Qualification tests. The annotators recruited for this project were translators and reviewers who participated in FLORES translation (NILB Team et al., 2022) or have other professional translation experience. Typically, these annotators are trans- lators with at least two to three years of profes- sional translation experience, usually with domain expertise in journalism, education, social media or marketing. Two annotators are recruited for each language. They are allowed annotate our data only after passing a specifically developed qualifi- cation test. An annotator can fail the test no more
Reference data for 18 directions. To apply the Atm-OTT method to our data, we created reference translations for each of the translation directions by following steps:
Sample IM sentences for each language from the NLLB mined training data (NLLB Team
et al., 2022);
Translate them with the same settings in Section 2.2:
For each translation direction, drop the result- ing sentence pairs that got into the worst 20% by any of the criteria: shortest-tor to-logestt- ratio

---

for source and translation texts, Seq-LogProb. and LASER3 cosine similarity score between source and translation.
Method
Hallucinations
Omissions
Wass-to-Unil
0.49
0.43
Wass-to-Data
0.53
0.71
After that, about 600K sentence pairs per direction are left reference translations.
Wass-Combo
0.53
0.71
Wass-Mean
0.51
0.51
Computing scores. To compute attention distri- bution, average the encoder-decoder attention maps for the last decoder layer over heads and over target tokens, just like Guerreiro et al. (2022). Our Attn-OT score is then computed with the same for- mula as for Wass-Combo in Guerreiro et al. (2022), with the only slight difference: Swtu is scaled by matching 1% and 99% quantiles of Swid, instead of min-max scaling, improve computational sta- bility. Along with this score, we also evaluate Wass-to-Uniff and Wass-to-Data scores from Guer- reiro al. (2022), and their weighted average with weights inversely proportional to standard devia- tions: Wass-Mean.
Wass-to-Unif" tnif
0.55
0.65
Wass-to-Datar
0.51
0.69
Wass-Combo
0.52
0.69
Wass-Mean*
0.55
0.67
Table 2: ROC AUC scores for detection of hallucina- tions and omissions with OT-based methods, averaged across tranisation directions. The asterisk denotes the scores based on the attention maps with the EOS token excluded.
of the hallucination level and omission levels for each sentence pair.
Qualitatively, the results are similar to those for hallucination detection: internal methods perform equally well for all translation directions, whereas external methods deteriorate for low-resource di- rections. For the joint detection of hallucinations and omissions, BLASER 2.0-QE outperforms all other methods both for high-resource and for low- resource directions.
Dropping the EOS We observed that in the set- ting above, Wass-to-Unit has nearly zero rank cor- relation with hallucination severity. After inspect- ing the attention maps, found that about 75% of attention weight most heads distributed the end-of-sentence token. This probably compensates for the fact that the order of magnitude of its en- coder hidden state an order of magnitude smaller than for other tokens, which aligns with the obser-
D Natural vs Artificially Induced Pathologies
vations of Kobayashi et al. (2020). This makes a standard attention map highly non-uniform, and may obscure the more informative differences be- tween the attention maps for different translations. To mitigate this fact, we computed the second ver- sion of all scores, with dropping the attention to the EOS token and renormalizing it so that the sum attention to the other tokens equals again.
One of the difficulties when dealing with hallucina- tions (and, lesser extent, omissions) is that this is rare phenomenon. Therefore, previous work often resorted to artificially amplifying the prob- lem by applying various perturbations (Lee et al. (2019); Raunak et al. (2021), among others). How- ever, it is not clear whether conclusions made based on this synthetic data would transfer to pathologies generated by model in natural setting. This is especially important when evaluating detection methods: for example, using internal workings of model to detect its pathological behavior does not have to be helpful for pathologies the model did not generate "voluntarily". In this section, we com- pare performance of detection methods between two datasets: (i) our dataset with natural transla- tions and (ii) translations generated under perturba- tion (annotated using the same protocol).
Evaluation Table reports ROC AUC scores for all bT-based detection methods, with and without including the EOS token. Removong the EOS to- ken improves the Wass-to-Unif and Wass-Mean scores, but slightly negatively affects Wass-to-Dan -ata and Wass-Combe scores. Whatever method we use, its performance for hallucination detection is not much better than chance.
Detection of any pathology
Figure reports scores for detecting hallucinations and omissions jointly. The scores are computed as percentage of correctly ranked pairs w.r.t. the worst
Model perturbation. To encourage the model hallucinate or omit source information while still generating fluent text, we decrease the output acti-

---

Internal
External
Seq Logprob
COMET-QE
BL BLASER-QE
Attn Attn-OT -OT
ALTI
ALTI1
LaBSE
LASER
XNLI
English-Arabic
0.71
0.55
0.70
0.56
0.69
0.82
0.81
0.79
0.87
Arabic Arabic-English
0.75
0.68
0.64
0.54
0.78
0.79
0.86
English-
0.58
0.54
0.74
0.74
0.58
0.86
0.75
0.81
0.86
Russian-English
0.63
0.54
0.74
0.76
0.81
0.85
High- resource
English-Spanisth
0.72
0.64
0.73
0.67
0.67
0.81
0.80
0.90
0.83
Spanish-English
0.76
0.69
0.61
0.64
0.81
0.77
0.82
English-Germnn
0.67
0.58
0.80
0.66
0.70
0.86
0.70
0.85
0.87
German-English
0.64
0.66
0.74
0.67
0.68
0.71
0.81
English-Chinese
0.59
0.52
0.84
0.79
0.74
0.89
0.74
0.79
0.87
Enhlishe-Enlish
High-resource mean
0.69
0.62
0.72
0.65
0.71
0.83
0.76
0.79
0.85
English-Kashmir
0.57
Kashmir-English
0.66
0.60
0.53
0.59
0.57
0.57
0.57
0.48
0,70
Low-
English-Yoruba
0.58
0.48
0.83
resource
VrrlubaEngglihh
0.74
0.66
0.61
0.49
0.58
0.67
0.74
0.51
0,77
English-Manipuri
0.56
0.68
0.50
0.79
Manipuri- Engllshh
0.81
0.74
0.63
0.67
0.71
0.60
0.57
0.46
0.83
Low-resource mean
0.67
0.67
0.61
0.63
0.65
0.52
0.79
Zero-
Yoruba-Spankee
0.61
0.63
0.54
0.48
0.51
0.58
0.63
0.47
0.61
shot
Spanish-Yore Yoruba
0.56
0.57
0.53
0.70
0.49
0.70
Overall mean
0.67
0.64
0.68
0.62
0.66
0.75
0.71
0.67
0.81
Figure Results for sentence-level detection of both hallucinations and omissions.
vations of all the encoder-decoder attention layers by a constant multiplier a. Intuitively, this should imitate detachment from the source and increase hallucinatory rate. Indeed, translations generated this way are overall more pathological (see Fig- ure 11). We use a = 0.3 to match the average Scq-logprob of reference translations.
Hall.
Omis.
Natural
Artificial
Natural
Artificial
Log-rog
0.79
0.72
0.56
0.60
Internal
ALTI
0.74
0.83
0.48
0.51
ALTI
0.58
0.69
0.77
0.62
Atto-OT
0.53
0.61
0.70
0.65
COMET-QE
0.75
0.71
0.59
0.58
LaBSE
External
LASER
"Artificial" data might not be informative. Fig- ure 10 shows sentence-level detection scores for different methods of hallucination and omission detection (average over all translation directions). We can see that perturbing the translation model introduces biases into the evaluation of detection methods. For example, for hallucination detec- tion, Seq-Logprob outperforms ALTI on the nat- ural dataset and loses on artificial. For omission detection, ALTIT is the best for the natural dataset, while XNLI is better on the artificial.
XNLI
Figure 10: Sentence- detection scores for natural and artificial (generated under perturbation) pathologies.
formance to come from different distribution pathology types. We can see that even for these cu- rated subsets, the conclusions do not transfer from perturbed to natural pathologies.
Figure 12 shows similar results, but with frac- tions of data downsampled in way that the natural and perturbed data subsets have equal number of observations for each combination of pathology type, source dataset and translation direction. This is done to ensure that differences in detection per-
that data with translations gen- erated under perturbation has be used with cau- tion, especially when evaluating pathology detec- tion methods: the conclusions are likely to not transfer to the natural setting.
Overall,

---

Translations by NLL generated
under
naturally
perturbation
tn.4
Axtn
En-Au
Rv-rr
High resource
Fn-ls
Es-En
En-0
In-Kos
Kas-ön
Dn-Yor
resource
YorEn
MrxEn
Zero-
Werhh
Es-Yor
150
100
50
Hollucination, cartall
Good transition
Figure 11: Annotation results: translations generated
naturally vs under perturbation.
Hall.
Omis.
Natural
Artificial
Natural
Artificial
Seq-Loggrob
0.81 0.76
0.54
0.99
Internal
ALTI
0.77
0.83
0.47
0.51
ALTIT
0.58 0.65
0.76
0.61
Attn-OT
0.50 0.61
0.65
0.6
COMET-QE
0.72 0.72
0.50
0.8
External
LaBSE
0.81 0.83
0.68
0.60
LASER
0.74 0.73
0.61
0.99
XNLI
0.66 0.68
0.59 0.67
Figure 12: Word-Level detection results.
E
Discussion
E.1 Word-Level Detection
Figure 13 shows examples of word-level detection.
Logprob focuses on the beginnings. We notice that log-probability focuses more on the beginning of word or a sentence. This makes sense: model uncertainty in prediction is generally higher when beginning generation.
ALTI focuses on the endings. Differently, token contributions focus on word endings. This again expected: when generating a token that completes a word, source contribution is likely to be lower than for the other tokens. However, the predictions are still very reasonable for the last three examples, ALTI detects omissions and hallucinations more confidently than log-probability. Finally, that feature-based combination of the methods leads more refined results.

---

Input
Logprob
Output
Iim glad Larry Srayson never editad Milppeol,, woutchave had a akiaaoful time.
Masiegra do Lary syson nunca ochars Wilppdiis, hazris porado un Bampo tambiik.
Dios mix.
Komet
Kemes
Speaker Gingrich came percent.
segundo luger Reprever Kattles
Gingrchi
scovery
celebrity many
embsingnn disccuriiiietto
cccoiiitiioo oniuna ookebridad
viruallyy
jPai qaarquu iools hadas?
tratia de projecto ley Mesaamilada in envllmbittd lsogur social
ALTI
Grayson
direacfful
Mo allgro que Laryy Grayson nunca ad tara Witipedia,, habria pasadoo nempo temiiia!
Dioomit:
Somett
Former U.S. Speaker the louse New! Gingrich
second with
dente Cámara mantteil quedó lugerion
Unides, New:
However, the di scovery of hel tomb in 1922 made trim celebrity White whitermary tombs the peet were rereectbeddd this lomndisses vises befl sin only andistortedd
Sinemberrg berger descutr iniento Burbeee 1922 conmott Genunace eu ecridesi
[For que noll haces?
2010-11 0104.56
desomoliaso de
Combined
I'i glaci Larry Graysion never caitedd Wilppeda,, have had creadful tima:
Me aleg que Lany Srayson nunca cahtra Wilpedia, habria pasadoun skmppo tombble.
I:leez
Dios mibi
Somett
Kemet
Gingrich secondwwth
preskdenteen denterd de Cámana dh Regreveri andlende Estadis Unidos, Next Girrich quedló en segundo con porciento.
lowewer the elscovery of tomb 1902 made vim a celebriny (Mirk many tamma the wendrofitee comttwees left wirm diy undisttureed
(Por noll hooos?
Broyect
Figure 13: Examples of word-lever detection for translations from English Spanish. Hallucinated and omitted fragments are underlined.


Reviews:
[{"review_id": "yRNyuRRslE", "reviewer": "Reviewer_TKvD", "summary": "", "questions": "Did you measure annotation agreement and did you consider to resolve or keep annotation variation?\nThe choice of NLLB 600: selecting the smallest model, is this because of inference costs or better chance of finding omissions and hallucinations?\nHow did you combine the 3 sampling strategies? Did you take equal amounts from each of them in the final dataset? Is it interesting to look at the impact of sampling strategy somehow?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments!\n\nRegarding the annotator details: for each language, 2 annotators were recruited by a language service provider. Once recruited, both annotators were given a qualification test. To qualify to work on the project, annotators had to pass the test (passing grade = 96%).\n\nQ1: For each language, annotations were performed by one annotator and reviewed by another annotator. \n\nQ2: We chose NLLB-200-600M model for two reasons: (1) it is the most popular version of the model (by number of likes on HuggingFace), and (2) it would probably produce the largest number of hallucinations.\n\nQ3-Q5: the 3 sampling strategies were applied in different proportions, depending on what kind of data we had for a particular direction. Our released dataset will have the sampling strategy label and we will report the final proportions in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a very much needed dataset that will enable systematic tests of detection methods of hallucinations in machine translations. The experiments also show that a good language coverage is necessary to draw conclusions that generalize. The manual annotation effort is important to avoid artificial settings that do not reflect the real nature of the problem.\n\nThe paper is very well written, clearly presented and the dataset and detection methods are clearly described and discussed. The analyses and discussion contributes valuable information to this important topic and the resources are reusable and very valuable.", "reasons_to_reject": "There is almost no information about the annotation process and the work of the annotators. I would like to see some discussions about the difficulty of the annotation work. I can imagine that there i a lot of variation in opinions on hallucination and I would like to know how the authors dealt with discrepancies and disagreements.", "questions_to_authors": "Did you measure annotation agreement and did you consider to resolve or keep annotation variation?\nThe choice of NLLB 600: selecting the smallest model, is this because of inference costs or better chance of finding omissions and hallucinations?\nHow did you combine the 3 sampling strategies? Did you take equal amounts from each of them in the final dataset? Is it interesting to look at the impact of sampling strategy somehow?", "paper_topic_and_main_contributions": "The paper presents a multilingual dataset and benchmark for the detection of hallucinations and omissions in machine translation. The dataset consists of sentence-level and token-level manual annotations for 18 translation directions providing a valuable novel and comprehensive benchmark. The paper also includes results from known detection methods and also proposes a new metric for the reliable detection of omissions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5vHGcemQA4", "reviewer": "Reviewer_Kt9P", "summary": "", "questions": "I am not sure I totally understand what is the background of the annotators. The authors do say something about it, but I think it should be discussed with more details.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. In the current paper, we focused on using a single model for all translation directions, to facilitate comparison between them and evaluation of internal detection methods. Moreover, the only publicly available model family covering all the languages is NLLB, and adding another NLLB model (that shares training data and some hyperparemeters with NLLB-600M) would probably preserve a lot of model-based bias in the data.\n\nGiven the difficulty and time of the annotation process, we will not be able to add translations of another model in the final version of the current paper. But we are planning to address this in the future research.\n\n\n\nRegarding your question about annotations: Annotators who supported this work are professional translators with at least 2-3 years of professional translation experience, usually with domain expertise in journalism, education, social media and marketing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper should be accepted for EMNLP as it provides a very useful dataset for the research community, which was not available before. I find the annotation guidelines clear and simple. However, I do think more annotation examples should have been provided. The existing examples in Figure 3 are probably not enough.", "reasons_to_reject": "1. The paper should be accepted. However, it does go beyond the 8 pages requirement (the Conclusions section crosses over to the 9th page), which is not allowed in this setting.\n2. The translations annotated in this project, were all generated by the same model. I recommend adding another MT model to make sure the data is not biased.", "questions_to_authors": "I am not sure I totally understand what is the background of the annotators. The authors do say something about it, but I think it should be discussed with more details.", "paper_topic_and_main_contributions": "This paper is about developing a new dataset for detecting hallucinations and omissions in machine translation. The dataset is quite general; it contains 18 low and high resourced language pair directions. The dataset is quite useful, as it provides a way for MT systems to be trained to detect such pathological translation mistakes. The paper shows how existing system which were developed for detecting such pathologies, perform on the new dataset, with interesting results.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rLgi8qsXPs", "reviewer": "Reviewer_XBnn", "summary": "", "questions": "A: How were translators recruited, particularly for the low-resource languages?\n\nB: How might function words and languages' morphology affect token-level hallucination/omission annotations and evaluation scores? E.g. words like \"to\" and \"the\" behave very differently across languages (e.g. sometimes as affixes to content words, or sometimes as standalone words). In many cases, it seems unclear how to annotate function words given the provided hallucination/omission guidelines. This would affect token-level detection scores.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments!\n\nQA: The vendor has a large pool of resources in their community database and the vendor’s Community Management team would screen for these linguists to find those who might be a good match for the project needs. \n\nQB: We are not sure we understand the question, but it could be that the reviewer may allude to the fact that in some directions, the target language uses function words that the source language doesn't use. In that case, the target language would have more \"tokens\" than the source language, which could be confused with a hallucination phenomenon. This being said, annotators were advised to avoid jumping to conclusions whenever they identified erroneous target words. We do not believe that this is the case but do not have specific data on the topic at this point.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset is a useful resource for future work on hallucinations and omissions in machine translation systems. It is particularly interesting to see the differences in existing detection methods' performance between low and high resource language pairs.", "reasons_to_reject": "The main text is longer than 8 pages.\n\nThe dataset only contains hallucination and coverage errors for the 600M distilled NLLB model.\n\nThe limitations section is quite short; it could mention other limitations, e.g. ambiguity in the definition of hallucinations and omissions, difficulties in word-level annotation (e.g. depending on different languages' morphology), focusing on pairs to/from English, etc.", "questions_to_authors": "A: How were translators recruited, particularly for the low-resource languages?\n\nB: How might function words and languages' morphology affect token-level hallucination/omission annotations and evaluation scores? E.g. words like \"to\" and \"the\" behave very differently across languages (e.g. sometimes as affixes to content words, or sometimes as standalone words). In many cases, it seems unclear how to annotate function words given the provided hallucination/omission guidelines. This would affect token-level detection scores.", "paper_topic_and_main_contributions": "The paper presents a new dataset of machine translation hallucination and omission errors, including sentence-level and token-level annotations for eighteen translation directions involving nine languages. They evaluate multiple hallucination and omission detection methods on the dataset, finding that existing detection methods perform much better on high-resource language pairs. Sequence log-probability is best for sentence-level hallucination detection, and token attribution methods are best for omission detection or token-level detection.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a multilingual dataset and benchmark for the detection of hallucinations and omissions in machine translation.
The dataset is quite general; it contains 18 low and high resourced language pair directions. The dataset is quite useful, as it provides a way for MT systems to be trained to detect such pathological translation mistakes. The authors evaluate multiple hallucination and omission detection methods.  The paper shows how existing system which were developed for detecting such pathologies, perform on the new dataset, with interesting results.
Decision: Accept-Main