Venue: NAACL 2025

# Mitigating Hallucinated Translations in Large Language Models with

Hallucination-focused Preference Optimization

 Zilu Tang\({}^{*}\)

Boston University

ziltang@bu.edu &Rajen Chatterjee Apple rajen_c@apple.com &Sarthak Garg A Apple sarthak_garg@apple.com

###### Abstract

Machine Translation (MT) is undergoing a paradigm shift, with systems based on fine-tuned large language models (LLM) becoming increasingly competitive with traditional encoder-decoder models trained specifically for translation tasks. However, LLM-based systems are at a higher risk of generating hallucinations, which can severely undermine user's trust and safety. Most prior research on hallucination mitigation focuses on traditional MT models, with solutions that involve _post-hoc_ mitigation -- detecting hallucinated translations and re-translating them. While effective, this approach introduces additional complexity in deploying extra tools in production and also increases latency. To address these limitations, we propose a method that intrinsically learns to mitigate hallucinations during the model training phase. Specifically, we introduce a data creation framework to generate hallucination focused preference datasets. Fine-tuning LLMs on these preference datasets reduces the hallucination rate by an average of \(96\%\) across five language pairs, while preserving overall translation quality. In a zero-shot setting our approach reduces hallucinations by \(89\%\) on an average across three unseen target languages.

1
Footnote 1: Work done during internship at Apple

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 3410–3433

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

LLMs are gaining popularity for various NLP applications, including machine translation. Fine-tuning LLMs for MT has been proven to be highly data-efficient, requiring orders of magnitude less parallel data than large standalone multilingual MT models, while achieving increasingly competitive performance [7, 10, 1]. Moreover, there is a significant and ongoing effort within the research community to push the performance limits of foundational LLMs and expand their multilingual capabilities [11, 12, 13].

Despite these advantages, LLM-based models are more prone to _hallucinations_: the models generates information that is inaccurate or entirely fabricated. This issue has lead to a growing research area, focusing on the causes, detection, and mitigation of hallucinations in LLMs [14]. In the context of MT, hallucinations manifest as highly pathological translations, which can lead to misunderstandings in conversations, potentially damaging relationships and undermining user trust in the system [15].

Most of the existing research on hallucination mitigation in MT has focused on traditional encode-decoder models, establishing effective post-hoc mitigation strategies [16, 17, 18]. These strategies first detect whether a translation contains hallucination, and if so, generate and present a _mitigated_ translation to the user. In practical scenarios, using post-hoc mitigation has several drawbacks: i) the need for deploying an additional hallucination detector in production; ii) running the hallucination detector on every translation, which increases cost and latency; and iii) re-running inference if a translation hallucinates (which often much slower than regular inference).

To address these issues, we propose a framework that intrinsically integrates hallucination mitigation during the LLM development phase, aiming to minimize hallucinations from the outset. Specifically, we apply post-hoc mitigation strategies _offline_ on a large-scale monolingual corpus, generating a corpus of model hallucinations alongside their corresponding mitigated translations. We then fine-tune the LLM using Contrastive Preference Optimization (CPO) [13], guiding the model away from hallucinations.

Our approach requires no additional human-annotated data, is easily scalable across many language pairs, and is highly effective -- achieving a \(96\%\) reduction in hallucination rates across five

language pairs without sacrificing general translation quality. It also generalizes well, achieving an average \(89\%\) reduction in hallucination rates across three unseen target languages. Overall, our main contributions include:

* Proposing a novel approach for creating hallucination-focused preference datasets.
* Identifying the most effective fine-tuning technique for leveraging this preference dataset.
* Exploring the cross-lingual generalization capabilities of the fine-tuned models in a zero-shot setting.
* Determining the most effective post-hoc mitigation strategies for LLM based translation models.

## 2 Dataset Creation Framework

One of the techniques for fine-tuning LLMs for translation is preference optimization Xu et al. (2024) which uses a dataset of triplets, consisting of a source sentence \(x\), its preferred translation \(y_{p}\), and a dispreferred translation \(y_{d}\). Preference optimization trains the model to prioritize the generation of preferred set of translations over dispreferred ones. Xu et al. (2024) focus on optimizing general translation quality, and hence in their datasets, \(y_{p}\) and \(y_{d}\) differ only in quality and do not explicitly consider the notion of hallucination. For instance, both translations could be broadly correct, but one might be preferred over the other due to minor errors or subtle differences in style.

To address hallucinations, we develop a framework for automatically creating a _hallucination focused_ preference dataset and propose to fine-tune the LLM on this dataset to effectively mitigate hallucination generation. In this dataset, the dispreferred translations contain hallucinations, whereas the preferred translations do not. The set of dispreferred translations are derived from the LLM's own generated outputs. This is particularly important as it enables the model to learn from its own errors and correct them. Our approach for creating this preference dataset is completely unsupervised and can easily scale to multiple languages without any human annotation. At a high level, the dataset creation process consists of translating monolingual data using the LLM and automatically detecting hallucinations (Section 2.1) and mitigating them using existing post-hoc methods (Section 2.2)

### Hallucination Detection

In the first step, we construct a set of source sentences and their corresponding dispreferred translations containing hallucinations. To achieve this, we translate publicly available monolingual corpora \(\mathcal{D}_{m}\) from the source language into the target languages using the model \(\mathcal{M}\), which we aim to fine-tune for reducing hallucinations. We then automatically identify translations \(y~{}(y\coloneqq\mathcal{M}(x))\) that exhibit hallucination using the state-of-the-art hallucination detector model based on BLASER 2.0-QE Chen et al. (2023); Dale et al. (2023b). BLASER 2.0-QE is a reference-free machine translation quality estimation metric that predicts cross-lingual semantic similarity between a source sentence \(x\) its translation \(y\). It operates on a scale of \(1\)-\(5\), where \(1\) denotes completely unrelated sentences and \(5\) signifies fully semantically equivalent sentences. We re-normalize the BLASER score to a hallucation score (HS), with a higher value indicating a greater likelihood of hallucination in \(y\):

\[\texttt{HS}(x,y)=1-\frac{\texttt{BLASER}(x,y)}{5} \tag{1}\]

After fixing a threshold \(T\), we classify a translation as containing hallucination if its hallucination score exceeds the threshold. Collecting such instances where hallucinations are detected provides us with a hallucination dataset \(\mathcal{D}_{h}\), which consists of source sentences and their corresponding hallucinated translations as follows:

\[\mathcal{D}_{h}\coloneqq\left\{(x,y)\mid\texttt{HS}(x,y)\geq T\,\forall\,x \in\mathcal{D}_{m}\right\} \tag{2}\]

### Post-hoc HALUcination Mitigation

The second step involves mitigating the hallucinated translations in \(\mathcal{D}_{h}\) to create hallucination-free alternatives. Previous works Dale et al. (2023a); Guerreiro et al. (2023a) have proposed several post-hoc mitigation strategies, though they are typically applied during test time. In contrast, we explore using these strategies offline to build a preference fine-tuning corpus. We consider a few notable strategies, outlined below:

**Fallback System**Guerreiro et al. (2023a) demonstrated that simply switching to a different fallback translation system when hallucinations occur is an effective mitigation strategy. Following this, we employ the NLLB-3.3B model NLLB Team et al. (2022) as a fallback.

 

Candidate Generation and SelectionDale et al. (2023) propose generating multiple alternative translation candidates from the original model and selecting one of them as the mitigated translation based on a specific criterion. This approach involves two degrees of freedom: (i) candidate generation and (ii) candidate selection. To generate \(n\) candidates we explore the following strategies:

* **MC beam**: Using \(n\) iterations of beam search with Monte Carlo dropout Gal and Ghahramani (2016).
* **Temperature sampling**: Sampling from the full probability distribution, adjusted by a temperature parameter \(t\), to control the sharpness of the distribution.
* **Nucleus sampling**: Sampling from a set of tokens that covers top \(p\)% of the posterior probability distribution at each step Holtzman et al. (2020).
* **Epsilon sampling**: Sampling from a set of tokens where each token has a probability greater than or equal to a threshold \(\epsilon\) Hewitt et al. (2022); Freitag et al. (2023).

To select the best candidate, we explore the following algorithms:

* **MBR decoding**: Selects the candidate that maximizes the average utility with respect to all other candidates. Kumar and Byrne (2004); Freitag et al. (2022). We evaluate utility between two candidates using chrF Popovic (2015), LaBSE Feng et al. (2022), and COMET Rei et al. (2022).
* **Re-ranking**: Selects the candidate that maximizes utility with respect to the source sentence, using LaBSE and COMET Rei et al. (2020) as utility metrics.

We compare the effectiveness these strategies in mitigating hallucinations, analyzing the impact of different generation and sampling methods in Section 5.

We select the best mitigation strategy based on a held out development set and use it to generate alternative translations \(\tilde{y}\) corresponding to each sample \((x,y)\in\mathcal{D}_{h}\). We construct our hallucination focused preference fine-tuning dataset \(\mathcal{D}_{p}\) by retaining samples where the alternative translation successfully mitigates hallucination \((\mathsf{HS}(x,\tilde{y})<T)\). Formally \(\mathcal{D}_{p}\) is defined as follows:

\[\mathcal{D}_{p}:=\{(x,\tilde{y},y)\mid\mathsf{HS}(x,\tilde{y})<T\,\forall\,( x,y)\in\mathcal{D}_{h}\} \tag{3}\]

## 3 Fine-tuning Using CPO

We fine-tune the baseline LLM \(\mathcal{M}\) using our hallucination-focused preference dataset \(\mathcal{D}_{p}\) through CPO, a variant of Direct Preference Optimization DPO Rafailov et al. (2023), which has shown to be effective for fine-tuning LLMs on the translation task. The CPO objective is formally defined as follows:

\[\mathcal{L}_{CPO}=\mathcal{L}_{NLL}+\mathcal{L}_{P} \tag{4}\]

where

\[\mathcal{L}_{P}=-\mathbb{E}_{(x,y_{p},y_{d})\sim\mathcal{D}_{p}}\log\sigma( \beta\log\frac{\pi_{\theta}(y_{p}|x)}{\pi_{\theta}(y_{d}|x)}) \tag{5}\]

\[\mathcal{L}_{NLL}=-\mathbb{E}_{(x,y_{p},y_{d})\sim\mathcal{D}_{p}}\log\pi_{ \theta}(y_{p}|x) \tag{6}\]

In equations above, \(x\), \(y_{p}\) and \(y_{d}\) represent the source sentence, preferred (hallucination free) translation and dispreferred (hallucination containing) translation, respectively, sampled from the preference dataset \(\mathcal{D}_{p}\). The policy \(\pi_{\theta}\) refers to the conditional probability distribution from the model \(\mathcal{M}\), \(\sigma\) is the sigmoid function and \(\beta\) is a scaling hyperparameter from Rafailov et al. (2023).

The CPO objective combines the standard negative log-likelihood NLL loss, which encourages the model to generate \(y_{p}\), and the preference loss \(\mathcal{L}_{p}\), which aims to increase the probability gap between \(y_{p}\) and \(y_{d}\). The preference loss term explicitly instructs the model to prioritize the generation of \(y_{p}\) and reject \(y_{d}\). In Section 6.2 we show that this loss term is crucial for reducing the model's likelihood of generating hallucinations.

In our dataset, we ensure that \(y_{p}\) always has higher quality than \(y_{d}\), as measured by hallucination score \((\mathsf{HS}(x,y_{p})<T\) and \(\mathsf{HS}(x,y_{d})\geq T)\). However different preference pairs may exhibit varying quality gaps. To account for this variation in quality gaps in the preference fine-tuning, we introduce a scaling term to \(\mathcal{L}_{P}\). A preference pair \((y_{p},y_{d})\) with larger quality gap provides a more informative data point, so we design the scaling term to assign greater weight to pairs with a larger gaps, proportional to the quality ratio of \(y_{p}\) and \(y_{d}\). With this scaling term, the modified preference loss (\(\mathcal{L}^{\prime}_{p}\)) is defined as follows2: 

\[{\mathcal{L}^{\prime}}_{p}= -\mathbb{E}_{(x,y_{p},y_{d})\sim D_{p}}\left[\log\sigma(\beta\log \tfrac{\pi_{\theta}(y_{p}|x)}{\pi_{\theta}(y_{d}|x)}\right. \tag{7}\] \[+\beta\log\frac{\phi(x,y_{p})}{\phi(x,y_{d})})\right]\]

where, \(\phi\) is a scoring function that measures the quality of a translation given the source. We choose \(\phi\) to be the hallucination score (H5). With this change, our final CPO loss is shown in equation 8

\[{\mathcal{L}^{\prime}}_{CPO}={\mathcal{L}^{\prime}}_{p}+{\mathcal{L}}_{NLL} \tag{8}\]

## 4 Experimental Setup

### Evaluation Metrics

Given a model \(\mathcal{M}\), we evaluate it on a monolingual dataset \(\mathcal{D}\) using _hallucination rate_. Hallucination rate (HR) computes the ratio of source sentences for which model produces translations containing hallucinations:

\[\texttt{HR}(\mathcal{M},\mathcal{D})=\frac{\lfloor\{x\mid\texttt{HS}(x, \mathcal{M}(x))\geq T~{}\forall x\in\mathcal{D}\}\rfloor}{\lvert\mathcal{D} \rvert} \tag{9}\]

where \(\lvert\cdot\rvert\) counts the number of elements in a set.

We split the monolingual corpus \(\mathcal{D}_{m}\) into \(\mathcal{D}_{m}^{train}\) (train), \(\mathcal{D}_{m}^{dev}\) (dev) and \(\mathcal{D}_{m}^{test}\) (test) sets. The hallucination-focused preference dataset (\(\mathcal{D}_{p}\)) is derived from \(\mathcal{D}_{m}^{train}\) as described as Section 2. We evaluate the baseline and fine-tuned LLMs using hallucination rates computed against unseen set \(\mathcal{D}_{m}^{test}\). All the hyperparameters and the best post-hoc mitigation strategy for preparing the fine-tuning set are selected based on \(\mathcal{D}_{m}^{dev}\).

To ensure that improvements in hallucination mitigation do not come at the expense of general translation quality, we also evaluate the baseline and fine-tuned models on the WMT'22 and WMT'23 testsets using three COMET models: wmt22-comettiwi-da, wmt23-comettiwi-da-xxl, and XCOMET-XXL. This evaluation methodology aligns with that of Xu et al. (2024).

### Baseline Model and Language Coverage

We choose ALMA-7B-R as our baseline LLM. Built upon LLAMA-2 (Touvron et al., 2023), ALMA-7B-R has been extensively optimized for translation through multiple rounds of fine-tuning, including continued pre-training on multilingual data, supervised fine-tuning with parallel corpora and preference tuning using CPO. ALMA-7B-R has shown competitive performance, matching or surpassing top systems in WMT shared evaluation, and even GPT-4 (OpenAI et al., 2024), making it a strong baseline for our hallucination mitigation experiments.

ALMA-7B-R supports translation across ten language directions: English\(\leftrightarrow\){Czech (_cs_), German (_de_), Icelandic (_is_), Russian (_ru_) and Chinese (_zh_)}. However due to resource constraints, in our study, we focus on a subset of five language pairs: _en\(\rightarrow\){cs, de, is, ru, zh_}.

### HALUication Focused Preference Dataset Construction

We follow the data creation framework outlined in Section 2 to construct a hallucination focused preference fine-tuning dataset, as detailed below:

#### 4.3.1 Monolingual Data

As our study is restricted to language pairs with English as source, we randomly sample English sentences from the NewsCrawl dataset (Kocmi et al., 2022)3 for \(\mathcal{D}_{m}\). We sample \(0.5\)M sentences each for \(\mathcal{D}_{m}^{dev}\) and \(\mathcal{D}_{m}^{test}\), and these evaluation sets are shared across all language pairs. To create preference sets for each language pair, we sample separate \(\mathcal{D}_{m}^{train}\) sets, with sizes of \(2\)M (_en\(\to\)zh_), \(5\)M (_en\(\to\)cs_, _enh\(\to\)is_, _en\(\to\)ru_), or \(10\)M (_en\(\to\)de_) sentences. The sizes are determined based on hallucination rates of the baseline model for each language pair, with larger sets allocated to language pairs exhibiting lower hallucination rates, ensuring that the resulting preference sets are of comparable sizes across all language pairs. All the above datasets are cleaned by applying a series of filters to remove noisy samples.4

Footnote 3: [https://data.statmt.org/news-crawl/](https://data.statmt.org/news-crawl/) (\(2023\) release)

Footnote 4: Appendix A provides more information on the filtering process, and monolingual data statistics.

#### 4.3.2 Hallucination Detection

As outlined in Section 2.1, for each language pair, we translate the corresponding \(\mathcal{D}_{m}^{train},\mathcal{D}_{m}^{dev},\mathcal{D}_{m}^{test}\) sets using the baseline ALMA-7B-R into the target language. We then create the corresponding hallucination datasets \(\mathcal{D}_{h}^{train},\mathcal{D}_{h}^{dev},\mathcal{D}_{h}^{test}\) by retaining translations where hallucination score exceeds the threshold \(T\). We set \(T\) to be \(0.5\) based on manual verification of the resulting \(\mathcal{D}_{h}^{dev}\) sets for _en\(\rightarrow\)zh_ and _en\(\rightarrow\)de_. Native chinese and german speakers verified that \(97\%\) and \(87\%\) of translations in the _en\(\rightarrow\)zh_ and _en\(\rightarrow\)de_ sets, respectively, did contain highly pathological errors. Consequently, this 

threshold is adopted for all language pairs throughout our study, unless otherwise specified. The number of samples in hallucination datasets for each split and language pair, along with the corresponding hallucination rates (%) are summarized in Table 1. For additional analysis on hallucination patterns see Appendix 6.4. Our experiments indicate that hallucinations occur on different source sentences for different languages, and the presence of specific features (e.g. quotes, urls, all cap phrases) could significantly increase the likelihood of hallucination.

#### 4.3.3 Post-hoc Hallucination Mitigation

We evaluate the post-hoc mitigation strategies described in Section 2.2 on \({\mathcal{D}_{h}^{dev}}\). Given a sample \((x,y_{d})\in{\mathcal{D}_{h}^{dev}}\), where \(y_{d}\) contains hallucinations, each mitigation strategy \(\mathcal{S}\) attempts to generate an alternative translation \(\tilde{y}\coloneqq\mathcal{S}(x)\) which is likely free of hallucinations. We evaluate these strategies using _mitigation rate_ (MR), which is the ratio of samples where \(\tilde{y}\) successfully mitigates hallucinations. Higher MR values indicate better performance.

\[\texttt{MR}(\mathcal{S},{\mathcal{D}_{h}})=\frac{|\{x\mid\textsc{HS}(x, \mathcal{S}(x))<T~{}\forall(x,y_{d})\in{\mathcal{D}_{h}}\}|}{|{\mathcal{D}_{ h}}|} \tag{10}\]

For the _Fallback_ strategy, we use a beam size of \(40\). For _Candidate Generation and Selection_ approach, we generate \(n=40\) candidates using temperature sampling with \(t\in\{0.8,1,1.5\}\) in conjunction with either nucleus sampling with \(p=0.9\) or epsilon sampling with \(\epsilon=0.02\). For MCBeam, we generate candidates using a beam size of \(5\). When using COMET with MBR, we use eamt22-cometinho-da which is a distilled model that takes as input the source sentence, translation and reference translation. For COMET with Re-ranking, we employ the wmt20-comet-qe-da, which only takes the source sentence and translation as input.

A detailed comparison of the mitigation strategies is presented in Section 5.1. We use the best performing strategy (re-ranking using LaBSE) to construct our preferences datasets \({\mathcal{D}_{p}^{train}}\) from \({\mathcal{D}_{h}^{train}}\) for all language pairs as described in Section 2.2. The number of samples in these preference datasets across all language pairs is presented in Table 2.5

Footnote 5: Detailed statistics comparing the hallucination scores and lengths of preferred vs. dispreferred translations can be found in Appendix D, and example preference pairs in Appendix I.

### Combining Hallucination and Translation Quality Preference Datasets

While \({\mathcal{D}_{p}^{train}}\) is specifically constructed to mitigate hallucinations, fine-tuning solely on this dataset can lead to a decline in general translation quality. To address this, we mix \({\mathcal{D}_{p}^{train}}\) with \({\mathcal{D}_{alma}^{train}}\), the preference dataset originally used to fine-tune the baseline ALMA-R model by Xu et al. (2024), which focuses on overall translation quality. Combining the two sets helps preserve the original translation quality while improving hallucination mitigation. \({\mathcal{D}_{atma}^{train}}\) is comparable in size to \({\mathcal{D}_{p}^{train}}\), with detailed statistics provided in Table 18

### Fine-tuning Using CPO

We adhere to a fine-tuning setup that closely follows the methodology described in Xu et al. (2024). In line with their approach, we fine-tune LoRA adapters Hu et al. (2022) and utilize the same prompt structure.6 For the modified preference loss function (\({\mathcal{L}_{p}^{\prime}}\)), we use HS as the scoring function \(\phi\) for \({\mathcal{D}_{p}^{train}}\) and COMET as the scoring function for the \({\mathcal{D}_{adma}^{train}}\) dataset.7 We normalize the scoring functions of both datasets to ensure their ranges align with each other.

Footnote 6: Details on hyperparameters are available in Appendix B.

Most hyper-parameters are optimized based on the hallucination rate of the fine-tuned model on the smaller \({\mathcal{D}_{h}^{dev}}\) sets to facilitate quick iterations. However, when multiple configurations yield similar results, we decide based on the full development set \(D_{m}^{dev}\).

## 5 Results

We present the comparison between different post-hoc mitigation strategies in Section 5.1 and main results of our fine-tuned models in Section 5.2

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & \({\mathcal{D}_{h}^{train}}\) & \({\mathcal{D}_{h}^{dev}}\) & \({\mathcal{D}_{h}^{test}}\) \\ \hline _en\(\rightarrow\)cs_ & \(2085\) (\(0.04\)) & \(202\) (\(0.04\)) & \(179\) (\(0.04\)) \\ _en\(\rightarrow\)de_ & \(673\) (\(0.01\)) & \(47\) (\(0.01\)) & \(39\) (\(0.01\)) \\ _en\(\rightarrow\)is_ & \(3682\) (\(0.08\)) & \(384\) (\(0.08\)) & \(388\) (\(0.08\)) \\ _en\(\rightarrow\)ru_ & \(1933\) (\(0.04\)) & \(186\) (\(0.04\)) & \(196\) (\(0.04\)) \\ _en\(\rightarrow\)zh_ & \(8470\) (\(0.45\)) & \(2178\) (\(0.46\)) & \(2192\) (\(0.46\)) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Hallucination count (HR in %) for ALMA–7B-R.

 



### Post-hoc Mitigation Strategies

Table 3 summarizes the mitigation rates of various strategies across different selection methods and utility metrics, focusing on the top performing sampling settings.8 All strategies significantly reduce hallucinations, with even the worst performing one achieving an average mitigation rate of over \(96\%\). The optimal setting, achieved through epsilon sampling with \(\epsilon=0.02\) followed by reranking with LaBSE, results in an impressive average mitigation rate of \(99.6\%\). Notably, we observe that for both MBR and Re-rank, LaBSE consistently outperforms COMET. This aligns with previous research on hallucination detection, which has shown LaBSE to be superior to COMET[14]. Furthermore, model-based metrics for MBR, such as COMET and LaBSE outperform chrF. Comparing both candidate selection methods overall, Re-rank outperforms MBR. The Fallback strategy using NLLB-3.3B achieves a mitigation rate of \(96.5\%\). While quite substantial, it falls short of the best results, possibly due to the baseline ALMA-7B-R being a stronger model, generating higher quality and more diverse translations.9

Footnote 8: Table 14 in Appendix shows several sampling methods.

Footnote 9: Comparison of NNLB-3.3B and ALMA-7B-R on general translation quality is shown in Table 25, 26 in Appendix.

### Fine-tuning Using CPO

We present the main results in Table 4.10 Our primary baseline, ALMA-7B-R, achieves an average hallucination rate of \(0.127\)%. ALMA-7B-R is a much stronger baseline compared to traditional encoder-decoder based NLLB-3.3B, which exhibits an average hallucination rate of \(1.73\)%, nearly \(14\) times higher than that of ALMA-7B-R. This difference is expected, given that ALMA-7B-R is a stronger translation model, as reflected by its superior average COMET scores on the WMT testsets. Examining the hallucination rates across all language pairs, we observe that the _en_-\({}^{*}\!\!zh\) language pair consistently shows the highest hallucination rates across all the models.

Footnote 10: Individual COMET model scores for WMT’22 and WMT’23 across each language pair are detailed in Table 25, 26, 27, 28.

Next, we analyze the results obtained by fine-tuning ALMA-7B-R on different preference datasets. Fine-tuning using our hallucination focused preference dataset \(\mathcal{D}_{p}^{train}\), gives us model \(\mathcal{M}_{p}\). The hallucination rate of this model drops significantly from \(0.127\%\) to an average of \(0.005\%\). This demonstrates the effectiveness of our unsupervised preference data creation approach, resulting in a remarkable \(96\)% reduction. We additionally confirm the effect of the hallucination mitigation in Appendix G with a top-n-gram based hallucination detector[10]. However, along the reduction in hallucinations, we observe a decline in general translation quality, with the average COMET score dropping by \(1.0\) from the baseline.

To mitigate this drop in translation quality, we fine-tune the model using a combined dataset \(\mathcal{D}_{p}^{train}\cup\mathcal{D}_{alma}^{train}\), which gives us model \(\mathcal{M}_{p+a}\). By balancing training between hallucination mitigation and general translation tasks, we observe an improvement of \(0.8\) points in the average COMET

score, bringing the model nearly on par with the baseline performance, while still maintaining hallucination rate of \(0.005\%\). For more detailed general translation quality comparisons, refer to Section L in the Appendix. Examples of hallucinated translations from the baseline model, which are mitigated by our fine-tuned model, are shown in the Appendix J. To establish an upper bound, we apply the best post-hoc mitigation strategy to the hallucinations from the baseline ALMA-7B-R model, reporting this as ALMA-7B-R + _post-hoc_ in Table 4. This represents using the post-hoc mitigation system during test time. Our findings indicate that our best model, with a hallucination rate of \(0.005\)%, comes very close to this upper bound of \(0.002\)%, without requiring any additional mitigation systems at test time.

## 6 Analysis and Discussions

### Cross-lingual Zero-shot Generalization

To assess the cross-lingual generalization of our fine-tuning approach in reducing hallucinations on unseen language pairs, we conducted zero-shot experiments comparing baseline ALMA-7B-R with our best fine-tuned model (\({\cal M}_{p+a}\)) in a zero-shot setting. In these experiments, we translated our test set \({\cal D}^{test}_{m}\) from English into three target languages -- French (_fr_), Italian (_it_), and Spanish (_es_), none of which were prominently present in the pre-training and fine-tuning stages of ALMA-7B-R.

Table 5 presents the hallucination rates and COMET scores for both models across these language pairs.11 Notably, both models perform well, despite the target languages being unseen during training. The baseline model achieves an average COMET score of \(83.31\), with the fine-tuned model trailing slightly at \(83.17\). However, in terms of hallucination rates, the fine-tuned model significantly outperforms the baseline, reducing the average hallucination rate from \(0.273\)% to \(0.03\)%, representing an \(89\)% reduction. These results demonstrate that our fine-tuning approach generalizes effectively to unseen language pairs, substantially reducing hallucinations without significant loss in general translation quality.

Footnote 11: The COMET scores were computed using the _wmt22-comethivi-da_ model.

### Ablation of Loss Function Components

As shown in equation 8, the CPO loss consists of two components: i) preference loss and ii) NLL loss. We conduct an ablation study to understand the contribution of each component. When only the NLL loss is active, it corresponds to supervised fine-tuning (SFT) on the source and mitigated translations. We optimized the hyperparameters corresponding for each loss configuration based on hallucination rates on the \({\cal D}^{dev}_{h}\) set and then evaluated both the baseline and the fine-tuned models on the full \({\cal D}^{dev}_{m}\) set.

Table 6 summarizes the results of these ablations. The findings reveal that using only the preference loss results in poor performance, with a hallucination rate of \(3.556\%\), which is significantly worse than the baseline ALMA-7B-R (\(0.127\%\)). In contrast, using only the NLL loss yields a lower hallucination rate of \(0.078\%\), outperforming the baseline. However, the best performance is achieved when both losses are combined, reducing the hallucination rate to just \(0.005\%\). This demonstrates the effectiveness of the CPO loss over simple SFT using mitigated translations, highlighting the complementary benefits of preference and cross-entropy losses.

### Ablation of Data Quantity vs. Quality

To create \({\cal D}^{train}_{p}\), we select dispreferred translations with a hallucination score \(\geq 0.5\). Lowering this threshold yield more training samples, but risks including translations that do not accurately reflect true hallucinations, thus reducing the quality of the preference dataset. To explore the tradeoff between data quantity and quality, we conducted an experiment by creating a version of \({\cal D}^{train}_{p}\) with a lower threshold of \(0.45\). We fine-tuneed the baseline on

\begin{table}
\begin{tabular}{c|r|c c c} \hline  & \multicolumn{3}{c}{\({\cal M}_{p}\)} \\  & \multirow{2}{*}{ALMA–7B–R} & \multicolumn{3}{c}{\({\cal L}^{\prime}_{P}\)} & \({\cal L}_{NLL}\) & \({\cal L}^{\prime}_{CPO}\) \\ \cline{1-1} \cline{3-5} \cline{5-5} \multicolumn{1}{l|}{\(en\!\rightarrow\!\!cs\)} & \(202\) & \(10\) & \(216\) & \(7\) \\ \(en\!\rightarrow\!\!de\) & \(47\) & \(5\) & \(124\) & \(1\) \\ \(en\!\rightarrow\!\!is\) & \(384\) & \(72\) & \(441\) & \(37\) \\ \(en\!\rightarrow\!\!ru\) & \(186\) & \(83836\) & \(127\) & \(2\) \\ \(en\!\rightarrow\!\!zh\) & \(2178\) & \(174\) & \(931\) & \(73\) \\ \hline Avg. HR (\%) & \(0.127\) & \(3.556\) & \(0.078\) & \(0.005\) \\ \hline \end{tabular}
\end{table}
Table 6: Hallucination counts (HR in %) of ALMA-7B-R and \({\cal M}_{p}\) using different loss variants on \({\cal D}^{dev}_{m}\).

\begin{table}
\begin{tabular}{c|c c|c c} \hline  & \multicolumn{3}{c|}{HR \% (L)} & \multicolumn{3}{c}{COMET (\(\uparrow\))} \\ \hline  & ALMA-7B-R & \({\cal M}_{p+a}\) & ALMA-7B-R & \({\cal M}_{p+a}\) \\ \hline _en\(\!\rightarrow\!\!es\)_ & \(0.164\) & \(0.007\) & \(83.30\) & \(83.25\) \\ _en\(\!\rightarrow\!\!f\)_ & \(0.399\) & \(0.077\) & \(83.05\) & \(82.39\) \\ _en\(\!\rightarrow\!\!i\)_ & \(0.256\) & \(0.007\) & \(83.57\) & \(83.87\) \\ \hline Average & \(0.273\) & \(0.030\) & \(83.31\) & \(83.17\) \\ \hline \end{tabular}
\end{table}
Table 5: Cross-lingual zero-shot results.

 

both versions of the preference dataset and evaluated the models on \({\cal D}_{m}^{dev}\). As shown in Table 7, lowering the threshold to increase the dataset size led to a decline in performance, indicating that the quality of the preference data is more crucial than its quantity.

### Hallucination Characterization

To gain a deeper understanding of the nature of hallucinations, we conducted a detailed analysis of the source sentences and the corresponding hallucinated translations on the test set \({\cal D}_{h}^{test}\)

Source sentencesWe examined source sentences to identify any patterns that might consistently trigger hallucinations when translating to different target languages. Table 8 presents these statistics of the overlap of source sentences between hallucination samples of different language pairs. For e.g., in the \(en\text{$\rightarrow$}zh\) language pair, \(2178\) source sentences generate hallucinations, however only \(5\text{-}19\) of source sentences result in hallucinations when translating other target languages. A similar trend is observed across all language pairs. This indicates that the source sentences do not exhibit strong patterns that trigger hallucinations across different tanguages.

Manual analysis of the examples also show a trend that presence of quotes, urls/online handles, or words/phrases in all capital letters in the source sentence triggers hallucinations. In Table 9 we perform a chi-squared test to test whether the presence of such features has a statistically significant impact on triggering hallucinations in the baseline model. We find that different language pairs have different source triggers.

TranslationsIn our analysis of hallucinated translations, we observed a substantial number of oscillatory hallucinations, characterized by repetitive sequences within the translation. These oscillatory hallucinations can be effectively identified using a top n-gr am based hallucination detector Rauzak et al. (2021, 2022); Guerreiro et al. (2023c, a. This detector flags a translation as a hallucination if the count of the top n-gram in the translation exceeds that of the source by a specified threshold. Based on prior works, we set n-gram to \(4\) and the threshold to \(2\). We find that \(60\%\) to \(80\%\) of the hallucinations were oscillatory in nature. The statistics for all language pairs are presented in Table 10.

### Evaluation at Different Hallucination Score Thresholds

Our main evaluation results in Table 4 use a hallucination score threshold of \(0.5\). This threshold is also applied to create hallucination focused preference datasets. To assess whether our approach is biased toward this threshold, we re-evaluated both the baseline (ALMA-7B-R) and our best fine-tuned model (\({\cal M}_{p+a}\)) at a few lower thresholds. It's important to note that as we lower the threshold, the distinction between hallucination and nonhallucination becomes increasingly blurred. However, a well-tuned model should still show improved performance over the baseline. Table 11 presents the evaluation results at different hallucination score thresholds (\(0.5\), \(0.45\), and \(0.4\)). While our \({\cal M}_{p+a}\) consistently outperforms ALMA-7B-R across all thresholds, the performance gap decreases as the threshold is lowered.

\begin{table}
\begin{tabular}{l|r r r r r} \hline  & \multicolumn{1}{c}{\(en\text{$\rightarrow$}cs\)} & \multicolumn{1}{c}{\(en\text{$\rightarrow$}de\)} & \multicolumn{1}{c}{\(en\text{$\rightarrow$}is\)} & \multicolumn{1}{c}{\(en\text{$\rightarrow$}ru\)} & \multicolumn{1}{c}{\(en\text{$\rightarrow$}zh\)} \\ \hline \(en\text{$\rightarrow$}cs\) & \(202\) & \(3\) & \(9\) & \(10\) & \(17\) \\ \(en\text{$\rightarrow$}de\) & \(3\) & \(47\) & \(3\) & \(2\) & \(7\) \\ \(en\text{$\rightarrow$}is\) & \(9\) & \(3\) & \(384\) & \(10\) & \(16\) \\ \(en\text{$\rightarrow$}ru\) & \(10\) & \(2\) & \(10\) & \(186\) & \(17\) \\ \(en\text{$\rightarrow$}zh\) & \(17\) & \(7\) & \(16\) & \(17\) & \(2178\) \\ \hline \end{tabular}
\end{table}
Table 8: Number of common source sentences between \({\cal D}_{h}^{test}\) sets of different language pairs.

\begin{table}
\begin{tabular}{l|l l} \hline  & \(0.5\) (default) & \(0.45\) \\ \hline \(en\text{$\rightarrow$}cs\) & \(7\) & \(24\) \\ \(en\text{$\rightarrow$}de\) & \(1\) & \(5\) \\ \(en\text{$\rightarrow$}is\) & \(37\) & \(135\) \\ \(en\text{$\rightarrow$}ru\) & \(2\) & \(10\) \\ \(en\text{$\rightarrow$}zh\) & \(73\) & \(259\) \\ \hline Avg. rate (\%) & \(0.005\) & \(0.018\) \\ \hline \end{tabular}
\end{table}
Table 7: Hallucination counts (HR in %) on \({\cal D}_{m}^{dev}\) after fine-tuning with \({\cal D}_{p}^{train}\) collected at different thresholds.

\begin{table}
\begin{tabular}{l c c c c c} \hline  & \(en\text{$\rightarrow$}ce\) & \(en\text{$\rightarrow$}dzh\) & \(en\text{$\rightarrow$}cs\) & \(en\text{$\rightarrow$}vis\) & \(en\text{$\rightarrow$}ru\) \\ \hline quotes & \(0.54\) & **2e-9** & **3e-6** & \(0.56\) & **5e-11** \\ urls & \(0.32\) & **4e-19** & \(0.86\) & \(0.91\) & \(0.33\) \\ caps & \(0.06\) & \(0.48\) & **0.02** & \(0.08\) & **1e-3** \\ \hline \end{tabular}
\end{table}
Table 9: Chi-square p-values of features’ impact on hallucination in \({\cal D}_{h}^{test}\). We **bold** entries with statistical significance (p ¡ 0.05).

 



### Distribution of Hallucination Scores

Figure 1 illustrates the distribution of hallucination scores for the _en-\(\rightarrow\)zh_ pair on \({\cal D}^{test}_{m}\) before and after fine-tuning. The top plot shows the full scale distribution from \(0\)-\(1\), while the bottom image provides a zoomed-in view focused on the critical range of \(0.5\)-\(1\), which highlights the hallucination-prone section. In the top plot, the distribution post-fine-tuning (in orange) shifts markedly to the left, indicating an overall improvement in translation quality across the dataset. In the bottom plot, we observe that the remaining hallucinations post-fine-tuning are primarily concentrated near the threshold, with fewer instances with extreme hallucination scores. Plots for all language pairs can be found in Appendix 2.

## 7 Related Work

Prior works on hallucination detection include identifying repeated n-gram patterns in translations Raunak et al. (2021), utilizing internal model information such as attention weights Lee et al. (2019); Berard et al. (2019); Ferrando et al. (2022b); Bouchaud et al. (2023); Voita et al. (2021); Xu et al. (2023); Guerreiro et al. (2023), and estimating uncertainty using the model's sequence log-probability Guerreiro et al. (2023c). Other works have explored external models based on quality estimation (COMET-QE) and cross-lingual sentence similarity LASER LaBSE, XNLI, BLASER-QE Dale et al. (2023a, b).

To mitigate hallucinations, prior works have primarily focused on _post-hoc_ solutions. These include using a fallback model Guerreiro et al. (2023a), generating multiple candidates and selecting the best using a re-ranker Guerreiro et al. (2023c), or applying consensus-based decoding strategies such as Minimum Bayes Risk (MBR) Eikema and Aziz (2020). Other approaches have explored contrastive decoding by leveraging probabilities from different models Li et al. (2023), using previous output tokens Su and Collier (2023), or utilizing a contrastive input Sennrich et al. (2024). While all these approaches mitigate hallucinations during or after inference, our approach takes an orthogonal path by addressing the issue directly within the model itself.

## 8 Conclusion

In this work, we presented a framework for mitigating translation hallucinations in large language models (LLMs). To the best of our knowledge, this is among the first works to demonstrate how to mitigate translation hallucination in LLMs. In this framework, we propose an unsupervised method to create a hallucination-focused preference dataset, which is easily scalable across multiple languages. Fine-tuning LLMs using this dataset through reference optimization reduces hallucination rates by an average of 96%, while preserving general translation quality. Additionally, our method generalizes well in a cross-lingual zero-shot setting, achieving an 89% reduction in hallucination rates across three previously unseen target languages.

\begin{table}
\begin{tabular}{r|c c|c c|c c} \hline Threshold & \multicolumn{2}{c|}{\(0.5\)} & \multicolumn{2}{c|}{\(0.45\)} & \multicolumn{2}{c}{\(0.4\)} \\ \hline  & ALMA–7B-R & \({\cal M}_{p+a}\) & ALMA–7B-R & \({\cal M}_{p+a}\) & ALMA–7B-R & \({\cal M}_{p+a}\) \\ \hline _en\(\rightarrow\)cs_ & \(179\) & \(4\) & \(380\) & \(45\) & \(1388\) & \(385\) \\ _en\(\rightarrow\)de_ & \(39\) & \(1\) & \(59\) & \(6\) & \(199\) & \(111\) \\ _en\(\rightarrow\)is_ & \(388\) & \(35\) & \(1271\) & \(353\) & \(4873\) & \(1722\) \\ _en\(\rightarrow\)ru_ & \(196\) & \(0\) & \(297\) & \(35\) & \(765\) & \(226\) \\ _en\(\rightarrow\)sh_ & \(2192\) & \(80\) & \(6024\) & \(608\) & \(17994\) & \(3967\) \\ \hline Average count & \(599\) & \(24\) & \(1606\) & \(209\) & \(5044\) & \(1282\) \\ Average HR (\%) & \(0.127\) & \(0.005\) & \(0.34\) & \(0.044\) & \(1.067\) & \(0.271\) \\ \hline \end{tabular}
\end{table}
Table 11: Evaluation results at different H5 threshold values: showing hallucination count and HR (%).





## Limitations

* In this work we explored only \(en{\rightarrow}X\) language pairs due to time and resource constraints. We leave the exploration of other directions as a future work.
* Since natural translation hallucination is very rare, we need to translate huge amount of monolingual data to create a reasonable amount of hallucination focused preference dataset, thus making our approach time and compute intensive.
* Our approach depends on a hallucination detector. The language pairs of interest must be supported by the detector, as well as some analysis might be required to decide hallucination detector threshold.

## Ethics Statement

This work, in our knowledge, does not pose any ethical concerns. It proposes approaches to make AI models safe and trustworthy. Still, our models might generate some hallucinations like any other AI models. The original data, model, tools, and open-source software used in the paper are publicly available and has been mentioned in the corresponding sections.

## Acknowledgements

We would like to thank Hendra Setiawan and Robin Schmidt for replicating ALMA-R and CP0, Andrew Finch, Qin Gao, Stephan Peitz, and Stephen Pulman for providing their insights and valuable feedback.

## References

* Alves et al. [2020] Duarte Miguel Alves, Jose Pombal, Nuno M Guerreiro, Pedro Henrique Martins, Joao Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, Jose G. C. de Souza, and Andre Martins. 2020. Tower: An open multilingual large language model for translation-related tasks. In _First Conference on Language Modeling_.
* Aryabumi et al. [2024] Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Ciuruz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Ustun, and Sara Hooker. 2024. Aya 23: Open weight releases to further multilingual progress.
* Berard et al. [2019] Alexandre Berard, Ioan Calapodescu, and Claude Roux. 2019. Naver labs Europe's systems for the WMT19 machine translation robustness task. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 526-532, Florence, Italy. Association for Computational Linguistics.
* Chen et al. [2023] Mingda Chen, Paul-Ambroise Duquenne, Pierre Andrews, Justine Kao, Alexandre Mourachko, Holger Schwenk, and Marta R. Costa-jussa. 2023. BLASER: A text-free speech-to-speech translation evaluation metric. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 9064-9079, Toronto, Canada. Association for Computational Linguistics.
* Dale et al. [2023a] David Dale, Elena Voita, Loic Barrault, and Marta R. Costa-jussa. 2023a. Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity Even better. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 36-50, Toronto, Canada. Association for Computational Linguistics.
* Dale et al. [2023b] David Dale, Elena Voita, Janice Lam, Prangthip Hansanti, Christophe Rogers, Elaine Kalbassi, Cynthia Gao, Loic Barrault, and Marta Costa-jussa. 2023b. HalOmi: A manually annotated benchmark for multilingual hallucination and omission detection in machine translation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 638-653, Singapore. Association for Computational Linguistics.
* Dubey et al. [2024] Abhimanyu Dubey, Abhinav Jauhr, Abhinav Pandey, Abhishe Kadian, Ahmad Al-Dahale, Aiesha Lettman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.
* Eikema and Aziz [2020] Bryan Eikema and Wilker Aziz. 2020. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_, pages 4506-4520. International Committee on Computational Linguistics.
* Feng et al. [2022] Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 878-891, Dublin, Ireland. Association for Computational Linguistics.
* Ferrando et al. [2022a] Javier Ferrando, Gerard I. Gallego, Belen Alastruye, Carlos Escolano, and Marta R. Costa-jussa. 2022a. Towards opening the black box of neural machine translation: Source and target interpretations of the transformer. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 8756-8769, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

Javier Ferrando, Gerard I. Gallego, and Marta R. Costajussa. 2022b. Measuring the mixing of contextual information in the transformer. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 8698-8714, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Freitag et al. (2023) Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. 2023. Epsilon sampling rocks: Investigating sampling strategies for minimum Bayes risk decoding for machine translation. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 9198-9209, Singapore. Association for Computational Linguistics.
* Freitag et al. (2022) Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022. High quality rather than high model probability: Minimum bayes risk decoding with neural metrics. _Transactions of the Association for Computational Linguistics_, 10:811-825.
* Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In _Proceedings of The 33rd International Conference on Machine Learning_, volume 48 of _Proceedings of Machine Learning Research_, pages 1050-1059, New York, New York, USA. PMLR.
* Guerreiro et al. (2023a) Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and Andre F. T. Martins. 2023a. Hallucinations in large multilingual translation models. _Transactions of the Association for Computational Linguistics_, 11:1500-1517.
* Guerreiro et al. (2023b) Nuno M. Guerreiro, Pierre Colombo, Pablo Piantanida, and Andre Martins. 2023b. Optimal transport for unsupervised hallucination detection in neural machine translation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13766-13784, Toronto, Canada. Association for Computational Linguistics.
* Guerreiro et al. (2023c) Nuno M. Guerreiro, Elena Voita, and Andre Martins. 2023c. Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 1059-1075, Dubrovnik, Croatia. Association for Computational Linguistics.
* Hewitt et al. (2022) John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 3414-3427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Hu et al. (2022) Edward J. Hu, Telong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mengch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Dengyel, Guillaume Lample, Lucile Saulnier, Celio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Mistral 7b.
* Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jegou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models.
* Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
* Kocmi et al. (2022) Tom Kocmi, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novak, Martin Popel, and Maja Popovic. 2022. Findings of the 2022 conference on machine translation (WMT22). In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 1-45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Kumar et al. (2023) Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. 2023. Language generation models can cause harm: So what can we do about it? an actionable survey. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 3299-3321, Dubrovnik, Croatia. Association for Computational Linguistics.
* Kumar and Byrne (2004) Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_, pages 169-176, Boston, Massachusetts, USA. Association for Computational Linguistics.
* Lee et al. (2019)Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2019. Hallucinations in neural machine translation.
* Lisa et al. (2016) Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettle 

moyer, and Mike Lewis. 2023. Contrastive decoding: Open-ended text generation as optimization. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12286-12312, Toronto, Canada. Association for Computational Linguistics.
* Liao et al. (2024) Baohao Liao, Christian Herold, Shahram Khadivi, and Christof Monz. 2024. Ikun for wmt24 general mt task: Llms are here for multilingual machine translation.
* Costa-jussa NLLB Team et al. (2022) Marta R. Costa-jussa NLLB Team, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Hefernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzman, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation.
* OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyammal Anaddat, et al. 2024. Gpt-4 technical report.
* Popovic (2015) Maja Popovic. 2015. chrf: character n-gram f-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation, WMT@EMNLP 2015, 17-18 September 2015, Lisbon, Portugal_, pages 392-395. The Association for Computer Linguistics.
* 16, 2023_.
* Raunak et al. (2021) Vikas Raunak, Arul Menezes, and Marcin Junczynski-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1172-1183, Online. Association for Computational Linguistics.
* Raunak et al. (2022) Vikas Raunak, Matt Post, and Arul Menezes. 2022. SALTED: A framework for SAlient long-tail translation error detection. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 5163-5179, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Rei et al. (2022) Ricardo Rei, Ana C Farinha, Jose G.C. de Souza, Pedro G. Ramos, Andre F.T. Martins, Luisa Coheur, and Alon Lavie. 2022. Searching for COMETINHO: The little metric that could. In _Proceedings of the 23rd Annual Conference of the European Association for Machine Translation_, pages 61-70, Ghent, Belgium. European Association for Machine Translation.
* Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702, Online. Association for Computational Linguistics.
* Sennrich et al. (2024) Rico Sennrich, Jannis Vamvas, and Aliireza Mohammadshahi. 2024. Mitigating hallucinations and off-target machine translation with source-contrastive and language contrastive decoding. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 21-33, St. Julian's, Malta. Association for Computational Linguistics.
* Su and Collier (2023) Yixuan Su and Nigel Collier. 2023. Contrastive search is what you need for neural text generation. _Trans. Mach. Learn. Res._, 2023.
* Townhidul Islam Tommoy et al. (2024) S. M Towhidul Islam Tommoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A comprehensive survey of hallucination mitigation techniques in large language models.
* Touvron et al. (2020) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Essiobu, Jude Fernandes, Jeremy Fu, Wenynfu Fu, Brian Fuller, Cynthia Gao, Vedantaj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloundmann, A. V. Korenev, Punit Singh Kouva, Marie-Anne Lachaux, Thibaut Lavril, Jenye Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molbygbo, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Pavin Xu, Zhengxu Yan, Iliyam Zarov, Yuchen Zhang,Angela Fan, Melanie Kambandur, Sharan Narang, Aurelieen Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Sciolom. 2023. Llama 2: Open foundation and fine-tuned chat models. _ArXiv_, abs/2307.09288.
* Voita et al. (2021) Elena Voita, Rico Sennrich, and Ivan Titov. 2021. Analyzing the source and target contributions to predictions in neural machine translation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1126-1140, Online. Association for Computational Linguistics 

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In _Proceedings of the 41st International Conference on Machine Learning_, volume 235 of _Proceedings of Machine Learning Research_, pages 55204-55224. PMLR.
* Xu et al. (2023) Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Marianna J Martindale, and Marine Carpuat. 2023. Understanding and detecting hallucinations in neural machine translation via model introspection. _Transactions of the Association for Computational Linguistics_, 11:546-564.

## Appendix A Monolingual Data Filtering

To prepare the monolingual data for translation, we apply the following four filters in sequence. Table 12 shows the statistics of monolingual data before and after applying the filters.

Heuristic filterremoves empty lines, replaces '\n' with '<NEWLINE>', eliminates sentences containing unprintable unicode characters, as well as those with Chinese decoding errors, and excludes rows with HTML or JSON-like elements.

Length filtersplits the sentence by whitespace (since the source language is English), and removes sentences that are shorter than \(5\) words or longer than \(100\) words.

Deduplication filterremoves exact duplication with drop_duplicates function from Pandas library12.

Footnote 12: [https://pandas.pydata.org](https://pandas.pydata.org).

Language ID filteridentifies the language of each sentence using the fasttext model Joulin et al. (2017, 2016) and removes sentences that fall below the language probability threshold of \(0.5\).

## Appendix B Hyperparameters for Fine-tuning Using CPO

For the preference fine-tuning process, we only train the LoRA parameters, specifically targeting _down_proj_, _q_proj_, _k_proj_, and _v_proj_ with a rank of \(16\). We set the maximum sequence length to \(768\) tokens, utilize the Hugging Face accelerator with Fully Sharded Data Parallel (FSDP), and train on eight H100 GPUs, typically completing training in less than an hour. Inferences are performed on V100s, and takes roughly 7 GPU hours on \({\cal D}^{dev/test}_{h}\) and 1150 GPU hours on \({\cal D}^{dev/test}_{m}\) . The value of \(\beta\) is set to \(0.1\), consistent with the findings of Rafailov et al. (2023) and Xu et al. (2024). We conduct a partial grid search for hyperparameters, varying the _batch size_ from \(\{16,32,64,128,256,512\}\) and the _learning rate_ from \(\{2e-5,5e-5,1e-4,2e-4,5e-4\}\). Through our experimentation, we find that setting epoch to \(1\) generally suffices for optimal performance. We use beam size of \(5\) for baseline and all fine-tuned models.

The best hyperparameters we found for \({\cal M}_{p}\) and \({\cal M}_{p+a}\) are listed in Table 13

## Appendix C Comparing Generation Methods for Post-hoc Mitigation strategies

Section 5.1 compares different mitigation strategies across different selection methods and utility metrics, focusing on the top performing sampling strategies. Here we compare different sampling strategies in Table 14 (MR) and Table 15 (COMET -- wmt22-cometkiwi-da). Contrary to previous studies Guerreiro et al. (2023c); Dale et al. (2023a) we find that MC-beam performs significantly worse than other sampling methods on both MR and COMET. We speculate that this is due to dropout not being used in the training of Llama-2, which is the backbone LLM for ALMA-7B-R. We find temperature \(t=1\) to perform best, with higher values of \(t\) significantly degrading both metrics. Using epsilon

\begin{table}
\begin{tabular}{l c c} \hline  & \multicolumn{2}{c}{Before filtering After filtering} \\ \hline \({\cal D}^{dev}_{m}\) & _en\(\rightarrow\)X_ & \(500\)K & \(473\)K \\ \hline \({\cal D}^{test}_{m}\) & _en\(\rightarrow\)X_ & \(500\)K & \(473\)K \\ \hline \multirow{3}{*}{\({\cal D}^{train}_{m}\)} & _en\(\rightarrow\)cs_ & \(5\)M & \(4.73\)M \\  & _en\(\rightarrow\)de_ & \(10\)M & \(9.46\)M \\ \cline{1-1}  & _en\(\rightarrow\)is_ & \(5\)M & \(4.73\)M \\ \cline{1-1}  & _en\(\rightarrow\)ru_ & \(5\)M & \(4.73\)M \\ \cline{1-1}  & _en\(\rightarrow\)zh_ & \(2\)M & \(1.89\)M \\ \hline \end{tabular}
\end{table}
Table 12: Monolingual data statistics.

 

sampling with \(\epsilon=0.02\) consistently improves results.

## Appendix D Hallucination Focused Preference Dataset Statistics

We report the character length statistics (mean, median, p95, and p99) for the source, preferred, and dispreferred samples in Table 16. Dispreferred samples have significantly longer lengths due to a large proportion of oscillatory hallucinations. Additionally, the hallucination score (HS) statistics (mean, median, p95, and p99) for the preferred and dispreferred data are shown in Table 17. We combine \({\cal D}_{p}^{train}\) with \({\cal D}_{alma}^{train}\) to fine-tune \({\cal M}_{p+a}\). Table 18 lists the dataset size of \({\cal D}_{alma}^{train}\).

## Appendix E Standard CPO vs. Scaled CPO

We conducted an evaluation on \({\cal D}_{h}^{dev}\) to compare the performance of standard (\({\cal L}_{CPO}\)) vs. scaled CPO (\({\cal L}^{\prime}{}_{CPO}\)) losses. Our results show that \({\cal L}^{\prime}{}_{CPO}\) achieves an average hallucination rate of \(0.774\%\), outperforming \({\cal L}_{CPO}\), which has an average rate of \(1.028\%\). Table 19 presents a comparison of the two methods across all five language pairs.

### Intuition behind the scaling for preference loss

Following the notations in Section 7, let \(\psi\) denote the quality gap \(\phi(x,y_{p})\) and \(\phi(x,y_{d})\) as \(\psi=\frac{\phi(x,y_{p})}{\phi(x,y_{d})}\). \(\psi\) is a constant term added inside the sigmoid in our loss function \(L^{\prime}_{p}\)

\[L^{\prime}_{p}=-\mathbb{E}\log\sigma\left(\beta\log\frac{\pi_{ \theta}(y_{p}\mid x)}{\pi_{\theta}(y_{d}\mid x)}+\beta\log\psi\right) \tag{11}\]

Simplifying the sigmoid using \(\sigma(x)=\frac{1}{1+e^{-x}}\):

\[L^{\prime}_{p}=-\mathbb{E}\log\left(\frac{1}{1+e^{-\beta\log \frac{\pi_{\theta}(y_{p}|x)}{\pi_{\theta}(y_{d}|x)}-\beta\log\psi}}\right) \tag{12}\]

\[L^{\prime}_{p}=-\mathbb{E}\log\left(\frac{1}{1+e^{-\beta\log( \frac{\pi_{\theta}(y_{p}|x)}{\pi_{\theta}(y_{d}|x)}+\log\psi)}}\right) \tag{13}\]

\[L^{\prime}_{p}=-\mathbb{E}\log\left(\frac{1}{1+e^{-\beta\log( \frac{\pi_{\theta}(y_{p}|x)}{\pi_{\theta}(y_{d}|x)},\psi)})}\right) \tag{14}\]

\[L^{\prime}_{p}=-\mathbb{E}\log\left(\frac{1}{1+e^{\log(\frac{ \pi_{\theta}(y_{p}|x)}{\pi_{\theta}(y_{d}|x)},\psi)}-\beta}\right) \tag{15}\]

\[L^{\prime}_{p}=-\mathbb{E}\log\left(\frac{1}{1+(\frac{\pi_{\theta}(y_{p}|x)} {\pi_{\theta}(y_{d}|x)}\cdot\psi)^{-}\beta}\right)\] (16) 

\[L^{\prime}_{p}=-\mathbb{E}\log\left(\frac{1}{1+(\frac{\pi_{\theta}(y_{d}|x)}{\pi _{\theta}(y_{p}|x)}\cdot\frac{1}{\psi})^{\beta}}\right) \tag{17}\]

\[L^{\prime}_{p}=\mathbb{E}\log\left(1+\left(\frac{\pi_{\theta}(y_{d}~{}|~{}x)}{ \pi_{\theta}(y_{p}~{}|~{}x)}\frac{1}{\psi}\right)^{\beta}\right) \tag{18}\]

Therefore the quality gap \(\psi\) acts as a multiplicative weight to the ratio of model probabilities for the preferred and dispreferred candidates.

## Appendix F Common Hallucinations Before and After Fine-tuning

We compute the overlap in the hallucinated samples from ALMA-7B-R and \(\mathcal{M}_{p+a}\) in Table 20. _Common source_ column indicates the number of source sentences on which both baseline and fine-tuned models hallucinate, while the _Common pairs_ column reflects the number of identical (source, translation) pairs. For example, for _en\(\rightarrow\)zh_, \(\mathcal{M}_{p+a}\) generates \(80\) hallucinations on \(\mathcal{D}^{test}_{m}\), of which \(30\) (\(37.5\%\)) share the same source sentences that led to hallucinations in the baseline ALMA-7B-R. As expected, the percentage is lower when considering (source, translation) pairs, at \(3.75\%\). It would be valuable to further investigate whether the high proportion of source sentences that still result in hallucinations after fine-tuning are due to underlying data quality issues, limitations in the modeling technique, or a combination of both.

## Appendix G Evaluation with an Alternative Hallucination Detector

Our main evaluation result in Table 4 shows an effective mitigation rate of \(96\%\) using BLASER-QE, the same hallucination detection model used during dataset construction. To confirm the effect of

\begin{table}
\begin{tabular}{c|r r r|r r|r r|r r|r r r} \hline  & Number of & \multicolumn{6}{c|}{Length} \\ \cline{3-13}  & samples & \multicolumn{3}{c|}{Mean} & \multicolumn{3}{c|}{Median} & \multicolumn{3}{c|}{P95} & \multicolumn{3}{c}{p99} \\  & & \(x\) & \(y_{p}\) & \(y_{d}\) & \(x\) & \(y_{p}\) & \(y_{d}\) & \(x\) & \(y_{p}\) & \(y_{d}\) & \(x\) & \(y_{p}\) & \(y_{d}\) \\ \hline _en\(\rightarrow\)cs_ & \(2063\) & \(168\) & \(190\) & \(1016\) & \(132\) & \(144\) & \(1102\) & \(434\) & \(511\) & \(1535\) & \(538\) & \(661\) & \(1972\) \\ _en\(\rightarrow\)de_ & \(671\) & \(156\) & \(199\) & \(1306\) & \(117\) & \(152\) & \(1258\) & \(426\) & \(554\) & \(2447\) & \(513\) & \(677\) & \(2770\) \\ _en\(\rightarrow\)is_ & \(3598\) & \(153\) & \(185\) & \(761\) & \(120\) & \(140\) & \(940\) & \(408\) & \(502\) & \(1245\) & \(549\) & \(677\) & \(1361\) \\ _en\(\rightarrow\)rn_ & \(1931\) & \(164\) & \(197\) & \(852\) & \(129\) & \(151\) & \(655\) & \(424\) & \(522\) & \(1522\) & \(543\) & \(673\) & \(1789\) \\ _en\(\rightarrow\)zh_ & \(8349\) & \(144\) & \(71\) & \(283\) & \(116\) & \(57\) & \(297\) & \(348\) & \(170\) & \(495\) & \(503\) & \(251\) & \(540\) \\ \hline Average & \(3322\) & \(157\) & \(168\) & \(844\) & \(123\) & \(129\) & \(850\) & \(408\) & \(452\) & \(1449\) & \(529\) & \(588\) & \(1686\) \\ \hline \end{tabular}
\end{table}
Table 16: Statistics of length in characters for source (\(x\)), preferred (\(y_{p}\)), and dispreferred (\(y_{d}\)) samples in \(\mathcal{D}^{train}_{p}\).

\begin{table}
\begin{tabular}{l l c c} \hline  & Hallucination Count & \multicolumn{3}{c}{Count} \\ \hline  & ALPma-7B-R & \(\mathcal{M}_{p+a}\) & Common Common source (source+trans.) \\ \hline _en\(\rightarrow\)cs_ & \(179\) & \(4\) & \(2\) & \(0\) \\ _en\(\rightarrow\)de_ & \(39\) & \(1\) & \(0\) & \(0\) \\ _en\(\rightarrow\)is_ & \(388\) & \(35\) & \(10\) & \(4\) \\ _en\(\rightarrow\)ru_ & \(196\) & \(0\) & \(0\) & \(0\) \\ _en\(\rightarrow\)zt_ & \(2192\) & \(80\) & \(34\) & \(3\) \\ \hline \end{tabular}
\end{table}
Table 20: Common source and (source, target) pairs between ALMA-7B-R and \(\mathcal{M}_{p+a}\) on \(\mathcal{D}^{test}_{m}\).

 

mitigation is beyond fitting to the same metric, biasing our results, we additionally evaluate the same translation with an alternative hallucination detector: top n-gram detector (Raunak et al., 2021). This detector has high accuracy for detecting oscillatory/repetitive hallucination, which is a major category of hallucination seen from Section 6.4. We use the same hyperparameter as Raunak et al. (2021): n-gram size of \(4\) and threshold of \(2\). In Table 21, we see a \(92\%\) drop in hallucination rate on average from \(0.81\%\) to \(0.06\%\), re-affirming that the mitigation is not biased towards a single metric.

## Appendix H Statistics of Hallucination and Non-hallucination Samples

Table 22 shows source and translation character length statistics (mean, median, p95, and p99) for hallucination (\({\cal D}_{h}^{test}\)) and non-hallucination (\({\cal D}_{nh}^{test}\)) cases of the test set (\({\cal D}_{m}^{test}\)), where translations are generated by ALMA-7B-R. We observe that the length statistics for source sentences are nearly identical between hallucination and non-hallucination samples. However, on the translation side, hallucinated translations are significantly longer than their non-hallucinated counterparts. For instance, the average length of hallucinated translations (\(839\) characters) is \(5.6\) times longer than that of non-hallucinated translations (\(150\) characters) across all language pairs. Additionally, for the non-hallucinated subset, the average source-to-target length ratio is nearly \(1:1\), while for the hallucinated subset, it is \(1:5.7\).

## Appendix I Examples of Preference Pairs in our Dataset

Table 23 includes examples of preference pairs in \({\cal D}_{p}^{train}\) demonstrating that preferred translations recover from the pathological hallucinations present in the dispreferred translation.

## Appendix J Qualitative analysis of translation

Table 24 demonstrates examples where our fine-tuned model \({\cal M}_{p+a}\) successfully mitigates hallucinations over the baseline model ALMA-7B-R. The pattern of hallucinations and their mitigations are very similar to those observed in our preference dataset.

## Appendix K Visualizing Hallucination and COMET Score Distributions

Distribution of scoresFigure 2 and Figure 3 show the distribution of hallucination and COMET scores, respectively, for ALMA-7B-R and \({\cal M}_{p+a}\). We observe that the distribution of hallucination score for _en_\(\rightarrow\){_cs, is, zh_} shift slightly to the left after fine-tuning, indicating reduction in hallucination score. In contrast, the distributions for COMET are so closely overlapped that no definitive conclusions can be drawn.

Regression of scoresFigure 4 and Figure 5 display regression plots for hallucination and COMET scores, respectively, comparing ALMA-7B-R and \({\cal M}_{p+a}\). The X-axis represents hallucination (or COMET) score for translations obtained with ALMA-7B-R, while the Y-axis shows the score for translations obtained with \({\cal M}_{p+a}\). The regression plots for hallucination clearly indicate improvements in the majority of translations across all language pairs, with the exception of _en_\(\rightarrow\)_de_, which exhibits slightly higher regression. Conversely, the regression plots for COMET yield mixed results, making it challenging to draw definitive conclusions.

 



\begin{table}
\begin{tabular}{c|l|p{142.3pt} p{56.9pt}|p{78.0pt}|} \hline \multicolumn{1}{c|}{} & \multicolumn{3}{c|}{Source} & \multicolumn{1}{c|}{Preferred} \\ \hline _en\(\rightarrow\)de_ & Five state assemblies of Madhya Pradesh, Chhattisgarh, Rajasthan, Telangana and Mizoram will go to pols on different days beginning November 7 and counting of votes will take place on December 3. & Am 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30., 31., (repeat) & Die Wahlen der fünf Landesparlaemente in Madhya Pradesh, Chhattisgarh, Rajasthan, Telangana und Mizoram infaden an verschiedeen



\begin{table}
\begin{tabular}{p{42.7pt}|p{190.3pt}|p{85.4pt}|p{65.4pt}} \hline Direction & Source & ALMA-7B-R & \(\mathcal{M}_{p+a}\) \\ \hline \multirow{3}{*}{_en\(\rightarrow\)de_} & Because she was backed into a corner. & Weil sie sich in eine Bredouille manövriert hatte. & Weil sie in eine Ecke gedringt wurde. \\ \cline{2-3}  & Drought plans stepped up due to water demand amid “hottest June on record” & Aufgrund der erüohten Wasserverflugkreit aufgrund der erüohten Wassersteinufgrand der erüohten Wasserverflugbarkeit aufgrund (repeat) & Durréplane verschärt wegen get eigengender Wasserverflugbarkeit angesechits des ,,heissetsten Juni auf rekordniveau" \\ \hline \multirow{3}{*}{_en\(\rightarrow\)cs_} & They are still largely pessimistic about the direction of prices and the stock market; about peace and stability in the world; and about the climate and prospect of extreme weather events. & Jejch názory na smer cen a akicoveño truhu, miru miru miru miru miru míry myiry miry miry miry miry miry miry miry miry miry miry (repeat) & Stale jsou z velke cásti pesimistcit di ohhedne vyovie cen a trlu sennympi papify, o miru miru a stabilite ve svèete, a táake ohledne klimatu a coekavanych extremmrich meteorologyckych jevêu. \\ \cline{2-3}  & It had been blessed by a monk and was meant to offer her protection. & Byla ji venouna a muà alena jlig poskytuomt ochranu. (repeat) \\ \hline \multirow{3}{*}{_en\(\rightarrow\)is_} & This is ridiculous, totally ridiculous. & Petta er fyir neán allar helllur, gjorléga fyürrie noan allar hellur. & Petta er faränlegt, algörléga fárann legt. \\ \cline{2-3}  & Jordan and Henry actually managed to find a blind spot! & Jordan og Henry fundu raunverleiga eithvaß tyfr ser! peir fundu raunverleiga eithvaß fyrit sefer! Peir fundu raunverulega eithvaß fürvir sefer! Peir fundu raunverulega eithvaß furir sefer! Peir fundu raunverulega eithvaß fürsefer! (repeat) \\ \hline \multirow{3}{*}{_en\(\rightarrow\)nu_} & A third commented: ”Chelsea were 12th and Arsenal word, don’t talk..” & Terentih dogabil!

Figure 2: Hallucination score (HS) distribution for ALMA-7B-R and \({\cal M}_{p+a}\) on \({\cal D}^{test}_{m}\). Right plots are zoomed-in on hallucination regions.

 

Figure 3: COMET score (Unbabel/wmt22-comettiwi-da) distribution for ALMA–7B–R and \({\cal M}_{p+a}\) on \({\cal D}^{test}_{m}\).

 

Figure 4: Regression plots showing hallucination score (HS) for ALMA-7B–R and \({\cal M}_{p+a}\) on \({\cal D}^{test}_{m}\).

 

Figure 5: Regression plots showing COMET score (Unbabel/wmt22-comekiwi-da) for ALMA-7B-R and \({\cal M}_{p+o}\) on \({\cal D}^{test}_{m}\).

 



\begin{table}
\begin{tabular}{c|l l c c|l c c c c} \hline  & BLEU & XCOMET & KIWI-22 & KIWI-XXL & BLEU & XCOMET & KIWI-22 & KIWI-XXL \\ \hline NLLB-3.3B & \(35.26\) & \(81\) & \(77.69\) & \(72.96\) & \(31.74\) & \(84.17\) & \(79.88\) & \(77.1\) \\ ALMA-7B-R & \(28.59\) & \(84.71\) & \(78.68\) & \(76.08\) & \(31.78\) & \(88.94\) & \(80.97\) & \(80.57\) \\ \(\mathcal{M}_{p}\) & \(28.32\) & \(84.05\) & \(78.48\) & \(75.43\) & \(31.6\) & \(88.27\) & \(80.7\) & \(79.81\) \\ \(\mathcal{M}_{p+a}\) & \(28.31\) & \(85.01\) & \(78.66\) & \(76.06\) & \(31.69\) & \(88.67\) & \(80.94\) & \(80.35\) \\ \hline  & \multicolumn{6}{c}{_zh\(\rightarrow\)en_} & \multicolumn{6}{c}{_X\(\rightarrow\)en_ average} \\ \hline NLLB-3.3B & \(22.15\) & \(82.77\) & \(77.15\) & \(71.89\) & **29.72** & \(82.65\) & \(78.24\) & \(73.98\) \\ ALMA–7B-R & \(22.51\) & \(89.01\) & \(79.6\) & \(77.63\) & \(27.63\) & \(87.55\) & **79.75** & **78.09** \\ \(\mathcal{M}_{p}\) & \(22.71\) & \(88.35\) & \(79.46\) & \(77.36\) & \(27.54\) & \(86.89\) & \(79.55\) & \(77.53\) \\ \(\mathcal{M}_{p+a}\) & \(22.5\) & \(88.99\) & \(79.57\) & \(77.79\) & \(27.5\) & **87.56** & \(79.72\) & \(78.07\) \\ \hline \end{tabular}
\end{table}
Table 26: WMT’23 COMET and sacreBLEU scores for _X\(\rightarrow\)en_ directions. **XCOMET** = Unbabel/COMET-XCMET-XML, **KIWI-22** = Unbabel/COMET-wmt22-cometkiwi-da, **KIWI-XXL** = Unbabel/COMET-wmt23-cometkiwi-da-xxl. We reproduce all baseline model results. Best results per eval metric is shown in **bold** and second best is underlined.

\begin{table}
\begin{tabular}{r|l l c c|l c c c} \hline  & BLEU & XCOMET & KIWI-22 & KIWI-XXL & BLEU & XCOMET & KIWI-22 & KIWI-XXL \\ \hline  & \multicolumn{6}{c}{_en\(\rightarrow\)de_} & \multicolumn{6}{c}{_ru\(\rightarrow\)en_} \\ \hline NLLB-3.3B & \(35.26\) & \(81\) & \(77.69\) & \(72.96\) & \(31.74\) & \(84.17\) & \(79.88\) & \(77.1\) \\ ALMA-7B-R & \(28.59\) & \(84.71\) & \(78.68\) & \(76.08\) & \(31.78\) & \(88.94\) & \(80.97\) & \(80.57\) \\ \(\mathcal{M}_{p}\) & \(28.32\) & \(84.05\) & \(78.48\) & \(75.43\) & \(31.6\) & \(88.27\) & \(80.7\) & \(79.81\) \\ \(\mathcal{M}_{p+a}\) & \(28.31\) & \(85.01\) & \(78.66\) & \(76.06\) & \(31.69\) & \(88.67\) & \(80.94\) & \(80.35\) \\ \hline  & \multicolumn{6}{c}{_zh\(\rightarrow\)en_} & \multicolumn{6}{c}{_X\(\rightarrow\)en_ average} \\ \hline NLLB-3.3B & \(22.15\) & \(82.77\) & \(77.15\) & \(71.89\) & **29.72** & \(82.65\) & \(78.24\) & \(73.98\) \\ ALMA–7B-R & \(22.51\) & \(89.01\) & \(79.6\) & \(77.63\) & \(27.63\) & \(87.55\) & **79.75** & **78.09** \\ \(\mathcal{M}_{p}\) & \(22.71\) & \(88.35\) & \(79.46\) & \(77.36\) & \(27.54\) & \(86.89\) & \(79.55\) & \(77.53\) \\ \(\mathcal{M}_{p+a}\) & \(22.5\) & \(88.99\) & \(79.57\) & \(77.79\) & \(27.5\) & **87.56** & \(79.72\) & \(78.07\) \\ \hline \end{tabular}
\end{table}
Table 26: WMT’23 COMET and sacréBLEU scores for _X\(\rightarrow\)en_ directions. **XCOMET** = Unbabel/COMET-XCMET-XXL, **KIWI-22** = Unbabel/COMET-wmt22-cometkiwi-da, **KIWI-XXL** = Unbabel/COMET-wmt23-cometkiwi-da-xxl. We reproduce all baseline model results. Best results per eval metric is shown in **bold** and second best is underlined.





\begin{table}
\begin{tabular}{c|l l c c|l c c c c} \hline \hline  & BLEU & XCOMET & KIWI-22 & KIWI-XXL & BLEU & XCOMET & KIWI-22 & KIWI-XXL \\ \hline  & \multicolumn{4}{c}{_en\(\rightarrow\)de_} & \multicolumn{4}{c}{_en\(\rightarrow\)cs_} \\ \hline NLLB-3.3B & 34.16 & 95.62 & 83.35 & 82.36 & 36.27 & 89.29 & 84.15 & 81.65 \\ ALMA–7B-R & 27.01 & 96.68 & 83.41 & 83.94 & 25.21 & 90.24 & 84.95 & 86.49 \\ \(\mathcal{M}_{p}\) & 27.7 & 95.84 & 83.26 & 82.58 & 27.26 & 89.67 & 84.45 & 83.87 \\ \(\mathcal{M}_{pA}\) & 27.52 & 96.4 & 83.21 & 83.24 & 25.65 & 90.37 & 84.8 & 85.6 \\ \hline  & \multicolumn{4}{c}{_en\(\rightarrow\)is_} & \multicolumn{4}{c}{_en\(\rightarrow\)zh_} \\ \hline NLLB-3.3B & 23.46 & 79.3 & 79.63 & 75.42 & 31.91 & 81.42 & 75.05 & 65.62 \\ ALMA-7B-R & 20.81 & 85.45 & 81.53 & 83.94 & 30.5 & 89.66 & 81.88 & 82.77 \\ \(\mathcal{M}_{p}\) & 22.19 & 86.71 & 81.74 & 83.33 & 32.57 & 89.87 & 81.9 & 81.57 \\ \(\mathcal{M}_{pA}\) & 22.11 & 87.26 & 81.68 & 83.71 & 31.85 & 90.31 & 82.08 & 82.67 \\ \hline  & \multicolumn{4}{c}{_en\(\rightarrow\)ru_} & \multicolumn{4}{c}{_en\(\rightarrow\)X_ average} \\ \hline NLLB-3.3B & 30.22 & 91.08 & 83.35 & 82.35 & 31.2 & 87.34 & 81.11 & 77.48 \\ ALMA-7B-R & 23.43 & 93.35 & 84.04 & 86.5 & 25.39 & 91.08 & 83.16 & **84.72** \\ \(\mathcal{M}_{p}\) & 24.93 & 92.3 & 83.8 & 84.45 & **26.93** & 90.88 & 83.03 & 83.16 \\ \(\mathcal{M}_{p+a}\) & 23.79 & 93.19 & 84.06 & 86.15 & 26.18 & **91.51** & **83.17** & 84.27 \\ \hline \hline \end{tabular}
\end{table}
Table 27: WMT’22 COMET and sacreBLEU scores for _cn\(\to\)X_ directions. **XCOMET** = Unbabel/COMET-Xcomet-YOL, **KIWI-22** = Unbabel/COMET-wmt22-comettiwi-da, **KIWI-XXL** = Unbabel/COMET-wmt23-comettiwi-da-xxl. We reproduce all baseline model results. Best results per eval metric is shown in **bold** and second best is underlined.

\begin{table}
\begin{tabular}{r|l r r r|l r r r} \hline \hline  & BLEU & XCOMET & KIWI-22 & KIWI-XXL & BLEU & XCOMET & KIWI-22 & KIWI-XXL \\ \hline  & \multicolumn{4}{c}{_de\(\rightarrow\)en_} & \multicolumn{4}{c}{_cs\(\rightarrow\)en_} \\ \hline NLLB-3.3B & 29.45 & 91.35 & 81.02 & 82.11 & 49.03 & 85.94 & 81.72 & 80.25 \\ ALMA-7B-R & 31.32 & 93.6 & 81.4 & 83.61 & 43.71 & 89.32 & 82.37 & 82.91 \\ \(\mathcal{M}_{p}\) & 31.09 & 93.2 & 81.18 & 82.76 & 43.44 & 88.88 & 82.19 & 81.96 \\ \(\mathcal{M}_{pA}\) & 30.94 & 93.69 & 81.31 & 83.31 & 42.94 & 89.6 & 82.36 & 82.57 \\ \hline  & \multicolumn{4}{c}{_is\(\rightarrow\)en_} & \multicolumn{4}{c}{_zh\(\rightarrow\)en_} \\ \hline NLLB-3.3B & 34.27 & 74.8 & 79.87 & 79.22 & 20.96 & 82.28 & 75.38 & 68.36 \\ ALMA-7B-R & 38.86 & 86.6 & 81.49 & 85.63 & 22.32 & 89.47 & 78.9 & 76.5 \\ \(\mathcal{M}_{p}\) & 39.32 & 86.43 & 81.42 & 85.63 & 22.1 & 88.79 & 78.61 & 75.95 \\ \(\mathcal{M}_{pA}\) & 38.61 & 86.54 & 81.41 & 85.54 & 22.08 & 89.25 & 78.68 & 76.31 \\ \hline  & \multicolumn{4}{c}{_ru\(\rightarrow\)en_} & \multicolumn{4}{c}{_X\(\rightarrow\)en_ average} \\ \hline NLLB-3.3B & 40.17 & 89.43 & 80.87 & 78.39 & 34.78 & 84.76 & 79.77 & 77.67 \\ ALMA-7B-R & 38.91 & 92.27 & 81.57 & 81.22 & **35.02** & 90.25 & **81.15** & **81.97** \\ \(\mathcal{M}_{p}\) & 39.1 & 91.94 & 81.35 & 80.8 & 35.01 & 89.85 & 80.95 & 81.42 \\ \(\mathcal{M}_{pA}\) & 38.47 & 92.54 & 81.55 & 81.08 & 34.61 & **90.33** & 81.06 & 81.76 \\ \hline \hline \end{tabular}
\end{table}
Table 28: WMT’22 COMET and sacréBLEU scores for _X\(\rightarrow\)en_ directions. **XCOMET** = Unbabel/COMET-XComet-YOL, **KIWI-22** = Unbabel/COMET-wmt22-comettawi-da, **KIWI-XXL** = Unbabel/COMET-wmt23-comettawi-da-xxl. We reproduce all baseline model results. Best results per eval metric is shown in **bold** and second best is underlined.