OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection (2024)

Abstract: Recently, there has been considerable attention on detecting hallucinations and omissions in Machine Translation (MT) systems. The two dominant approaches to tackle this task involve analyzing the MT system’s internal states or relying on the output of external tools, such as sentence similarity or MT quality estimators. In this work, we introduce OTTAWA, a novel Optimal Transport (OT)-based word aligner specifically designed to enhance the detection of hallucinations and omissions in MT systems. Our approach explicitly models the missing alignments by introducing a “null” vector, for which we propose a novel one-side constrained OT setting to allow an adaptive null alignment. Our approach yields competitive results compared to state-of-the-art methods across 18 language pairs on the HalOmi benchmark. In addition, it shows promising features, such as the ability to distinguish between both error types and perform word-level detection without accessing the MT system’s internal states.1

heading: 1 Introduction
text: Concerns regarding hallucination (as known as fabrication) of Machine Translation (MT) systems (Raunak et al., 2021; Müller and Sennrich, 2021; Guerreiro et al., 2023a) have led to considerable efforts towards the creation of diagnostic datasets (Zhou et al., 2021; Guerreiro et al., 2023c; Dale et al., 2023b) and developing detection methods (Guerreiro et al., 2023b; Dale et al., 2023a). In addition to hallucination errors, which occur when words in the translation are detached from the source sequence, addressing omission errors is also crucial—these are cases where words from the source sequence are absent in the translation.
*Equal contribution †Work is done during an internship at Noah’s Ark Lab. 1Our code is publicly available at https://github.com/
chenyangh/OTTAWA
Existing methods for detecting hallucinations and omissions in MT either focus on analyzing the model’s internal states (e.g. cross-attention) or rely on the output of external tools, such as crosslingual similarity estimators (Feng et al., 2022; Heffernan et al., 2022) or MT quality estimators (Rei et al., 2020; Duquenne et al., 2023). On one hand, internal methods are limited to scenarios where white-box access to the MT system is assumed, and are further limited by dependencies on specific model architectures (e.g. attention-based). On the other hand, external methods heavily rely on scalar outputs from models optimized for related tasks, such as MT quality or sentence similarity, but not directly optimized for hallucination detection. While hallucinations and omissions constitute low-quality translations, the opposite does not always hold. Furthermore, despite their effectiveness in detecting hallucinations and omissions, existing internal and external methods cannot distinguish
6322
between these two error types. This is because they frame the task as anomaly detection, relying on a single outlier score to detect both error types.
Cross-lingual word alignment (Brown et al., 1993a; Och and Ney, 2003) aligns source to target tokens in parallel sentence pairs, and it provides unique insight for translation errors like hallucination and omission. Following the definition by Guerreiro et al. (2023c), hallucinated translation happens when the generated content is largely detached from the source text; therefore, we can measure hallucination by the amount of “misaligned” words in the translated text. Similarly, an omission error is likely if a significant amount of source words are not found in the translation.
Despite the strong relevance between word alignments and translation errors, alignment tools have not yet been widely applied for hallucination and omission detection. In particular, we observe that existing alignment approaches (Sabet et al., 2020; Azadi et al., 2023; Arase et al., 2023) are often based on the assumption that the translation is correct, such that most of the source or target words can be aligned with each other. This assumption, however, does not hold for scenarios where translation errors have occurred; hence, the direct application of existing word alignment tools does not work well for detecting translation errors.
In this paper, we utilize Optimal Transport problem (Villani, 2009; Cuturi, 2013) and propose OTTAWA, an Optimal TransporT Adaptive Word Aligner that is specifically tailored to aligning words for the pathological cases where significant hallucination and omission errors are present. Our approach explicitly models the missing alignments by introducing a “null” vector. Specifically, we propose a novel one-side constrained Optimal Transport (OT) problem, which allows the null vector to be adaptively aligned to any number of words. Moreover, we set the null vector as a centric point, which is not biased during the computation of the OT problem. We show an example of our OT-based method in Figure 1.
We conduct extensive experiments using the newly proposed HalOmi benchmark (Dale et al., 2023b), designed for hallucination and omission detection, across a total of 18 language pairs. Results show that our word aligner-based approach, equipped with OTTAWA, performs on par with state-of-the-art internal and external detection methods (Guerreiro et al., 2023b; Dale et al., 2023a). In contrast to previous approaches, we
show that our method can distinguish between hallucination and omission errors, while OTTAWA being crucial for performances compared to using existing word aligners.

heading: 2 Related Work
text: Hallucination remains a persistent concern in machine translation systems, which have recently gained increased attention in the field. Zhou et al. (2021) studied token-level hallucinations in machine translation on a Chinese-English dataset, which they constructed by randomly corrupting some tokens in the translation sentences. Guerreiro et al. (2023c) developed a dataset of 3.4k naturally occurring German-English translations, manually annotated at both the sentence and token levels for hallucinations and omissions detection. HalOmi (Dale et al., 2023b) expanded the language coverage for this task by manually annotating hundreds of translation pairs across 18 language pairs.
Methods for detecting hallucinations and omissions in machine translation can be broadly categorized into two types: internal, which analyzes the translation model’s own outputs and states, and external, which relies on additional tools or data outside the model itself (Dale et al., 2023a). Guerreiro et al. (2023b) developed a method that employs various optimal transport-based distances (Kantorovich, 1942; Villani, 2009) to evaluate the abnormality in a machine translation (MT) model’s internal cross-attention distribution of a given translation, by making comparisons with those from a collection of high-quality translations. This method not only requires access to the internal states of the MT model but also necessitates a large collection of cross-attention maps from ground truth parallel sequences for the targeted language pair.
Dale et al. (2023a) investigated the use of other internal MT model components, such as length-normalized sequence log-probability (SeqLogprob) and the token-token layer-wise interaction framework (ALTI) 2, as alternative abnormality estimators that do not require external translation data. In addition, the authors suggested directly using scores from external MT quality estimator models, such as COMET-QE (Rei et al., 2020), or employing the cosine similarity between the source and translation as generated by cross-lingual sentence similarity models, including LASER (Heffer-
2Originally proposed by (Ferrando et al., 2022) for interpreting MT systems
nan et al., 2022), LaBSE (Feng et al., 2022). Word alignment (Brown et al., 1993a,b) has been extensively studied over the years in both cross-lingual (Nagata et al., 2020; Dou and Neubig, 2021) and monolingual (Nagata et al., 2020; Lan et al., 2021) settings. The task is approached with supervised learning methods (Och and Ney, 2003; Östling and Tiedemann, 2016), which require sentence-level parallel data, as well as unsupervised methods that do not rely on such data. The latter approach involves leveraging token-token attention matrices (Ghader and Monz, 2017; Zenkel et al., 2020; Zhang and van Genabith, 2021) or pairwise similarity matrices between tokens’ embedding vectors (Sabet et al., 2020; Azadi et al., 2023), both extracted from a multilingual pretrained language models (Devlin et al., 2019; Conneau et al., 2020). The application of Optimal Transport (OT, Monge, 1781; Kantorovich, 1942; Villani, 2009; Cuturi, 2013) in natural language processing tasks was mainly focused on similarity estimation between textual segments (Huang et al., 2016; Alqahtani et al., 2021; Lee et al., 2022; Mysore et al., 2022), and more recently for word alignment (Azadi et al., 2023; Arase et al., 2023).

heading: 3 Method
text: We formulate the unsupervised word alignment task in § 3.1, and then provide an overview on how standard and partial optimal transport are employed to address the task in § 3.2 and § 3.3, respectively. In § 3.4, we describe our newly proposed optimal transport word aligner, OTTAWA, specifically adapted to enhance the detection of MT hallucinations and omissions.

heading: 3.1 Unsupervised Word Alignment
text: Given a source word sequence s=[s1, . . . , sm] and a target word sequence t=[t1, . . . , tn], let E(s)=[e(s)1 , . . . , e (s) m ] ∈ Rm×D and E(t)=[e (t) 1 , . . . , e (t) n ] ∈ Rn×D be the embedding matrices for the source and target sequences, respectively, where D is the dimensionality of the embeddings. A cost matrix C ∈ Rm×n is defined such that Ci,j is the cosine distance between the i-th source vector and the j-th target vector.
The goal is to compute the alignment matrix Γ ∈ {0, 1}m×n, where Γi,j = 1 if the i-th source word is aligned to the j-th target word, such that the aligned words have the smallest distance in the cost matrix C. A straightforward approach to obtain
Γ involves making greedy decisions on each word alignment in both forward (source-to-target) and reverse (target-to-source) directions. Specifically, given the cost matrix C, the forward alignment Γ (F ) i,j can be obtained by:
Γ (F ) i,j = { 1, if j = argminj Ci,j 0, otherwise
(1)
and similarly, the reverse alignment Γ(R)i,j is computed by:
Γ (R) i,j = { 1, if i = argminiCi,j 0, otherwise
(2)
The final alignment matrix Γ is obtained by taking the element-wise multiplication of Γ(F ) and Γ(R). However, Sabet et al. (2020) found that greedy decisions tend to ignore the word pairs of relatively lower similarity. Therefore, the authors propose to formulate the word alignment as an assignment problem (Kuhn, 1955), which consequently involves solving the following optimization problem:
Γ∗ = argmin Γ∈U(A)
∑
i,j
Ci,jΓi,j (3)
where U (A) = {Γ ∈ {0, 1}m×n : Γ1n ≤ 1n,Γ ⊤1m ≤ 1m, ∑
i,j Γi,j = min(m,n)} is the set of all binary matrices with the sum of row and column equal to 1. The optimization in Eq. 3 can be solved by linear programming (Bourgeois and Lassalle, 1971).

heading: 3.2 Standard Optimal Transport
text: Chi et al. (2021) have noticed that binary assignments obtained by Eq. 3 pose challenges when aligning source and target texts of markedly different lengths. This is because the binary assignment assumes that each word in the shorter text corresponds to a word in the longer text, while the additional words in the longer text remain unaligned. To address this issue, Chi et al. (2021); Dou and Neubig (2021) leverage standard optimal transport as an alternative solution for word alignment. The goal is to compute a matrix P ∗ ∈ Rm×n+ as follows:
P ∗ = argmin P∈U(O)
∑
i,j
Ci,jPi,j (4)
where U (O) = {P ∈ Rm×n+ : P1n = µ, P⊤1m = ν}, µ = (1/m, . . . , 1/m), and
ν = (1/n, . . . , 1/n). The binary matrix Γ can be obtained by replacing the cost matrix C with the matrix P ∗ in Eq. 1 and Eq. 2, followed by performing element-wise multiplication on the resultant matrices. It is important to note that standard OT strictly requires the use of marginal constraints on µ and ν, which enforce the alignment of each source or target word. Therefore, computing forward and reverse alignments is crucial for performance, as these computations implicitly model null alignments. This necessity arises because standard OT-based word alignment assumes that there is a one-to-one mapping between each word in the source sequence and a corresponding word in the target sequence, and vice versa.

heading: 3.3 Partial Optimal Transport
text: The one-to-one mapping assumption of standard OT is not always applicable, particularly in tasks with unbalanced source-target word alignments, such as monolingual text summarization. In response, Arase et al. (2023) introduce partial optimal transport (Peyré et al., 2016; Chapel et al., 2020) word aligner that relaxes the one-to-one mapping assumption, explicitly allowing for source or target words to remain naturally unaligned. In their setting, the transport map P can be found by solving the optimization problem as in Eq. 4 but with different constraints:
U (P ) = {P ∈ Rm×n+ : P1n ≤ µ, P⊤1m ≤ ν, ∑
i,j
Pi,j = m} (5)
where µ and ν are uniform distributions, m ∈ (0, 1) is a hyperparameter controlling how many source and target words are aligned. Then, a threshold τ is used to obtain the binary alignment matrix Γ from the matrix P . Although the partial OT-based drops the strong assumption of the standard OT, it requires extensive search over the hyperparameters m and τ to determine whether a source and target word are considered aligned or not. Furthermore, the conditions in Eq. 5 favor the alignment of closely similar words, because the marginal constraints for either side are not guaranteed. Consequently, this promotes a high recall for alignments at the expense of precision when source and target words are not highly similar.

heading: 3.4 OTTAWA
text: Accurately identifying unaligned words in the target text is crucial for detecting MT hallucina-
tions. Similarly, identifying unaligned words in the source text enhances the detection of omissions. We propose explicitly handle null alignments as a mapping to special null words, denoted as sm+1 and tn+1, which are appended to the source s and target t sequences, respectively. We denote the embeddings corresponding to sm+1 and tn+1 as e (s) m+1 and e (t) n+1, respectively. Our approach addresses forward and reverse alignments separately, yet employs the same method for both. Therefore, we describe how Γ(R) is obtained, as Γ(F ) can be derived analogously. It is important to note that in performing a reverse alignment, only e(s)m+1 is appended to E(s), while E(t) remains unchanged, and vice versa for forward alignment.
Let esm+1 be denoted as e (∅) hereafter for simplicity. So far, e(∅) can be any vector but to make the OT problem meaningful, we need to impose constraints on the distance between e(∅) and the target word vectors E(t). Firstly, e(∅) should be equidistant to every target vector to avoid bias. In addition, these equal distances should be of the same scale as the average pair-wise distances between the source and target vectors to avoid aggressive alignment to e(∅).
To this end, we extend the cost matrix C to C̄ ∈ R(m+1)×n, where C̄i,j = Ci,j for i ∈ [1,m] and C̄m+1,j = d, where d is the distance between e(∅) and the target word representations (by our assumption, it is the same for all target words). One can define d in multiple ways. For example, a natural choice is to use the mean of pair-wise distances between the source and target vectors. However, we choose to use the median over the mean to avoid the influence of the outliers. It is also based on the intuition that e(∅) should serve as a centric point, which is not biased during the computation of the optimal transport map.
For e(∅) to be realizable in the vector space RD, the equal distance d has a lower bound dmin (see Appendix A). We finally define d = max(dmin, c), where c is the median of all pair-wise distances between the source and target words. We reformulate the optimization problem for the transport map in Eq. 4 by replacing the cost matrix C with C̄ and relaxing the constraints as follows:
U (∅,R) = {P ∈ R(m+1)×n+ : P1n ≤ µ′, P⊤1m+1 = ν}
(6)
where µ′ = (µ, 1) allows e(∅) to have a maximum marginal of 1, and to be aligned with any number
of target vectors. We denote the solution of this optimization problem as P (∅,R). The reverse alignment matrix Γ(R) is computed as in Eq. 2 by replacing C with P (∅,R). The forward alignment matrix Γ(F ) is computed analogously. The final alignment matrix Γ is obtained by taking the element-wise product between Γ(F ) and Γ(R).
Our optimization problem in the set of Eq. 6 is a special case of partial optimal transport, it combines the properties of Eq. 4 and Eq. 5. For example, in the computation of Γ(R), we relax the marginal constraint on the source side, a necessity given that the marginal for e(∅) can range between 0 and 1. As a result, the model has the freedom to adaptively decide whether or not to make null alignment based on our defined median distance d. Conversely, the marginal constraint on the target side remains unchanged, ensuring that each target vector is compared with the centric point e(∅).
To apply word alignment for hallucination and omission detections, we propose to analyze the number of non-aligned words from the final alignment matrix Γ. For both source and target sentences, we obtain the ratio of the misaligned words, given by:
r(R) = 1
m
m∑
i=1
I
  n∑
j=1
Γi,j > 0
  (7)
r(F ) = 1
n
n∑
j=1
I
( m∑
i=1
Γi,j > 0
) (8)
where I(·) is the indicator function. Further, we accumulate the transported mass to e(∅) as the confidence of missing alignments, given by:
c(R) = 1
n
n∑
j=1
P (∅,R) m+1,j ; c (F ) = 1
m
m∑
i=1
P (∅,F ) i,n+1
The combined score r(R) + c(R) is used to detect hallucinations, and r(F ) + c(F ) is for omissions.

heading: 4 Experimental Setup

heading: 4.1 Datasets and Evaluation Metrics
text: We conducted experiments using the recently proposed HalOmi benchmark (Dale et al., 2023b), which contains manually annotated data for MT hallucination and omission detection across 18 language pairs. It is constructed by pairing English sentences with translations in 5 high-resource languages (ar:Arabic, ru:Russian, es:Spanish,
de:German, and zh:Mandarin) and 3 lowresource languages (ks:Kashmiri, mni:Manipuri, and yo:Yoruba), and a one zero-shot3 SpanishYoruba pair. Each language pair includes between 145 to 197 examples, totaling 2,865 overall. Each example is annotated with 4 labels: no, small, partial, and full hallucination, with a similar scheme applied for omissions. Therefore, we utilized the multi-class ROC AUC variant, as defined in HalOmi (Dale et al., 2023b).

heading: 4.2 Baselines and Models
text: We conduct experiments to compare internal and external approaches for detecting hallucinations and omissions in MT, alongside our newly proposed word aligner-based approach. More precisely, consider Seq-Logprob, ALTI, ALTIT, and ATT-OT (originally proposed by (Guerreiro et al., 2023b)). Similarly, for external approaches, we report results from MT quality estimators LaBSE (Feng et al., 2022) and BLASER-QE (Barrault et al., 2023), as they are the best performers in this category according to (Dale et al., 2023b). We refer the readers to (Dale et al., 2023b) for a detailed of these baselines.

heading: 4.3 Implementation Details
text: Our implementation is based on PyTorch (Paszke et al., 2019), and we use the POT library4 for all the Optimal Transport-based methods. For the entropyregulized O (including the standard OT and partial OT), there is a regularization parameter ϵ that controls the sparsity of the OT solution, where 0.1 is the default value. In our experiments, we set ϵ to a low value of 0.05 to encourage the confidence of the sovler. However, we observed that the performance is not sensitive, but an overly small ϵ may lead to numerical instability.
In all experiments, we adopt a word-level representation approach which is consistent with standard practices in word alignment task (Sabet et al., 2020; Azadi et al., 2023). Specifically, we first generate token-level representations for each token in the sequence and establish a token-to-word index mapping. Then, we apply mean pooling to average the tokens representations corresponding to the same word, resulting in word-level representations. In our main experiment uses the representations of the last layer of LaBSE (Feng et al.,
3The MT system utilized for generating the data was not trained on Spanish-Yoruba parallel corpora.
4https://pythonot.github.io/
under 3 approaches: Internal , External , and Our word alignment . I1, I2, I3, I4 : Seq-Logprob, ALTI, ALTIT, and ATT-OT. E1, E2 : LaBSE and BLASER-QE MT quality estimators. Our : OTTAWA using LaBSE token embeddings. All scores are scaled within the range of [0, 100] following (Dale et al., 2023b).
2022). When experimenting with other models, we use the hidden representation of the 8th layer of mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020), respectively, following (Dou and Neubig, 2021; Chi et al., 2021; Azadi et al., 2023).
Our OTTAWA, PMIAlign, and the standard OT do need extra hyper-parameters to determine word alignments. However, for partial OT, we set m = 0.5 and tested a few thresholds. Specifically, we set τ to 0.1, 0.05, and 0.025 of the maximal value in marginal µ and ν, given by max(1/m, 1/n), and used the best threshold 0.05 to report the scores. To obtain the hallucination and omission, all baseline aligners use the reverse missing alignment ratio
Eq. 7 to estimate hallucination and use Eq. 8 to obtain omission scores.

heading: 5 Results

heading: 5.1 Sentence-Level Detection
text: Table 1 shows the performances of methods across 3 approaches for hallucination and omission detection across HalOmi’s 18 language pairs: 4 methods representing the internal approach, 2 for the external approach, and 1 showcasing our newly introduced word alignment approach. The latter approach employs our OTTAWA word aligner, which leverages LaBSE to obtain token-level representations.
Although our method performs best on only 5 out of 18 language pairs for hallucination and 7 out of 18 for omission, it outperforms other methods in terms of overall average performance. More precisely, we outperform all baselines in highresource languages for both hallucination and omission detection. However, we underperform in lowresource languages (except for omission detection) and zero-shot language pairs. Overall, although we underperform by 4% compared to BLASER-QE (E2) on average hallucination detection across the 18 language pairs, we surpass that model by 5% on omission. It is worth mentioning that BLASER-QE is a strong baseline, it is an MT quality estimator that leverages the SONAR (Duquenne et al., 2023) sentence embeddings. It was carefully tuned on 123.4k source-translation pairs, derived from outputs of 24 MT systems and covering 62 language pairs, all manually labeled by humans for MT quality estimation.
However, our method significantly improves upon LaBSE (E2), another MT quality estimator. While we report close tight (1% gain on HalOmi average) on hallucination, we significantly outperform it by 6% on omission. These results demonstrate that LaBSE’s token-level representations contain more expressive information for detecting omissions compared to the scalar values generated for MT quality estimation.
Finally, we observe that our method provides consistent and balanced performance in detecting both hallucinations and omissions compared to internal methods. For instance, although SeqLogprob (I1) matches our method on hallucinations, it significantly underperforms compared to us by 23% on omissions. Similarly, ALTIT (I3) has the smallest gap (2%) with our method on omission but performs poorly, achieving only 57% compared to our method’s 79%, on hallucination.

heading: 5.2 Ablation Study
text: We conduct ablation studies on our approach, testing variants that modify its two key components: the Embedding vector representation model and the word Aligner. Table 2 presents the average scores for both HalOmi high-resource (HR) and low-resource (LR) language pair clusters, along with the overall average. The complete results across the 18 language pairs are presented in Table 5 in Appendix B.
Embedding We conduct experiments with OTTAWA, utilizing embeddings derived from unsupervised pre-trained models such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020). These models are commonly used in crosslingual word alignment (Sabet et al., 2020; Azadi et al., 2023). We notice that OTTAWA with LaBSE representations significantly outperforms the bestunsupervised representation baseline (mBERT) by 6% and 3% on the overall HalOmi average score on hallucination and omission, respectively. This is expected given that LaBSE representations are more task-specific than those from unsupervised models, having been extensively fine-tuned for translation pair quality estimation using carefully curated data for low-resource languages.
Consequently, the performance gap between LaBSE and models such as mBERT is larger in low-resource languages (11% and %3 on omissions) compared to high-resource languages (4% and 0% on hallucinations). Overall, the results are promising, demonstrating that hallucinations and omissions can be detected even without access to the MT system or a robust MT quality estimator. This approach serves as an acceptable alternative, particularly in high-resource settings. However, its performance in low-resource languages necessitates further investigation.
Aligner We compared OTTAWA with state-ofthe-art word aligner methods, including SimAlign (Sabet et al., 2020) (Itermax), PMI-
Align (Azadi et al., 2023), standard OT (Chi et al., 2021; Dou and Neubig, 2021), and POT (Arase et al., 2023), all utilizing LaBSE representations.
We first observe that the standard alignment methods work reasonably well on detecting omissions but not hallucinations. This is an understandable outcome, which can be explained as follows: 1) the hallucinated translations often contain a significant amount of "detached" words. For example, the word "cat" may be translated to "dog". This is particularly problematic for standard alignment methods as they assume the word "cat" should have a corresponding word in the translation, resulting in aligning "cat" to "dog" nevertheless. 2) the omission errors in the HalOmi dataset are often early terminated translations. In this case, the translated words are still mapped to the source words, and the standard methods are still able to align the existing words and detect the omission errors.
However, only our method works well in the detection of both hallucinations and omissions, where there is a 7% overall average improvement over both PMIAlign and standard OT in hallucination errors. This outcome is consistent with our motivation that the common assumption among standard word-alignment tools, which is that a mapping exists between all source and target words, prevents them from effectively identifying all errors.

heading: 5.3 Distinguish Hallucination and Omission
text: Figure 2 shows LaBSE cosine similarity scores for samples with either hallucination or omission
labels, aggregated from the 10 high-resource language pairs datasets. We only focus on highresource languages, given that the performance of all methods significantly diminishes with lowresource ones. The primary goal of this experiment is to study the models’ abilities to differentiate between types of errors. Although LaBSE quality estimator can effectively detect both hallucinations and omissions, the figure clearly shows that it fails to distinguish between them. The majority of samples for both hallucinations and omissions exhibit cosine similarity scores below −0.8.
Figure 3 shows the hallucination (x-axis) and omission (y-axis) scores produced by OTTAWA(using LaBSE embeddings), on the same samples used in Figure 2. The figure demonstrates that our approach effectively distinguishes between omission and hallucination errors, with omission errors concentrated below the diagonal, and hallucination errors concentrated above it. In most cases, our word alignment approach, which operates without any supervision signal, accurately distinguishes omissions from hallucinations, and vice versa. This unique feature offered by our approach can help researchers to precisely diagnose and comprehend error types in MT systems, thereby facilitating more efficient development and the implementation of targeted solutions.

heading: 5.4 Word-Level Detection
text: Following the experimental setup in (Dale et al., 2023b), we evaluate our OTTAWA for word-level hallucination and omission detection. Table 3 com-
pares its performance with the best internal approach baselines identified in their study: LogProb (the Stand. variant) and ALTI (the sum variant). It is worth noting that the LogProb and ALTI baselines are different from those in Table 1 despite the name resemblance. Specifically, ALTI is a combination of two baselines from Table 1: ALTI (hallucination results), and ALTIT (omission results). Conversely, the LogProb baseline is obtained by processing the sentence pairs through the MT model twice (once for the source and once for the target), unlike SeqLogProb in Table 1, which involves only one pass.
Additionally, (Dale et al., 2023b) conducted ablations with these baselines, creating two variants for each: Standard and Contrastive for LogProb, and Mean and Max for ALTI. In Table 3, we report only the results for the best-performing variant of each baseline. Results indicate that word-level detection poses a greater challenge for our approach, which significantly under-performs compared to the best baseline (ALTI) by 8% and 7% in the overall average for hallucination and omission detection, respectively. However, we find the results encouraging when considering that unsupervised world-level detection with current state-of-the-art methods was not feasible without white-box access to the MT system, as external approaches operate on sentence level only.

heading: 5.5 Word Alignment
text: We run experiments on cross-lingual word alignment using 6 datasets, as in SimAlign (Sabet et al., 2020) and PMIAlign (Azadi et al., 2023), with gold word alignment labels. We includes English-Czech (en-cs) (Mareček, 2011), GermanEnglish (de-en) (Koehn, 2005), English-Persian (en-fa) (Tavakoli and Faili, 2014), English-French (en-fr) (Och and Ney, 2000), English-Hindi (enhi) (Koehn et al., 2005) and Romanian-English (ro-en) (Mihalcea and Pedersen, 2003) language
pairs. For evaluation metrics, we utilize the standard Alignment Error Rate (AER) (Och and Ney, 2003), and aligners use the mBERT representations.
As shown in Table 4, OTTAWA performs on par with another state-of-the-art word aligner on the cross-lingual alignment task. More precisely, it matches the best-performing baselines in three languages (en-cs, de-en, and en-hi), while slightly outperforming them by 1% in the en-fr and ro-en language pairs. We anticipated no significant gains in word alignment, as the datasets for cross-lingual word alignment are constructed from clean translations, making null alignments less concerning.

heading: 6 Conclusion and Future Work
text: In this work, we propose a new word alignmentbased approach for detecting hallucinations and omissions in MT systems. We develop OTTAWA, an innovative word aligner designed specifically for this purpose. While experiments show promise in MT hallucination and omission detection, this area remains an intriguing direction for future research exploration. We plan to focus on one-tomany alignments for pathological translations.

heading: 7 Limitations
text: Our Optimal Transport (OT)-based word alignment method relies on pretrained word embeddings, which may not be available for low and scarce resource languages. Also, the performance of our word aligner on low resources heavily depends on representations obtained from a supervised MT quality estimator, which may not be accessible or does not support certain languages. Another limitation of our study is the exclusive focus on null alignments, which are central to our task of interest. However, we do not address other complex cases such as one-to-many and many-to-many alignments.

heading: Acknowledgements
text: We thank the anonymous reviewers for their insightful comments. Osmar R. Zaïane is supported by the Amii Fellow Program.

heading: A Equidistant Vector
text: Assume that we have vectors e1, . . . , en in RD. Here we discuss how to find an equidistant vector e(∅) with respect to cosine distance. For that we need to solve a system of N − 1 equations:
dist(e∅, e1) = dist(e∅, ej)
for all j = 2, . . . , N .
Recall the definition of the cosine distance:
dist(x,y) = 1− x · y∥x∥∥y∥ .
Hence, the system of equations we need to solve is equivalent to the system of linear equations:
e∅ · e1 ∥e∅∥∥e1∥ = e∅ · ej ∥e∅∥∥ej∥ .
Slightly rewriting it, we get:
e∅ · (
e1 ∥e1∥ − ej∥ej∥
) = 0.
Note that usually for our word alignment problems N < D, hence this system always has a solution. However, we want to find the minimum possible equal distance so that the NULL vector is meaningful for the OT task. For that, assume that e(∅) lies in the span of the given vectors: e∅ = ∑N k=1 akek. Inserting it in the previous system, we obtain a new homogeneous system on the coefficients ak:
Ea = 0,
where Ejk = ek · (
e1 ∥e1∥ − ej ∥ej∥
) is a (N −1)×N
matrix. For general positions of word representation vectors, the kernel of E is one-dimensional. Projection onto it could be computed as IN−E+E, where E+ is Moore-Penrose inverse and IN is N × N identity matrix (Campbell and Meyer, 2009). In this case a will be the basis vector for the kernel subspace. After finding a, one can construct e∅ = ∑N k=1 akek and compute dmin = dist(e∅, e1). Note that there is no equidistant vector with a smaller distance than dmin.

heading: B Additional Results
text: We present the comparisons of different word embeddings and alignment methods in Table 5.
methods. OTTAWA refer to using OTTAWA word aligner with LABSE embeddings. W1, W2, W3, W4 : SimAlign-Itermax (Sabet et al., 2020), PMIAlign (Azadi et al., 2023), OT (Azadi et al., 2023), POT (Arase et al., 2023) word aligners with LaBSE embeddings. R1, R2 : OTTAWA with embeddings obtained from mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020).

heading: Future Work
text: While experiments show promise in MT hallucination and omission detection, this area remains an intriguing direction for future research exploration. We plan to focus on one-tomany alignments for pathological translations