Towards Precise Localization of Critical Errors in Machine Translation (2024)

Abstract: The advent of large language models has experienced a remarkable improvement in the field of machine translation. However, machine translation is still vulnerable to critical meaning deviations, which may incur catastrophic issues in social or ethical contexts. In particular, existing critical error detection primarily focuses on identifying sentence-level errors, leaving the precise localization of such errors within the sentence unaddressed. In this paper, we introduce a new task, word-level critical error detection (WCED), to detect critical errors at a fine-grained level in machine translation sentences. The task aims to identify the parts of a machine translation that contain catastrophic meaning distortions. We hypothesize that the ability to determine errors at the sentence level will positively influence the detection of more granular errors. We propose a sentence-level error detection module to predict which words in a sentence have critical errors. Experimental results demonstrate that our method outperforms existing methodologies and LLM in En-De, ZhEn, En-Ru, and En-Ko. Our method is helpful for determining the fine-grained location of errors. We hope that such studies will improve the capacity to address critical errors adeptly.

heading: 1 Introduction
text: Recent advancements in large language models (LLMs) have significantly improved the performance of machine translation (MT), leading to an increased demand for such systems (Moslem et al., 2023; Zhang et al., 2023; Wang et al., 2023; Xu et al., 2023). Concurrently, the necessity for quality estimation (QE), which automatically evaluates the output of MT systems, has also grown. QE measures the quality of MT outputs based on the source and the MT sentence without a human reference (Specia et al., 2009; Yankovskaya et al., 2019; Fomicheva et al., 2020; Zheng et al., 2021).
† Corresponding Author
Despite improvements in MT systems, critical errors remain a challenge (Karpinska and Iyyer, 2023). The QE sub-task, called critical error detection (CED), focuses on identifying instances where critical errors occur (Specia et al., 2021; Zerva et al., 2022). A critical error is a catastrophic meaning distortion with the potential to cause adverse personal or societal impacts (Sudoh et al., 2021; Freitag et al., 2021; Al Sharou and Specia, 2022). Hence, MT outputs must be checked and prevented from containing such critical errors. Although research into CED is conducted in response to this necessity, prior studies primarily concentrate on sentence-level verification to detect the presence of errors in MT sentences (Jiang et al., 2021; Chen et al., 2021; Rubino et al., 2021; Eo et al., 2022). This approach is insufficient as it fails to accurately identify the precise location of errors within sentences, making it challenging to
3000
determine the cause. To address the issues outlined above, we introduce a new task, word-level critical error detection (WCED), which focuses on identifying segments containing critical errors from the MT sentence. As shown in Figure 1, this task involves analyzing both the source and MT sentences to identify critical errors at the word level within the MT sentence. An example provided illustrates the mistranslation of “pacemaker” as “자동 심장 충격기 (automated external defibrillator)”. Such a translation error can lead to significant safety risks for patients due to the dissemination of incorrect information. Through WCED, we are able to identify the specific details of these translation errors, thereby enhancing interpretability.
To predict critical errors at the word level, We propose a method that utilizes a sentence-level detection module. We employ linear interpolation to incorporate the probability representations from the sentence-level detection module into the model calculations for the WCED. This is based on the hypothesis that with a separate module guiding errors within the sentence, errors will be more precisely recognized during the detailed analysis.
In the experiment, we demonstrate that the proposed method evaluates errors with higher accuracy than the baselines. Our model’s error detection capability outperforms cutting-edge LLMs such as GPT-3.5 (OpenAI-Blog, 2022) and GPT-4 (OpenAI, 2023). We compare our method with other approaches that utilize the sentence-level detection task, showing that our method is well-suited for the performance enhancement of WCED. Also, we conduct a qualitative analysis and find that our method reacts more sensitively to sentences containing critical errors. By indicating fine-grained critical translation error location, rather than merely detecting the presence of errors in sentences, we provide detailed information about the errors. We expect that incorporating such error detection into MT systems will inform users of potential risk areas. When linked with automatic post-editing, it enables corrections of specific erroneous parts, among other varied applications.
To summarize, our contributions are as follows:
• We enhance the comprehensibility of wordlevel critical errors by refining the sentencelevel CED into a more granular one at the word level.
• We propose a method to improve the perfor-
mance of the WCED. Our approach aims to increase sensitivity towards errors by utilizing the probability values from the sentence-level detection module.
• The experiments outperform the baseline and demonstrate the feasibility of our method. Therefore, our approach contributes to advancing the task of identifying and correcting critical errors in MT.

heading: 2 Related Work
text: QE was first proposed by Blatz et al. (2004), and its initial studies primarily relied on traditional natural language processing (NLP) techniques (Graham, 2015; Beck et al., 2016). However, the acceleration of deep learning, including the development of neural network architectures like Transformer (Vaswani et al., 2023) and BERT (Devlin et al., 2019), shifts the focus towards developing neural frameworks for QE. DeepQuest (Ive et al., 2018) proposes a framework that adopts a sentencelevel approach and generalizes it for documentlevel QE. OpenKiwi (Kepler et al., 2019) introduces a new open-source QE framework based on bidirectional LSTM. TransQuest (Ranasinghe et al., 2020) leverages a cross-lingual transformer, supporting two different architectures at the sentence level for QE.
The 2021 conference on machine translation (WMT21) introduces the CED as a sub-task for QE (Specia et al., 2021). Critical errors can appear as mistranslations, hallucinations, or deletions. Mistranslation is characterized by the incorrect translation of the source sentence, leading to a distortion of the original meaning. Hallucination refers to the introduction of content not present in the original sentence, while deletion involves the omission of content that is present in the source sentence. Prior research has introduced various methodologies for developing models that perform binary classification of critical errors. These methodologies encompass extracting sentence features (Jiang et al., 2021), employing task-specific classifiers (Chen et al., 2021), altering the architecture of models, and leveraging large volumes of synthetic data (Rubino et al., 2021), as well as incorporating additional information (Eo et al., 2022) to enhance performance.
Previous studies do not perform word-level detection on critical errors. Although existing research concentrates on identifying spans contain-
ing errors (Geng et al., 2023; Guerreiro et al., 2023; Kocmi and Federmann, 2023), these are not specifically designed to detect critical errors. This presents a significant challenge: even if a sentence contains a critical error, it is difficult to pinpoint where the error has occurred. Therefore, we propose the WCED that detects critical errors at the word level. This approach allows for a more detailed and accurate identification of critical errors within MT.

heading: 3 Word-level Critical Error Detection (WCED)
text: In this section, we define the task of word-level critical error detection (WCED) and propose methods to enhance the word-level error detection process. To provide appropriate guidance for exploring errors in a granular manner, we hypothesize an ideal sentence-level CED module capable of accurately identifying the presence of errors at the fine-grained level. Even in the absence of a perfect CED module, leveraging existing modules will be sufficient to assess their effectiveness in approximating our hypothesis.

heading: 3.1 Problem Formulation
text: WCED refers to identifying specific portions of a sentence that contain errors leading to significant changes in meaning, potentially causing confusion or misinterpretation for the reader. This task focuses on detecting critical errors at the word level. To achieve this, the model takes as input a source sentence s = {s1, s2, ..., sn} with length n and an MT sentence t = {t1, t2, ..., tm} with length m. The model assesses semantic differences between the source and MT sentences to determine words that contain critical errors. Consequently, each word ti in the MT sentence is associated with a label yi ∈ {ok, bad}, where 1 ≤ i ≤ m. The label ok indicates that the corresponding word ti is correctly translated, while bad signifies the presence of critical errors.

heading: 3.2 Proposed Method
text: As depicted in Figure 2, our proposed method involves taking a source sentence s and an MT sentence t as inputs and utilizing a model P (yi|s, t) to detect errors at the word level. The model concatenates the source and MT sentences using a separator token ([SEP]), and employs an encoder
structure to process this combined input. The encoder outputs a multi-dimensional last hidden state, represented by a d-dimensional vector. Word-level error detection extracts the hidden state corresponding to each token in the MT sentence. It determines whether each word contains an error, thereby providing word-level predictions ŷi ∈ {ok, bad}.
We introduce a sentence-level CED module P (c|s, t) that assesses the presence of critical errors in an MT sentence, adopting a binary classification approach for determining the existence of errors. Specifically, the CED module serves as a guide at the sentence level, providing broad indicators of errors within the sentence. The application of this module enables the WCED model to be more sensitively tuned to errors. This module infers the presence of errors at the sentence level by passing the hidden state of the first token ([CLS]) through a binary classification head. The binary variable c is represented as ok when no error is present in the MT sentence and bad when an error exists.
In this structure, leveraging the probability of the CED module P (c|s, t), we derive the final prediction score of error prediction at the word level as follows:
logS(yi) = λ logP (c|s, t) + logP (yi|s, t) (1)
where λ represents an additional weight used to adjust the influence of the CED module. By tuning this hyper-parameter, we can finely calibrate the contribution of the CED module to the final error detection score. This formula combines the probability of predicting an error at the word level from the WCED model P (y|s, t) with the probability of the existence of an error in the sentence obtained through the CED module P (c|s, t). The CED module is provided in pre-trained and frozen parameters during this process. The loss function L is defined as:
L = − 1 m
m∑
i=1
wyi logS(yi) (2)
where m denotes the number of tokens in the sentence, and wyi denotes the class weights given for the ok and bad tags. Through this approach, we present a WCED model that effectively responds to errors and more precisely identifies errors.

heading: 3.2.1 Inference
text: Each word in MT sentences is processed through the token-level linear layers, resulting in a distribution. We calculate the average of these distributions to construct a set of word-level tags, ŷ = {ŷ1, ŷ2, ..., ŷm}. From these tags, words identified as critical errors are marked with <v> and </v>.

heading: 4 Experiments

heading: 4.1 Settings
text: Datasets We leverage datasets from the WMT23 QE (Blain et al., 2023), which are founded on the expert-based multi-dimensional quality metrics (MQM) dataset (Freitag et al., 2021), to collect data for WCED across English-German (En-De), Chinese-English (Zh-En), and English-Russian (En-Ru) language pairs. This dataset defines major errors as those that result in significant alterations to the intended meaning, potentially leading to user misunderstandings. Given the contextual alignment with the definition of critical errors, we selectively extract sentences manifesting major errors from the dataset. We select data comprised exclusively of sentences that have reached a consensus on the presence of such errors and employ it within our experimental framework.
We utilize a dataset for the CED task in the English-Korean (En-Ko) language pair, employing an En-Ko CED dataset1. The En-Ko CED dataset comprises target sentences with critical errors and reference sentences. This dataset is designed to intentionally insert critical errors into the reference sentences, allowing for the identification of incorrect words by comparing the target sentences with the reference sentences. After conducting this task with GPT-4 and reviewing the overall results generated, we are confident that the task was executed as intended. We provide statistical details of the datasets in Appendix A.
Implementation Details The implementation is predicated on the COMET GitHub repository2, with experiments conducted using the large versions of XLM-RoBERTa (Conneau et al., 2020) and InfoXLM (Chi et al., 2021) as the backbone. Training is carried out on an NVIDIA RTX A6000 GPU, with a batch size 32 for 20 epochs. The criterion for early stopping is set at 10 epochs. Performance evaluations of the model are conducted in 2
1https://www.aihub.or.kr/ 2https://github.com/Unbabel/COMET/
epochs during training, with fine-tuning processes taking approximately two hours. The AdamW optimizer is employed for model optimization, setting the learning rate at 1.5e-05 during the finetuning. All hyperparameters are manually adjusted. We train a CED module utilizing mBERT (Libovický et al., 2019), XLM-RoBERTa, InfoXLM, and COMETKIWI (Rei et al., 2022). The training settings are consistent with those applied in the training of the WCED model.
Evaluation We adopt F1 score, recall, and precision in alignment with the evaluation criteria set forth by the WMT23 (Blain et al., 2023) error span detection task. Additionally, we incorporate the Matthews correlation coefficient (MCC) (Chicco and Jurman, 2020) as an evaluation metric for sentence-level detection. This metric, previously utilized in the evaluations of WMT21 (Specia et al., 2021) and WMT22 (Zerva et al., 2022) QE tasks, assesses the correlation between actual and predicted error detections.
Baselines Our method is compared against baselines composed of multilingual language models and methods applicable to sentence-level error detection.
• Fine-Tuning (FT): We employ mBERT, XLM-RoBERTa, and InfoXLM, which are multilingual language models, as our baselines, fine-tuning them.
• Multi-Task Learning (MTL): This method aims to enhance the performance of the primary task by simultaneously training on several related tasks (Ruder, 2017). It leverages the shared knowledge gained from various tasks to aid the model in learning more generalized representations. In this paper, we show the performance of conducting CED alongside WCED. The MTL model concurrently predicts sentence-level label c and word-level label yi. This framework is built upon the COMETKIWI and is capable of executing two tasks. Sentence-level modeling predicts the overall sentence evaluation score using the hidden state of the first input token, whereas word-level modeling assesses the correctness of MT tokens. We convert sentence-level score prediction into sentence-level binary classification for utilization.
• Soft Prompt Tuning (SPT): This approach involves augmenting the model’s input with
prompts represented in vector representation (Lester et al., 2021; Liu et al., 2022). We incorporate the last hidden state of the CED module into the input embeddings for use as input. This method facilitates knowledge transfer from the CED module to the WCED model, aiding in generating appropriate outputs. We maintain only the added embedding portions as trainable layers while training for the WCED task.
• Adaptive Prompting (AP): The vector representation used in SPT is modified into a form understandable in natural language. This approach involves integrating the results from the CED module into the text input of the WCED model. If the CED module predicts the presence of errors in the source and translated sentences, it adds the word ‘terrible’; otherwise, it adds ‘great’ in natural language. This prefix makes the presence of errors in the sentences more explicit, thereby providing a more intuitive reflection of the CED module’s outcomes in executing the WCED task.

heading: 4.2 Main Results
text: As shown in Table 1, we report the performance of predicting word-level critical errors. Our proposed method demonstrates enhanced performance for the majority of language pairs when compared to the comparative methods.
In our experiments, we select the InfoXLM and XLM-R models, which demonstrate significantly superior performance among the models considered in FT. Our method is an augmentation to the FT, allowing for direct comparison. Notably, our method achieves remarkable performance with InfoXLM, resulting in F1 scores of 0.4272, 0.4333, 0.7002, and 0.4037 for each language pair, compared to the FT scores of 0.3782, 0.3930, 0.6712, and 0.3492. These improvements can be attributed to utilizing methods within the CED module that provide precise guidelines for error detection.
The application of MTL reveals a performance decline across all language pairs relative to the FT, indicating that conducting sentence-level and wordlevel error detection simultaneously negatively impacts word-level detection. The results suggest that focusing on a single task within a single model may be more effective.
While the SPT method shows less efficient performance than ours, it still represents an improve-
ment over the FT. This improvement indicates that the hidden state of the CED module aids wordlevel detection, underscoring the positive impact of utilizing this module. However, we demonstrate through experimentation that using the representation from the CED module as input is less efficient than our approach, which directly adjusts the error prediction probability.
For the AP, results indicate a decrease in performance compared to FT for the En-De, Zh-En, and En-Ru. This method introduces confusion to the model by providing more definitive natural language outcomes rather than probabilities regarding the presence of errors. Conversely, the performance of the En-Ko language pair is superior with an F1 score of 0.4303 compared to all benchmarked methods. This suggests that providing certainty in sentences where the model is unsure about errors can be beneficial. However, it is important to note that, unlike the proposed method, AP does not consistently show performance improvements across all datasets.

heading: 4.3 Comparative Performance with Large Language Models
text: We experiment to compare the performance of LLMs in the WCED task against our method. We evaluate GPT-4 and GPT-3.5 in both zero-shot and few-shot settings. For GPT-4, we use the ‘gpt-40125-preview’ version; for GPT-3.5, we use the ‘gpt-3.5-turbo-0125’ version. We include a detailed description of the WCED task in the prompts and provide five demonstrations for the few-shot setting. Appendix B shows the prompts used in this experiment. We further present the performance of GEMBA-MQM (Kocmi and Federmann, 2023), a GPT-based approach for translation quality assessment.
Table 1 shows a comparison between the performance outcomes of ChatGPT and our method. GPT-4 demonstrates superior performance compared to GPT-3.5, showing significant improvements in zero-shot, few-shot scenarios, and GEMBA-MQM. GPT-4 exhibits relatively robust performance in the few-shot setting in the En-De,
En-Ru, and En-Ko language pairs. However, the examples did not positively impact task comprehension in the Zh-En. Our model significantly outperforms both GPT-3.5 and GPT-4 in the En-De, Zh-En, and En-Ru, indicating that the WCED task presents challenges for LLMs and highlighting the necessity of this work. Even in constrained settings, GPT-4 slightly surpasses the performance of our method in the En-Ko. However, despite its good performance, GPT-4 faces limitations such as cost per token and extended generation times. In contrast, our method is significant in that it can be rapidly utilized for error detection when integrated with MT systems, offering a practical advantage over the computational and financial costs associated with models like GPT-4.

heading: 5 Analysis

heading: 5.1 The Impact of Weight on CED Module
text: Figure 3 investigates the impact of λ, a weight that modulates the influence of the CED module, on the model by adding the probability P (c|s, t) from the CED module to P (yi|s, t) from the WCED model. For all language pairs, performance peaks when λ is set to 1 and diminishes from this point. The improvement in performance when λ is set to 0, where the CED module has no influence, suggests that both excessively low and high weights can negatively affect the model’s performance.

heading: 5.2 The Performance of CED Module
text: The performance of the CED module utilized in our experiments is presented in Table 3. Our experiments employ the model with the highest MCC for each language pair. As FT, we use mBERT, InfoXLM, and XLM-R, with the COMET method aiming to evaluate errors at the sentence level through InfoXLM and XLM-R models. Compared to the FT, the COMET demonstrates superior performance in this task. Unlike the word-level error detection, XLM-R generally exhibits higher performance than InfoXLM, particularly for EnRu and En-Ko, where InfoXLM scores are notably lower at 0.5178 and 0.3761, respectively, compared to the FT scores of 0.7601 and 0.5203. This indicates that while InfoXLM is efficient for word-level detection, XLM-R models are more effective for sentence-level detection.

heading: 5.3 The Impact of Using Separate Probabilities for each Label
text: Table 2 demonstrates the effect of adding the probability from the CED module on the labels. It involves combining the probability values corresponding to the absence of errors (ok) and the presence of errors (bad) from the sum of the probabilities from both the CED module and the WCED model. Thus, the experiment may involve operations that add values to only one of the labels, ok
or bad. The conclusion is that applying values to all labels across all language pairs proves superior. It is observed that adjusting values for only one label significantly decreases performance, especially in the En-De and En-Ru pairs. Moreover, for all language pairs except En-Ru, adjustments made to the ok label, indicating the absence of errors, have a more positive impact than adjustments made to the bad label.

heading: 5.4 Qualitative Examples
text: Figure 4 provides examples generated by the model for each language pair. For En-De, the source phrase ‘solvent trading’ implies a company operating continuously without debt, but its literal translation could be misinterpreted as trading in chemical solvents—an error not detected by the FT. Both GPT-4 and our method identify the erroneous segment, with our method explicitly focusing on the error. While the source mentions a specific company name in the Zh-En, the translation ambiguously renders it. The FT fails to detect this, and GPT-4 extracts an unrelated segment, whereas our
model accurately identifies the error’s location. For En-Ru, a misinterpretation regarding the subject of travel is noted. Our method accurately predicts this, whereas the FT detects only typographical errors, and GPT-4 includes ranges without errors. The En-Ko example demonstrates that GPT-4 and ours correctly identify sentences without errors, compared to FT. Consequently, our method exhibits a superior ability to detect critical errors compared to GPT-4 accurately and shows greater sensitivity to errors than FT.

heading: 6 Conclusion
text: In this study, we introduced WCED to detect fine-grained critical errors in MT. This task aimed to identify catastrophic meaning distortions at the word level effectively. We proposed a method that utilizes a sentence-level error detection module to predict words containing critical errors within sentences. Experimental results demonstrated that our proposed method significantly enhanced the accuracy of error detection compared to comparison
methods and ChatGPT. These findings indicated that a sentence-level detection module could accurately guide the task of detecting word-level errors. Our approach contributed to a more nuanced evaluation and understanding of MT quality and was crucial in preventing potential issues caused by critical errors. Future research may improve the WCED performance and extend its applicability across various languages and domains. Also, we hope that our methods can be adopted in developing and evaluating MT systems to improve their ability to cope with critical errors.
Limitations
The proposed method exhibits certain limitations in terms of computational efficiency. While employing the CED module, we fix its parameters to reduce computational overhead, leading to increased memory usage compared to conventional models. Also, as the module’s output needs to be generated for every sentence during the training process, the total training time extends by the module’s inference duration. Nevertheless, once training is complete, it demonstrates advantages in inference costs.
The CED module’s applicability is limited as it relies on the same training dataset as the model it is integrated with. To overcome these limitations and achieve similar or improved performance, we can address the issue of generality by enhancing the task with models trained on datasets from different tasks. Future study and development efforts should focus on concretizing and validating these ideas, thereby maximizing the practicality and efficiency of the model. Given the significance of preventing critical errors in MT systems, we hope our work will significantly contribute to advancements in this field.
Ethics Statement
We address the potential negative impact on individuals or society that may arise from erroneous translations, including the possibility of ethical issues related to race, gender, and religion. Our objective is to prevent such severe consequences and enhance the reliability of MT systems.

heading: Acknowledgements
text: This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea
government(MSIT)(RS-2024-00398115, Research on the reliability and coherence of outcomes produced by Generative AI). This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(NRF-2021R1A6A1A03045425). This work was supported by ICT Creative Consilience Program through the Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT)(IITP-2024-2020-0-01819).

heading: A Dataset Details
text: We propose statistical information about the dataset in Table 4. We follow previous study (Specia et al., 2021) and keep the proportion of labels containing errors at around 10% across all language pairs. The dataset does not include other minor errors.

heading: B Prompt Example for ChatGPT
text: We provide ChatGPT with the following instructions, which include a definition of a critical error and a detailed description of the task:
You are an expert at detecting the span of critical errors in translations. Given a source sentence and a translated sentence , print out the parts of the entire sentence containing critical errors , marked with <v> and </v>. A critical error is a translation error that completely changes the meaning of the source text to the extent that it can have a negative impact on individuals or society. Critical errors appear in the following forms:
- Mistranslation occurs when essential content is inaccurately translated , resulting in a change of meaning , or when it is either not translated at all (remaining in the source language) or translated into incomprehensible text.
- Hallucination refers to the addition of critical content in the translation that does not exist in the source , such as the insertion of profanity not present in the original text. - Deletion involves the omission of crucial content from the source sentence in the translation , which can include the removal of negations or offensive words that were present in the original.
We want to detect words in the translated sentence with these critical errors. If there are no critical errors , print `NONE '. No additional annotations are required.
Source: {source sentence} Translation: {target sentence}

heading: C Efficiency of CED module in LLM
text: Table 5 presents the results of experiments conducted by integrating the sentence-level CED module outputs with the LLM baseline. The results demonstrate a notable performance improvement, indicating that the CED module enhanced the performance of the LLM. This aligns with the trends demonstrated in our proposed method, which emphasizes the efficiency of integrating sentence-level features. Nevertheless, the performance indicates that our method, which controls probability during training, is more effective.

heading: Future Work
text: Future research may improve the WCED performance and extend its applicability across various languages and domains. Also, we hope that our methods can be adopted in developing and evaluating MT systems to improve their ability to cope with critical errors. Limitations
The proposed method exhibits certain limitations in terms of computational efficiency. While employing the CED module, we fix its parameters to reduce computational overhead, leading to increased memory usage compared to conventional models. Also, as the module’s output needs to be generated for every sentence during the training process, the total training time extends by the module’s inference duration. Nevertheless, once training is complete, it demonstrates advantages in inference costs. The CED module’s applicability is limited as it relies on the same training dataset as the model it is integrated with. To overcome these limitations and achieve similar or improved performance, we can address the issue of generality by enhancing the task with models trained on datasets from different tasks. Future study and development efforts should focus on concretizing and validating these ideas, thereby maximizing the practicality and efficiency of the model. Given the significance of preventing critical errors in MT systems, we hope our work will significantly contribute to advancements in this field