Extrinsic Evaluation of Machine Translation Metrics
Year: 2023

Abstract:
Automatic machine translation (MT) metrics are widely used to distinguish the quality of machine translation systems across large test sets (i.e., system-level evaluation). However, it is unclear if automatic metrics can reliably distinguish good translations from bad at the sentence level (i.e., segment-level evaluation). We investigate how useful MT metrics are at detecting segment-level quality by correlating metrics with the translation utility for downstream tasks. We evaluate the segment-level performance of widespread MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks (dialogue state tracking, question answering, and semantic parsing). For each task, we have access to a monolingual task-specific model and a translation model. We calculate the correlation between the metric’s ability to predict a good/bad translation with the success/failure on the final task for machine-translated test sentences. Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of downstream outcomes. We also find that the scores provided by neural metrics are not interpretable, in large part due to having undefined ranges. We synthesise our analysis into recommendations for future MT metrics to produce labels rather than scores for more informative interaction between machine translation and multilingual language understanding.

heading: 1 Introduction
text: Although machine translation (MT) is typically seen as a standalone application, in recent years MT models have been more frequently deployed as a component of a complex NLP platform delivering multilingual capabilities such as cross-lingual information retrieval (Zhang et al., 2022) or automated multilingual customer support (Gerz et al., 2021). When an erroneous translation is generated by the MT systems, it may add new errors in the task pipeline leading to task failure and poor user ex-
perience. For example, consider the user’s request in Chinese 剑桥有牙买加菜吗？ (“Is there any good Jamaican food in Cambridge?”) machinetranslated into English as “Does Cambridge have a good meal in Jamaica?”. The model will erroneously consider “Jamaica” as a location, instead of cuisine, and prompt the search engine to look up restaurants in Jamaica 1. To avoid this breakdown, it is crucial to detect an incorrect translation before it causes further errors in the task pipeline.
One way to approach this breakdown detection is using segment-level scores provided by MT metrics. Recent MT metrics have demonstrated high correlation with human judgements at the system level for some language pairs (Ma et al., 2019). These metrics are potentially capable of identifying subtle differences between MT systems that emerge over a relatively large test corpus. These metrics are also evaluated on respective correlation with human judgements at the segment level, however, there is a considerable performance penalty (Ma et al., 2019; Freitag et al., 2021b). Segment-level evaluation of MT is indeed more difficult and even humans have low inter-annotator agreement on this task (Popović, 2021). Despite MT systems being a crucial intermediate step in several applications, characterising the behaviour of these metrics under task-oriented evaluation has not been explored.
In this work, we provide a complementary evaluation of MT metrics. We focus on the segmentlevel performance of metrics, and we evaluate their performance extrinsically, by correlating each with the outcome of downstream tasks with respective, reliable accuracy metrics. We assume access to a parallel task-oriented dataset, a task-specific monolingual model, and a translation model that can translate from the target language into the language of the monolingual model. We consider the Translate-Test setting — where at test time, the examples from the test language are translated to the
1Example from the Multi2WoZ dataset (Hung et al., 2022)
13060
task language for evaluation. We use the outcomes of this extrinsic task to construct a breakdown detection benchmark for the metrics.
We use dialogue state tracking, semantic parsing, and extractive question answering as our extrinsic tasks. We evaluate nine metrics consisting of string overlap metrics, embedding-based metrics, and metrics trained using scores from human evaluation of MT. Surprisingly, we find our setup challenging for all existing metrics; demonstrating poor capability in discerning good and bad translations across tasks. We present a comprehensive analysis of the failure of the metrics through quantitative and qualitative evaluation.
Our contributions are summarised as follows: 1) We derive a new breakdown detection task, for evaluating MT metrics, measuring how indicative segment-level scores are for downstream performance of an extrinsic cross-lingual task (Section 3). We evaluate nine metrics on three extrinsic tasks covering 39 unique language pairs. The task outputs, the breakdown detection labels, and metric outputs are publicly available. 2 2) We show that segment-level scores, from these metrics, have minimal correlation with extrinsic task performance (Section 4.1). Our results indicate that these scores are uninformative at the segment level (Section 4.3) — clearly demonstrating a serious deficiency in the best contemporary MT metrics. In addition, we find variable task sensitivity to different MT errors (Section 4.2). 3) We propose recommendations on developing MT metrics to produce useful segment-level output by predicting labels instead of scores and suggest reusing existing post-editing datasets and explicit error annotations (See Section 5).

heading: 2 Related Work
text: Evaluation of machine translation has been of great research interest across different communities (Nakazawa et al., 2022; Fomicheva et al., 2021). Notably, the Conference on Machine Translation (WMT) has been organising annual shared tasks on automatic MT evaluation since 2006 (Koehn and Monz, 2006; Freitag et al., 2021b) that invites metric developers to evaluate their methods on outputs of several MT systems. Metric evaluation typically includes a correlation of the scores with human judgements collected for the respective translation
2https://huggingface.co/datasets/uoe-nlp/ extrinsic_mt_eval
outputs. But, designing such guidelines is challenging (Mathur et al., 2020a), leading to the development of several different methodologies and analyses over the years.
The human evaluation protocols include general guidelines for fluency, adequacy and/or comprehensibility (White et al., 1994) on continuous scales (Koehn and Monz, 2006; Graham et al., 2013) (direct assessments) or fine-grained annotations of MT errors (Freitag et al., 2021a,b) based on error ontology like Multidimensional Quality Metrics (MQM) (Lommel et al., 2014) or rank outputs from different MT systems for the same input (Vilar et al., 2007). Furthermore, the best way to compare MT scores with their corresponding judgements is also an open question (Callison-Burch et al., 2006; Bojar et al., 2014, 2017). The new metrics claim their effectiveness by comparing their performance with competitive metrics on the latest benchmark.
The progress and criticism of MT evaluation are generally documented in a metrics shared task overview (Callison-Burch et al., 2007). For example, Stanojević et al. (2015) highlighted the effectiveness of neural embedding-based metrics; Ma et al. (2019) show that metrics struggle on segmentlevel performance despite achieving impressive system-level correlation; Mathur et al. (2020b) investigate how different metrics behave under different domains. In addition to these overviews, Mathur et al. (2020a) show that meta-evaluation regimes were sensitive to outliers and minimal changes in evaluation metrics are insufficient to claim metric efficacy. Kocmi et al. (2021) conducted a comprehensive evaluation effort to identify which metric is best suited for pairwise ranking of MT systems. Guillou and Hardmeier (2018) look at a specific phenomenon of whether metrics are capable of evaluating translations involving pronominal anaphora. Recent works have also criticised individual metrics such as COMET (Amrhein and Sennrich, 2022) and BERTScore (Hanna and Bojar, 2021).
These works draw their conclusions based on some comparison with human judgement or on specific pitfalls of individual metrics. Our work focuses on the usability of the metrics as solely judged on their ability to predict downstream tasks where MT is an intermediate step (with a primary emphasis on segment-level performance). Taskbased evaluation has been well studied (Jones and Galliers (1996); Laoudi et al. (2006); Zhang et al.
(2022), inter alia) but limited to evaluating MT systems rather than MT metrics. Closer to our work is Scarton et al. (2019); Zouhar et al. (2021) which proposes MT evaluation as ranking translations based on the time to post-edit model outputs. We borrow the term of breakdown detection from Martinovski and Traum (2003) that proposes breakdown detection for dialogue systems to detect unnatural responses.

heading: 3 Methodology
text: Our aim is to determine how reliable MT metrics are for predicting success on downstream tasks. Our setup uses a monolingual model (e.g., a dialogue state tracker) trained on a task language and parallel test data from multiple languages. We use MT to translate a test sentence (from a test language to the task language) and then infer a label for this example using the monolingual model. If the model predicts a correct label for the parallel task language input but an incorrect label for the translated test language input, then we have observed a breakdown due to a material error in the translation pipeline. We then study if the metric could predict if the translation is suitable for the end task. We refer to Figure 1 for an illustration. We frequently use the terms test language and task language to avoid confusion with the usage of source language and target language in the traditional machine translation setup. In Figure 1, the task language is English and the test language is Chinese. We now describe our evaluation setup and the metrics under investigation.

heading: 3.1 Setup
text: For all the tasks described below, we first train a model for the respective tasks on the monolingual
setup. We evaluate the task language examples on each task and capture the monolingual predictions of the model. We consider the Translate-Test paradigm (Hu et al., 2020), we translate the examples from each test language into the task language. The generated translations are then fed to the task-specific monolingual model. We use either (i) OPUS translation models (Tiedemann and Thottingal, 2020), (ii) M2M100 translation (Fan et al., 2021) or (iii) translations provided by the authors of respective datasets. Note that the examples across all the languages are parallel and we therefore always have access to the correct label for a translated sentence. We obtain the predictions for the translated data to construct a breakdown detection benchmark for the metrics.
We consider only the subset of examples in the test language which were correctly predicted in the task language to avoid errors that arise from extrinsic task complexity. Therefore, all incorrect extrinsic predictions for the test language in our setup arise from erroneous translation. This isolates the extrinsic task failure as the fault of only the MT system. We use these predictions to build a binary classification benchmark—all target language examples that are correctly predicted in the extrinsic task receive a positive label (no breakdown) while the incorrect predictions receive a negative label (breakdown).
We consider the example from the test language as source, the corresponding machine translation as hypothesis and the human reference from the task language as reference. Thus, in Figure 1, the source is 剑桥有牙买加菜吗？, the hypothesis is “Does Cambridge have a good meal in Jamaica", and the reference will be “Is there any good Jamaican food in Cambridge". These triples are then
scored by the respective metrics. After obtaining the segment-level scores for these triples, we define a threshold for the scores, thus turning metrics into classifiers. For example, if the threshold for the metric in Figure 1 is 0.5, it would mark both examples as bad translations. We plot a histogram over the scores with ten bins for every setup and select the interval with the highest performance on the development set as a threshold. The metrics are then evaluated on how well their predictions for a good/bad translation correlate with the breakdown detection labels.

heading: 3.2 Tasks
text: We choose tasks that contain outcomes belonging to a small set of labels, unlike natural language generation tasks which have a large solution space. This discrete nature of the outcomes allows us to quantify the performance of MT metrics based on standard classification metrics. The tasks also include varying types of textual units: utterances, sentences, questions, and paragraphs, allowing a comprehensive evaluation of the metrics.

heading: 3.2.1 Semantic Parsing (SP)
text: Semantic parsing transforms natural language utterances into logical forms to express utterance semantics in some machine-readable language. The original ATIS study (Hemphill et al., 1990) collected questions about flights in the USA with the corresponding SQL to answer respective questions from a relational database. We use the MultiATIS++SQL dataset from Sherborne and Lapata (2022) comprising gold parallel utterances in English, French, Portuguese, Spanish, German and Chinese (from Xu et al. (2020)) paired to executable SQL output logical forms (from Iyer et al. (2017)). The model follows Sherborne and Lapata (2023), as an encoder-decoder Transformer model based on mBART50 (Tang et al., 2021). The parser generates valid SQL queries and performance is measured as exact-match denotation accuracy—the proportion of output queries returning identical database results relative to gold SQL queries.

heading: 3.2.2 Extractive Question Answering (QA)
text: The task of extractive question answering is predicting a span of words from a paragraph corresponding to the question. We use the XQuAD dataset (Artetxe et al., 2020) for evaluating extractive question answering. The XQuAD dataset was obtained by professionally translating examples from the
development set of English SQuAD dataset (Rajpurkar et al., 2016) into ten languages: Spanish, German, Greek, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese, and Hindi. We use the publicly available question answering model that finetunes RoBERTa (Liu et al., 2019) on the SQuAD training set. We use the Exact-Match metric, i.e., the model’s predicted answer span exactly matches the gold standard answer span; for the breakdown detection task. The metrics scores are produced for the question and the context. A translation is considered to be faulty if either of the scores falls below the chosen threshold for every metric.

heading: 3.2.3 Dialogue State Tracking (DST)
text: In the dialogue state tracking task, a model needs to map the user’s goals and intents in a given conversation to a set of slots and values, known as a dialogue state, based on a pre-defined ontology. MultiWoZ 2.1 (Eric et al., 2020) is a popular dataset for examining the progress in dialogue state tracking which consists of multi-turn conversations in English spanning across 7 domains. We consider the Multi2WoZ dataset (Hung et al., 2022) where the development and test set have been professionally translated into German, Russian, Chinese, and Arabic from the MultiWoZ 2.1 dataset. We use the dialogue state tracking model trained on the English dataset by Lee et al. (2019). We consider the Joint Goal Accuracy where the inferred label is correct only if the predicted dialogue state is exactly equal to the ground truth to provide labels for the breakdown task. We use oracle dialogue history and the metric scores are produced only for the current utterance spoken by the user.

heading: 3.3 Metrics
text: We describe the metrics based on their design principles: derived from the surface level token overlap, embedding similarity, and neural metrics trained using WMT data. We selected the following metrics as they are the most studied, frequently used, and display a varied mix of design principles.

heading: 3.3.1 Surface Level Overlap
text: BLEU (Papineni et al., 2002) is a string-matching metric that compares the token-level n-grams of the hypothesis with the reference translation. BLEU is computed as a precision score weighted by a brevity penalty. We use sentence-level BLEU in our experiments. chrF (Popović, 2017) computes a character n-gram
F-score based on the overlap between the hypothesis and the reference.

heading: 3.3.2 Embedding Based
text: BERTScore (Zhang et al., 2020) uses contextual embeddings from pre-trained language models to compute the similarity between the tokens in the reference and the generated translation using cosine similarity. The similarity matrix is used to compute precision, recall, and F1 scores.

heading: 3.3.3 Trained on WMT Data
text: WMT organises an annual shared task on developing MT models for several categories in machine translation (Akhbardeh et al., 2021). Human evaluation of the translated outputs from the participating machine translation models is often used to determine the best-performing MT system. In recent years, this human evaluation has followed two protocols: (i) Direct Assessment (DA) (Graham et al., 2013): where the given translation is rated from 0 to 100 based on the perceived translation quality and (ii) Expert based evaluation where the translations are evaluated by professional translators with explicit error listing based on the Multidimensional Quality Metrics (MQM) ontology. MQM ontology consists of a hierarchy of errors and translations are penalised based on the severity of errors in this hierarchy. These human evaluations are then used as training data for building new MT metrics. COMET metrics: Cross-lingual Optimized Metric for Evaluation of Translation (COMET) (Rei et al., 2020) uses a cross-lingual encoder (XLM-R (Conneau et al., 2020)) and pooling operations to predict score of the given translation. Representations for the source, hypothesis, and reference (obtained using the encoder) are combined and passed through a feedforward layer to predict a score. These metrics use a combination of WMT evaluation data across the years to produce different metrics. In all the variants, the MQM scores and DA scores are normalised to z-scores to reduce the effect of outlier annotations. COMET-DA uses direct assessments from 2017 to 2019 as training data while COMET-MQM uses direct assessments from 2017 to 2021 as training data. This metric is then fine-tuned with MQM data from Freitag et al. (2021a). UniTE metrics (Wan et al., 2022), Unified Translation Evaluation, is another neural translation metric that proposes a multi-task setup for the three strategies of evaluation: source-hypothesis, source-
hypothesis-reference, and reference-hypothesis in a single model. The pre-training stage involves training the model with synthetic data constructed using a subset of WMT evaluation data. Fine-tuning uses novel attention mechanisms and aggregate loss functions to facilitate the multi-task setup.
All the above reference-based metrics have their corresponding reference-free versions which use the same training regimes but exclude encoding the reference. We refer to them as COMET-QE-DA, COMET-QE-MQM, and UniTE-QE respectively. COMET-QE-DA in this work uses DA scores from 2017 to 2020. We list the code sources of these metrics in Appendix B.

heading: 3.4 Metric Evaluation
text: The meta-evaluation for the above metrics uses the breakdown detection benchmark. As the class distribution changes depending on the task and the language pair, we require an evaluation that is robust to class imbalance. We consider using macro-F1 and Matthew’s Correlation Coefficient (MCC) (Matthews, 1975) on the classification labels. The range of macro-F1 is from 0 to 1 with equal weight to positive and negative classes. We include MCC to interpret the MT metric’s standalone performance for the given extrinsic task. The range of MCC is between -1 to 1. An MCC value near 0 indicates no correlation with the class distribution. Any MCC value between 0 and 0.3 indicates negligible correlation, 0.3 to 0.5 indicates low correlation.

heading: 4 Results
text: We report the aggregated results for semantic parsing, question answering, and dialogue state tracking in Table 1 with fine-grained results in Appendix D. We use a random baseline for comparison which assigns the positive and negative labels with equal probability.

heading: 4.1 Performance on Extrinsic Tasks
text: We find that almost all metrics perform above the random baseline on the macro-F1 metric. We use MCC to identify if this increase in macro-F1 makes the metric usable in the end task. Evaluating MCC, we find that all the metrics show negligible correlation across all three tasks. Contrary to trends where neural metrics are better than metrics based on surface overlap (Freitag et al., 2021b), we find this breakdown detection to be difficult irrespective
of the design of the metric. We also evaluate an ensemble with majority voting of the predictions from the top three metrics per task. Ensembling provides minimal gains suggesting that metrics are making similar mistakes despite varying properties of the metrics.
Comparing the reference-based versions of trained metrics (COMET-DA, COMET-MQM, UniTE) with their reference-free quality estimation (QE) equivalents, we observe that referencebased versions perform better, or are competitive to, their reference-free versions for the three tasks. We also note that references are unavailable when the systems are in production, hence reference-based metrics are unsuitable for realistic settings. We discuss alternative ways of obtaining references in Section 4.4.
Between the use of MQM-scores and DA-scores during fine-tuning COMET variants, we find that both COMET-QE-DA and COMET-DA are strictly better than COMET-QE-MQM and COMET-MQM for question answering and dialogue state tracking respectively, with no clear winner for semantic parsing (See Appendix D).
The results on per-language pair in Appendix D suggest that no specific language pairs stand out as easier/harder across tasks. As this performance is already poor, we cannot verify if neural metrics can generalise in evaluating language pairs unseen during training. Case Study: We look at Semantic Parsing with an English-trained parser tested with Chinese inputs for our case study with the well-studied COMET-
DA metric. We report the number of correct and incorrect predictions made by COMET-DA across ten equal ranges of scores in Figure 2. The bars labelled on the x-axis indicate the end-point of the interval i.e., the bar labelled -0.74 contains examples that were given scores between -1.00 and -0.74.
First, we highlight that the threshold is -0.028, counter-intuitively suggesting that even some correct translations receive a negative score. We expected the metric to fail in the regions around the threshold as those represent strongest confusion. For example, “周日下午从迈阿密飞往克利夫 兰” is correctly translated as “Sunday afternoon from Miami to Cleveland” yet the metric assigns it a score of -0.1. However, the metric makes mis-
takes throughout the bins. For example, “我需要 预订一趟联合航空下周六的从辛辛那提飞往纽 约市的航班” is translated as “I need to book a flight from Cincinnati to New York City next Saturday.” and loses the crucial information of “United Airlines”; yet it is assigned a high score of 0.51. This demonstrates that the metric possesses a limited perception of a good or bad translation for the end task.
We suspect this behaviour is due to the current framework of MT evaluation. The development of machine translation metrics largely caters towards the intrinsic task of evaluating the quality of a translated text in the target language. The severity of a translation error is dependent on the guidelines released by the organisers of the WMT metrics task or the design choices of the metric developers. Our findings agree with Zhang et al. (2022) that different downstream tasks will demonstrate varying levels of sensitivity to the same machine translation errors.

heading: 4.2 Qualitative Evaluation
text: To quantify detecting which translation errors are most crucial to the respective extrinsic tasks, we conduct a qualitative evaluation of the MT outputs and task predictions. We annotate 50 false positives and 50 false negatives for test languages Chinese (SP), Hindi (QA), and Russian (DST) respectively. The task language is English. We annotate the MT errors (if present) in these examples based on the MQM ontology. We tabulate these results in Table 2 using COMET-DA for these analyses.
Within the false negatives, a majority of the errors (>48%) are due to the metric’s inability to detect translations containing synonyms or paraphrases of the references as valid translations. Further, omission errors detected by the metric are not crucial for DST as these translations often exclude
pleasantries. Similarly, errors in fluency are not important for both DST and SP but they are crucial for QA as grammatical errors in questions produce incorrect answers. Mistranslation of named entities (NEs), especially which lie in the answer span, is a false negative for QA since QA models find the answer by focusing on the words in the context surrounding the NE rather than the error in that NE. Detecting mistranslation in NEs is crucial for both DST and SP as this error category dominates the false positives. A minor typo of Lester instead of Leicester marks the wrong location in the dialogue state which is often undetected by the metric. Addition and omission errors are also undetected for SP while mistranslation of reservation times is undetected for DST.
We also find that some of the erroneous predictions can be attributed to the failure of the extrinsic task model than the metric. For example, the MT model uses an alternative term of direct instead of nonstop while generating the translation for the reference “show me nonstop flights from montreal to orlando”. The semantic parser fails to generalise despite being trained with mBART50 to ideally inherit some skill at disambiguiting semantically similar phrases. This error type accounts for 25% for SP, 20% for QA and 5% in DST of the total annotated errors. We give examples in Appendix C.

heading: 4.3 Finding the Threshold
text: Interpreting system-level scores provided by automatic metrics requires additional context such as the language pair of the machine translation model or another MT system for comparison 3. In this classification setup, we rely on interpreting the segment-level score to determine whether the translation is suitable for the downstream task. We find that choosing the right threshold to identify trans-
3https://github.com/Unbabel/COMET/issues/18
Extrinsic Task SP QA DST
BLEU 15.5 ± 08.8 16.1 ± 04.9 20.0 ± 0.00 chrF 44.0 ± 13.7 53.9 ± 07.8 30.7 ± 0.45
BERTScore 0.50 ± 0.21 0.54 ± 0.08 0.39 ± 0.21 COMET-DA 0.21 ± 0.35 0.30 ± 0.23 0.58 ± 0.08 COMET-MQM 0.03 ± 0.01 0.06 ± 0.01 0.02 ± 0.00 UniTE 0.04 ± 0.22 -0.40 ± 0.38 -0.01 ± 0.29
COMET-QE-DA 0.02 ± 0.07 0.02 ± 0.01 0.06 ± 0.01 COMET-QE-MQM 0.11 ± 0.01 0.00 ± 0.04 0.03 ± 0.00
UniTE-QE -0.01 ± 0.22 -0.24 ± 0.13 0.11 ± 0.18
Table 3: Mean and Standard Deviation of the best threshold on the development set for all the language pairs in the respective extrinsic tasks. The thresholds are inconsistent across language pairs and tasks for both bounded and unbounded metrics.
lations requiring correction is not straightforward. Our current method to obtain a threshold relies on validating candidate thresholds on the development set and selecting an option with the best F1 score. These different thresholds are obtained by plotting a histogram of scores with ten bins per task and language pair.
We report the mean and standard deviation of best thresholds for every language pair for every metric in Table 3. Surprisingly, the thresholds are inconsistent and biased for bounded metrics: BLEU (0–100), chrF (0–100), and BERTScore (0–1). The standard deviations across the table indicate that the threshold varies greatly across language pairs. We find that thresholds of these metrics are also not transferable across tasks. COMET metrics, except COMET-DA, have lower standard deviations. By design, the range of COMET metrics in this work is unbounded. However, as discussed in the theoretical range of COMET metrics 4, empirically, the range for COMET-MQM lies between -0.2 to 0.2, questioning whether lower standard deviation is an indicator of threshold consistency. Some language pairs within the COMET metrics have negative thresholds. We also find that some of the use cases under the UniTE metrics have a mean negative threshold, indicating that good translations can have negative UniTE scores. Similar to Marie (2022), we suggest that the notion of negative scores for good translations, only for certain language pairs, is counter-intuitive as most NLP metrics tend to produce positive scores.
Thus, we find that both bounded and unbounded metrics discussed here do not provide segmentlevel scores whose range can be interpreted mean-
4https://unbabel.github.io/COMET/html/faqs. html
ingfully across tasks and language pairs.

heading: 4.4 Reference-based Metrics in an Online Setting
text: In an online setting, we do not have access to references at test time. To test the effectiveness of reference-based methods here, we consider translating the translation back into the test language. For example, for an en parser, the test language tizh is translated into mten and then translated back to Chinese as mtzh. The metrics now consider mten as source, mtzh as hypothesis, and tizh as the reference. We generate these new translations using the mBART50 translation model (Tang et al., 2021) and report the results in Table 4.
Compared to the results in Table 1, there is a further drop in performance across all the tasks and metrics. The metrics also perform worse than their reference-free counterparts. The second translation is likely to add additional errors to the existing translation. This cascading of errors confuses the metric and it can mark a perfectly useful translation as a breakdown. The only exception is that of the UniTE metric which has comparable performance (but overall poor) due to its multi-task setup.

heading: 5 Recommendations
text: Our experiments suggest that evaluating MT metrics on the segment level for extrinsic tasks has considerable room for improvement. We propose recommendations based on our observations:
Prefer MQM for Human Evaluation of MT outputs: We reinforce the proposal of using the MQM scoring scheme with expert annotators for evaluating MT outputs in line with Freitag et al. (2021a). As seen in Section 4.2, different tasks have varying tolerance to different MT errors. With explicit errors marked per MT output, future classifiers can be trained on a subset of human evaluation
data containing errors most relevant to the downstream application.

heading: MT Metrics Could Produce Labels over
text: Scores: The observations from Section 4.2 and Section 4.3 suggest that interpreting the quality of the produced MT translation based on a number is unreliable and difficult. We recommend exploring whether segment-level MT evaluation can be approached as an error classification task instead of regression. Specifically, whether the words in the source/hypothesis can be tagged with explicit error labels. Resorting to MQM-like human evaluation will result in a rich repository of human evaluation based on an ontology of errors and erroneous spans marked across the source and hypothesis (Freitag et al., 2021a). Similarly, the post-editing datasets (Scarton et al. (2019); Fomicheva et al. (2022) , inter alia) also provide a starting point. An interesting exploration in this direction are the works by Perrella et al. (2022); Rei et al. (2022) that treat MT evaluation as a sequence-tagging problem by labelling the errors in an example. Such metrics can also be used for intrinsic evaluation by assigning weights to the labels and producing a weighted score.
Add Diverse References During Training: From Section 4.2, we find that both the neural metric and the task-specific model are not robust to paraphrases. We also recommend the inclusion of diverse references through automatic paraphrasing (Bawden et al., 2020) or data augmentation during the training of neural metrics.

heading: 6 Conclusion
text: We propose a method for evaluating MT metrics which is reliable at the segment-level and does not depend on human judgements by using correlation MT metrics with the success of extrinsic downstream tasks. We evaluated nine different metrics on the ability to detect errors in generated translations when machine translation is used as an intermediate step for three extrinsic tasks: Semantic Parsing, Question Answering, and Dialogue State Tracking. We find that segment-level scores provided by all the metrics show negligible correlation with the success/failure outcomes of the end task across different language pairs. We attribute this result to segment scores produced by these metrics being uninformative and that different extrinsic tasks demonstrate different levels of sensitivity to different MT errors. We propose recommenda-
tions to predict error types instead of error scores to facilitate the use of MT metrics in downstream tasks.

heading: 7 Limitations
text: As seen in Section 4.2, sometimes the metrics are unnecessarily penalised due to errors made by the end task models. Filtering these cases would require checking every example in every task manually. We hope our results can provide conclusive trends to the metric developers focusing on segment-level MT evaluation.
We included three tasks to cover different types of errors in machine translations and different types of contexts in which an online MT metric is required. Naturally, this regime can be extended to other datasets, other tasks, and other languages (Ruder et al., 2021; Doddapaneni et al., 2022). Further, our tasks used stricter evaluation metrics such as exact match. Incorporating information from partially correct outputs is not trivial and will be hopefully addressed in the future. We have covered 37 language pairs across the tasks which majorly use English as one of the languages. Most of the language pairs in this study are high-resource languages. Similarly, the examples in multilingual datasets are likely to exhibit translationese - unnatural artefacts from the task language present in the test language during manual translation; which tend to overestimate the performance of the various tasks (Majewska et al., 2023; Freitag et al., 2020). We hope to explore the effect of translationese on MT evaluation (Graham et al., 2020) and extrinsic tasks in future. The choice of metrics in this work is not exhaustive and is dependent on the availability and ease of use of the metric provided by the authors.

heading: 8 Ethics Statement
text: This work uses datasets, models, and metrics that are publicly available. Although the scope of this work does not allow us to have an in-depth discussion of biases associated with metrics (Amrhein et al., 2022), we caution the readers of drawbacks of metrics that cause unfair evaluation to marginalised subpopulations which are discovered or yet to be discovered. We will release the translations, metrics scores, and corresponding task outputs for reproducibility.

heading: 9 Acknowledgements
text: We thank Barry Haddow for providing us with valuable feedback on setting up this work. We thank Arushi Goel and the attendees at the MT Marathon 2022 for discussions about this work. We thank Ankita Vinay Moghe, Nikolay Bogoychev, and Chantal Amrhein for their comments on the earlier drafts. We thank the anonymous reviewers for their helpful suggestions. This work was supported in part by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1) and the University of Edinburgh (Moghe). We also thank Huawei for their support (Moghe). Sherborne gratefully acknowledges the support of the UK Engineering and Physical Sciences Research Council (grant EP/W002876/1).

heading: A Language Codes
text: Please find the language codes in Table 5.
B Implementation Details
We provide the implementation details of metrics and models in Table 6. All models are publicly available and required no training from our side. The metrics BERTScore, COMET family and UniTE family can run on both GPU and CPU. If run on GPU, the metrics run under 5 minutes for a given task and given language pair. No hyperparameters are required. We follow the standard train-dev-test split as released by the authors for DST (Hung et al., 2022) and SP (Sherborne and Lapata, 2022). As no development set is available for the XQuAD dataset, we use the first 200 examples as development set to choose the threshold but report the performance on the full test set.

heading: C Errors of COMET-DA
text: The proportion of errors from Section 4.2 are listed in Table 2. We also provide error examples in Figure 3.

heading: D Task-specific results
text: We now list the results across every language pair for all the tasks in Tables tables 7 to 11.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 8
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. 3
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. 8

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Incorporating information from partially correct outputs is not trivial and will be hopefully addressed in the future. We have covered 37 language pairs across the tasks which majorly use English as one of the languages. Most of the language pairs in this study are high-resource languages. Similarly, the examples in multilingual datasets are likely to exhibit translationese - unnatural artefacts from the task language present in the test language during manual translation; which tend to overestimate the performance of the various tasks (Majewska et al. , 2023; Freitag et al. , 2020). We hope to explore the effect of translationese on MT evaluation (Graham et al. , 2020) and extrinsic tasks in future. The choice of metrics in this work is not exhaustive and is dependent on the availability and ease of use of the metric provided by the authors