The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics
Year: 2023

Abstract:
Neural metrics for machine translation evaluation, such as COMET, exhibit significant improvements in their correlation with human judgments compared to traditional metrics based on lexical overlap, such as BLEU. Yet neural metrics are, to a great extent, “black boxes” that return a single sentence-level score without transparency about the decisionmaking process. In this work, we develop and compare several neural explainability methods and demonstrate their effectiveness for interpreting state-of-the-art fine-tuned neural metrics. Our study reveals that these metrics leverage token-level information that can be directly attributed to translation errors, as assessed through comparison of token-level neural saliency maps with Multidimensional Quality Metrics (MQM) annotations and with synthetically-generated critical translation errors. To ease future research, we release our code at https://github.com/Unbabel/COMET/ tree/explainable-metrics.

heading: 1 Introduction
text: Reference-based neural metrics for machine translation evaluation are achieving evergrowing success, demonstrating superior results over traditional lexical overlap-based metrics, such as BLEU (Papineni et al., 2002) and CHRF (Popović, 2015), in terms of both their correlation with human ratings and their robustness across diverse domains (Callison-Burch et al., 2006; Smith et al., 2016; Mathur et al., 2020; Kocmi et al., 2021; Freitag et al., 2022). However, lexical overlapbased metrics remain popular for evaluating the performance and progress of translation systems and algorithms. Concerns regarding trust and interpretability may help explain this (Leiter et al., 2022): contrary to traditional metrics, neural metrics are considered “black boxes” as they often use
∗ Equal contribution. Corresponding author: ricardo.rei@unbabel.com
increasingly large models (e.g., the winning metric of the WMT 22 Metrics shared task was a 10B parameter model (Freitag et al., 2022)).
While some recent work has focus on explaining the predictions made by reference-free quality estimation (QE) systems (Fomicheva et al., 2021; Zerva et al., 2022), explaining reference-based metrics has remained a largely overlooked problem (Leiter et al., 2022). It is an open question whether the observations from studies of explainable QE carry over to this scenario. Thus, in this work, we fill that gap by turning to state-of-theart reference-based metrics—we aim to interpret their decision-making process by exploiting the fact that these metrics show consistently good correlations with Multidimentional Quality Metrics (MQM) (Freitag et al., 2021, 2022; Sai et al., 2022), which are fine-grained quality assessments that result from experts identifying error spans in translation outputs (Lommel et al., 2014). We hypothesize that reference-based metrics leverage this tokenlevel information to produce sentence-level scores. To test this hypothesis, we assess whether our explanations – measures of token-level importance obtained via attribution and input attribution methods such as attention weights and gradient scores (Treviso et al., 2021; Rei et al., 2022b) – align with
1089
human-annotated spans (Fomicheva et al., 2021, 2022; Zerva et al., 2022), as illustrated in Figure 1.
Our analysis focuses on two main vectors: (i) understanding the impact of the reference information on the quality of the explanations; and (ii) finding whether the explanations can help to identify potential weaknesses in the metrics. Our main contributions are: • We provide a comparison between multiple ex-
plainability methods for different metrics on all types of evaluation: src-only, ref-only, and src+ref joint evaluation.
• We find that explanations are related to the underlying metric architecture, and that leveraging reference information improves the explanations.
• We show that explanations for critical translation errors can reveal weaknesses in the metrics.

heading: 2 Explaining Neural Metrics
text: We aim to explain sentence-level quality assessments of reference-based metrics by producing token-level explanations that align to translation errors. In what follows, we describe the metrics and how we produce the explanations that we study.

heading: 2.1 Metrics
text: We focus our analysis on two state-of-the-art neural metrics: COMET (Rei et al., 2020) and UNITE (Wan et al., 2022).1 While both metrics use a multilingual encoder model based on XLMR (Conneau et al., 2020), they employ distinct strategies to obtain sentence-level quality scores. On the one hand, COMET separately encodes the source, translation and reference to obtain their respective sentence embeddings; these embeddings are then combined to compute a quality score. On the other, UNITE jointly encodes the sentences to compute a contextualized representation that is subsequently used to compute the quality score. Interestingly, UNITE is trained to obtain quality scores for different input combinations: [mt; src] ( SRC ), [mt; ref] (REF), and [mt; src; ref] (SRC+REF). In fact, when the input is SRC , UNITE works like TransQuest (Ranasinghe et al., 2020); REF, like BLEURT (Sellam et al., 2020); and SRC+REF, like ROBLEURT (Wan et al., 2021).
1Ensembles composed of these two metrics were respectively ranked second and third in WMT 2022 Metrics shared task. The winner of WMT 2022 Metrics task — METRICXXL — is not publicly available (Freitag et al., 2022).

heading: 2.2 Explanations via Attribution Methods
text: In this work, we produce explanations using attribution methods that assign a scalar value to each translation token (i.e. a token-level attribution) to represent its importance. While many input attribution methods exist and have been extensively studied in the literature (Ribeiro et al., 2016; Shrikumar et al., 2017; Sundararajan et al., 2017; Jain and Wallace, 2019; Atanasova et al., 2020; Zaman and Belinkov, 2022), we focus specifically on those that have been demonstrated to be effective for explaining the predictions of QE models (Treviso et al., 2021; Fomicheva et al., 2022; Fernandes et al., 2022; Zerva et al., 2022) and extend them to our reference-based scenario. Concretely, we use the following techniques to extract explanations:2
• embed–align: the maximum cosine similarity between each translation token embedding and the reference and/or source token embeddings (Tao et al., 2022);
• grad ℓ2: the ℓ2-norm of gradients with respect to the word embeddings of the translation tokens (Arras et al., 2019);
• attention: the attention weights of the translation tokens for each attention head of the encoder (Treviso et al., 2021);
• attn × grad: the attention weights of each head scaled by the ℓ2-norm of the gradients of the value vectors of that head (Rei et al., 2022b).

heading: 3 Experimental Setting
text: MQM annotations. We use MQM annotations from the WMT 2021 Metrics shared task (Freitag et al., 2021),3 covering three language pairs — English-German (en→de), English-Russian (en→ru), and Chinese-English (zh→en) —in two different domains: News and TED Talks. For each incorrect translation, human experts marked the corresponding error spans. In our framework, these error spans should align with the words that the attribution methods assign higher importance to.
2For all attention-based methods, we ensemble the explanations from the top 5 heads as this has shown to improve performance consistently over selecting just the best head (Treviso et al., 2021; Rei et al., 2022b). Moreover, we use the full attention matrix, instead of relying only on cross attention information.
3https://github.com/google/ wmt-mqm-human-evaluation
models on the MQM data. ⋆Although UNITE SRC is a src-only evaluation metric, it was trained with reference information (Wan et al., 2022).
Models. For COMET, we use the latest publicly available model: wmt22-comet-da (Rei et al., 2022a).4 For UNITE, we train our own model using the same data used to train COMET in order to have a comparable setup5. We provide full details (training data, correlations with human annotations, and hyperparameters) in Appendix A. Overall, the resulting reference-based UNITE models (REF and SRC+REF) are on par with COMET.
Evaluation. We want our explanations to be directly attributed to the annotated error spans, in the style of an error detection task. Thus, we report Area Under Curve (AUC) and Recall@K.6 These metrics have been used as the main metrics in previous works on explainable QE (Fomicheva et al., 2021, 2022; Zerva et al., 2022).

heading: 4 Results

heading: 4.1 High-level analysis
text: Explanations are tightly related to the underlying metric architecture. The results in Ta-
4https://huggingface.co/Unbabel/ wmt22-comet-da
5Our implementation differs from the original work by Wan et al. (2022), See Appendix A for full details.
6In this setup, Recall@K is the proportion of words with the highest attribution that correspond to translation errors against the total number of errors in the annotated error span.
ble 1 show that the predictive power of the attribution methods differ between UNITE and COMET: attn × grad is the best method for UNITEbased models, while embed–align works best for COMET.7 This is expected as UNITE constructs a joint representation for the input sentences, thus allowing attention to flow across them; COMET, in contrast, encodes the sentences separately, so it relies heavily on the separate contextualized embeddings that are subsequently combined via elementwise operations such as multiplication and absolute difference. Interestingly, embed–align and attn × grad were the winning explainability approaches of the WMT 2022 Shared-Task on Quality Estimation (Zerva et al., 2022). This suggests that explainability methods developed for QE systems can translate well to reference-based metrics. We provide examples of explanations in Appendix C.
Reference information boosts explainability power. Table 1 also shows that, across all metrics, using reference information brings substantial improvements over using only the source information. Moreover, while reference-based attributions significantly outperform source-based attributions, combining the source and reference information to
7In Appendix B, we provide a comparison between the explanations obtained via embed–align with COMET and with its pretrained encoder model, XLM-R.
for COMET, UNITE REF and UNITE SRC+REF in terms of Recall@K on translations with critical errors: negations (NEG), hallucinations (HALL), named entity errors (NE), and errors in numbers (NUM).
obtain token-level attributions does not consistently yield superior results over using the reference alone. Notably, the best attribution method for COMET does not require any source information. This is interesting: in some cases, reference-based metrics may largely ignore source information, relying heavily on the reference instead.

heading: 4.2 How do the explanations fare for critical translation errors?
text: The MQM data analyzed until now consists primarily of high quality translations, with the majority of annotated errors being non-critical. However, it is important to assess whether our explanations can be accurately attributed to critical errors, as this may reveal potential metric shortcomings. To this end, we employ SMAUG (Alves et al., 2022)8, a tool designed to generate synthetic data for stresstesting metrics, to create corrupted translations that contain critical errors. Concretely, we generate translations with the following pathologies: negation errors, hallucinations via insertions, named entity errors, and errors in numbers.9
Explanations identify critical errors more easily than non-critical errors. Figure 2 shows that explanations are more effective in identifying critical errors compared to other non-critical errors (see Table 1). Specifically, we find significant performance improvements up to nearly 30% in Recall@K for certain critical errors. Overall, hallucinations are the easiest errors to identify across all neural metrics. This suggests that neural
8https://github.com/Unbabel/smaug 9We corrupt fully correct translations that are not an exact copy of the reference translation. Moreover, as the full suit of SMAUG transformations can only be applied to English data, we focus solely on zh→en translations. Overall, the synthetic dataset consists of 2610 translations. Full statistics about the corrupted data and examples are shown in Appendix A.2.
metrics appropriately identify and penalize hallucinated translations, which aligns with the findings of Guerreiro et al. (2022). Moreover, explanations for both UNITE models behave similarly for all errors except numbers, where the source information plays a key role in improving the explanations. Notably, contrary to what we observed for data with non-critical errors, COMET explanations are less effective than those of UNITE REF and UNITE SRC+REF for identifying critical errors.
Explanations can reveal potential metric weaknesses. Figure 2 suggests that COMET explanations struggle to identify localized errors (negation errors, named entity errors and discrepancies in numbers). We hypothesize that this behavior is related to the underlying architecture. Unlike UNITE-based metrics, COMET does not rely on soft alignments via attention between the sentences in the encoding process. This process may be key to identify local misalignments during the encoding process. In fact, the attention-based attributions for UNITE metrics can more easily identify these errors. COMET, however, encodes the sentences separately, which may result in grammatical features (e.g. numbers) being encoded similarly across sentences (Chi et al., 2020; Chang et al., 2022). As such, explanations obtained via embedding alignments will not properly identify these misalignments on similar features. Importantly, these findings align with observations made in (Amrhein and Sennrich, 2022; Raunak et al., 2022). This showcases how explanations can be used to diagnose and reveal shortcomings of neural-based metrics.

heading: 5 Conclusions and Future Work
text: In this paper, we investigated the use of explainability methods to better understand widely used neural metrics for machine translation evaluation, such as COMET and UNITE. Concretely, we analyzed how explanations are impacted by the reference information, and how they can be used to reveal weaknesses of these metrics. Our analysis shows that the quality of the explanations is tightly related to the underlying metric architecture. Interestingly, we also provide evidence that neural metrics like COMET may heavily rely on reference information over source information. Additionally, we show that explanations can be used to reveal reference-based metrics weaknesses such as failing to severely penalize localized critical errors. This opens up promising opportunities for future
research on leveraging explanations to diagnose reference-based metrics errors. To support these studies, we call for future datasets illustrating critical errors (e.g., challenge sets (Karpinska et al., 2022)) to be accompanied by annotated error spans.
Limitations
We highlight three main limitations of our work. First, although we have explored gradient-based explanations that take the whole network into consideration and have been shown to be faithful in previous work (Bastings et al., 2021), we do not explicitly explore how COMET combines the sentence representations in the feed-forward that precedes the encoder model to produce the sentence-level score.
Second, we have shown that combining attention with gradient information results in the best explanations for UNITE-based metrics. However, from a practical standpoint, running inference and extracting the explainability scores simultaneously may be more computationally expensive than other techniques: gradient-based metrics benefit from GPU infrastructure and require storing all gradient information.
Third, we have not explored extracting explanations in low-resource settings. That is because high-quality MQM annotations for such language pairs are not yet available. Nevertheless, further research in those settings is needed to access the broader validity of our claims.

heading: Acknowledgements
text: This work was partially supported by the P2020 programs (MAIA, contract 045909), the Portuguese Recovery and Resilience Plan (PRR) through project C645008882-00000055, Center for Responsible AI, by the European Research Council (ERC StG DeepSPIN, 758969), by EU’s Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), and by the Fundação para a Ciência e Tecnologia (contracts UIDB/50021/2020 and UIDB/50008/2020).

heading: A Model Details
text: In Section 2.1, we employed the latest publicly available model (wmt22-comet-da) for COMET, which emerged as a top-performing metric in the WMT 2022 Metrics task (Freitag et al., 2022). To ensure a comparable setting for UNITE (Wan et al., 2022), we trained our own model. In doing so, we utilized the same data employed in the development of the COMET model by (Rei et al., 2022a), without pretraining any synthetic data, as originally suggested. Additionally, our implementation did not incorporate monotonic regional attention, as our preliminary experiments revealed no discernible benefits from its usage. The hyperparameters used are summarized in Table 3, while Table 4 presents the number of Direct Assessments utilized during training. Furthermore, Table 5 displays the segment-level correlations with WMT 2021 MQM data for the News and TED domains.
Regarding computational infrastructure, a single NVIDIA A10G GPU with 23GB memory was used. The resulting UNITE model has 565M parameters while COMET has 581M parameters.
A.1 Output Distribution To better understand the output of the models and what scores are deemed low, we plotted the output distributions for the two models we used in our study. The average score for English→German data is 0.856 for the COMET model and 0.870 for the UNITE model we trained. From Figure 3 we can observe the distribution of scores. This means that the 0.6692 score from the example in Figure 1 corresponds to a low quality output (5th percentile).
A.2 SMAUG Corpus As we have seen in Section 4.2, we have created synthetic translation errors for the following pathologies: negation errors, hallucinations via insertions, named entity errors, and errors in numbers. Table 7 presents a summary of the examples created using SMAUG and in Table 8 we show examples of each error category.

heading: B Comparison between COMET and XLM-R Alignments
text: From Table 1, it is evident that the alignments between the reference and/or source and the translation yield effective explanations for COMET. This raises the question of how these alignments compare to the underlying encoder of COMET before
the fine-tuning process with human annotations. To investigate this, we examine the results for XLM-R without any fine-tuning, as presented in Table 2.
Overall, the explanations derived from the alignments of COMET prove to be more predictive of error spans than those obtained from XLM-R alignments. This suggests that during the fine-tuning phase, COMET models modify the underlying XLM-R representations to achieve better alignment with translation errors.

heading: C Examples
text: In Tables 9 and 10, we show examples of COMET explanations for Chinese→English and English→German language pairs, respectively. We highlight in gray the corresponding MQM annotation performed by an expert linguist and we sort the examples from highest to lowest COMET scores. From these examples we can observe the following:
• Highlights provided by COMET explanations have a high recall with human annotations. In all examples, subword tokens corresponding to translation errors are highlighted in red but we often see that not everything is incorrect.
• Explanations are consistent with scores. For example, in the third example from Table 10, the red highlights do not correspond to errors and in fact the translation only has a major error griffen . Nonetheless, the score assigned by COMET is a low score of 0.68 which is faithful to the explanations that was given even if the assessment does not agree with human experts.
explanations with the MQM annotation ( highlighted in gray ) reveals the source of correspondence between specific token-level translation errors and the resulting scores.
explanations with the MQM annotation ( highlighted in gray ) reveals the source of correspondence between specific token-level translation errors and the resulting scores.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Yes. Section 6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Assistance purely with the language of the paper along every section. Grammarly and DeepL write
B 3 Did you use or create scientific artifacts? Section 3 explains the methods we used. We will release the adaptations required to use the explainability methods over COMET framework, the UniTE model we trained, and all data synthetically-generated data.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 2

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: footnote on the first page. The License will be Apache 2.0
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? in the Appendix we have several statistics for training and testing data.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments?

heading: Appendix provides detailed information about the trained model.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix provides detailed information about the trained model including GPU infrastructure and total number of parameters.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix has all information needed about test data and performance of the models.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 2 and Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: This opens up promising opportunities for future
research on leveraging explanations to diagnose reference-based metrics errors. To support these studies, we call for future datasets illustrating critical errors (e. g. , challenge sets (Karpinska et al. , 2022)) to be accompanied by annotated error spans. Limitations
We highlight three main limitations of our work. First, although we have explored gradient-based explanations that take the whole network into consideration and have been shown to be faithful in previous work (Bastings et al. , 2021), we do not explicitly explore how COMET combines the sentence representations in the feed-forward that precedes the encoder model to produce the sentence-level score. Second, we have shown that combining attention with gradient information results in the best explanations for UNITE-based metrics. However, from a practical standpoint, running inference and extracting the explainability scores simultaneously may be more computationally expensive than other techniques: gradient-based metrics benefit from GPU infrastructure and require storing all gradient information. Third, we have not explored extracting explanations in low-resource settings. That is because high-quality MQM annotations for such language pairs are not yet available. Nevertheless, further research in those settings is needed to access the broader validity of our claims