Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains (2024)

Abstract: We introduce a new, extensive multidimensional quality metrics (MQM) annotated dataset covering 11 language pairs in the biomedical domain. We use this dataset to investigate whether machine translation (MT) metrics which are fine-tuned on humangenerated MT quality judgements are robust to domain shifts between training and inference. We find that fine-tuned metrics exhibit a substantial performance drop in the unseen domain scenario relative to both metrics that rely on the surface form and pre-trained metrics that are not fine-tuned on MT quality judgments.

heading: 1 Introduction
text: Automatic metrics are vital for machine translation (MT) research: given the cost and effort required for manual evaluation, automatic metrics are useful for model development and reproducible comparison between research papers (Ma et al., 2019). In recent years, the MT field has been moving away from string-matching metrics like BLEU (Papineni et al., 2002) towards fine-tuned metrics like COMET (Rei et al., 2020), which start with pre-trained models and then fine-tune them on human-generated quality judgments. Fine-tuned metrics have been the best performers in recent WMT metrics shared task evaluations (Freitag et al., 2022, 2023) and are recommended by the shared task organizers, who go so far as to say, “Neural fine-tuned metrics are not only better, but also robust to different domains.” (Freitag et al., 2022).
Given the growing popularity of fine-tuned metrics, it is important to better understand their behavior. Here, we examine the question of domain robustness of fine-tuned metrics. Fine-tuned metrics contain extra parameters on top of the pretrained model which are initialized randomly (or to zero) and then fine-tuned on human-generated MT
∗Work done during an internship at Amazon. †Corresponding author
Fine-tuned metrics have lower correlation on biomedical domain than WMT … despite other metrics having higher correlation on the biomedical domain
quality annotations. The primary source of those annotations is prior WMT metrics shared tasks, and domains in WMT are often carried over from year to year (e.g. news). This raises the question: are fine-tuned metrics in fact robust across any domain (including domains not seen in training)? Or can their apparent strong performance be attributed in part to the artificially good domain match between training and test data?
To answer these questions, we first collect human multidimensional quality metrics (MQM) annotations in the biomedical (bio) domain. Vocabulary overlap and error analysis suggest that this new dataset is distinct from the domains used in WMT. This data covers 11 language pairs and 21 translation systems, with 25k total judgments. In addition to the MQM annotations, we also create new high-quality reference translations for all directions. We release this data publicly, along with code for replication of our experiments.1
Next, we examine how different types of metrics perform on our new bio test set relative to the WMT test set. We find that fine-tuned metrics have substantially lower correlation with human
1github.com/amazon-science/bio-mqm-dataset
488
components have trainable parameters while use handcrafted heuristics or algorithms and decodes from a language model. The ref input is omitted in the case of reference-free metrics (i.e. quality estimation).
judgments in the bio domain, despite other types of metrics having higher correlation in the bio domain (see Figure 1), indicating they struggle with the training/inference domain mismatch. Finally, we present analysis showing that this performance gap persists throughout different stages of the finetuning process and is not the result of a deficiency with the pre-trained model.

heading: 2 Related Work
text: Metric types. Table 1 summarizes the different types of metrics that are commonly used to evaluate MT. The earliest type of MT metrics are SurfaceForm metrics, which are purely heuristic and use word- or character-based features. We consider three common Surface-Form metrics: BLEU (Papineni et al., 2002), TER (Snover et al., 2006) and CHRF (Popović, 2015). Metrics like COMET (Rei et al., 2020), BLEURT (Sellam et al., 2020), and UNITE (Wan et al., 2022) start with a pretrained language model and fine-tune it on humangenerated MT quality judgments. We denote these metrics Pre-trained+Fine-tuned.2 Another class of metrics also start with a pre-trained model but do not perform fine-tuning. Examples of such metrics include PRISM (Thompson and Post, 2020a,b), which uses the perplexity of a neural paraphraser, and BERTSCORE (Sun et al., 2022), which is based on cosine similarity of word embeddings. We denote such metrics Pre-trained+Algorithm metrics. More recently, metrics like GEMBA
2The WMT metrics task calls these “trained” metrics.
(Kocmi and Federmann, 2023) and AUTOMQM (Fernandes et al., 2023) have proposed prompting a large language model. We denote these as Pretrained+Prompt metrics.
Domain specificity. Domain specificity for MT metrics was first explored by C. de Souza et al. (2014) for Surface-Form metrics. Sharami et al. (2023) brought attention to the issue of domain adaptation for quality estimation (QE), offering solutions based on curriculum learning and generating synthetic scores similar to Heo et al. (2021), Baek et al. (2020), and Zouhar et al. (2023). Sun et al. (2022) examined general-purpose natural language generation metrics and documented their bias with respect to social fairness. For word-level QE, Sharami et al. (2023) reported the lack of robustness of neural metrics.

heading: 3 New Bio MQM Dataset
text: We create and release new translations and MQM annotations for the system submissions from 21 participants to the WMT21 biomedical translation shared task (Yeganova et al., 2021). To explore how different the bio domain is from the WMT22 metric task domains, we computed the vocabulary overlap coefficient between each domain. Bio had the smallest average overlap with the WMT domains (0.436) compared to 0.507, 0.486, 0.507, and 0.582 for e-commerce, news, social, and conversation, respectively. See Appendix A for full details and example sentences from each domain.

heading: 3.1 Dataset Creation
text: We created the bio MQM dataset in three steps. Annotations and translations were performed by expert linguists with experience in the medical domain (see Appendix C for full details).
Step 1: Reference re-translation. The original bio test set consists of bilingual abstracts from crawled academic papers, which might be written by non-native speakers (Névéol et al., 2020) or even MT (Thompson et al., 2024). Therefore, we create new professional reference translations.
Step 2: Reference quality. To ensure a high bar of quality for the reference translations, we ask a separate set of annotators to provide MQM annotations for the new references. Any issues identified by this round of MQM annotation are then fixed by a new set of translators, resulting in the final reference translations that we release in this dataset.
Step 3: MQM annotations. Finally, we conduct the main MQM annotation on the references and shared task system outputs. In this step, a single annotator rates all translations of a given document (from all systems and the reference).3 Our MQM schema follows Freitag et al. (2021) except that we add a Critical severity (assigned the same score as Major for backward compatibility). Full annotator instructions are in Appendix D.
The resulting dataset contains roughly 25k segment-level annotations spanning 11 translation directions.4 In contrast, most publicly available MQM data to date covers only a few language pairs.
3This allows us to distribute annotation jobs to multiple annotators while still allowing the annotator to access documentlevel context and ensuring that the whole document is ranked consistently.
4Pt→En, En↔De, En↔Es, En↔Ru, En↔Fr, Zh↔En
We use ~25% of the segments for each language pair as the train/dev set, leaving the rest as the test set (see Appendix B for exact sizes in each pair).
We compare error distributions on our new bio MQM dataset and the existing WMT MQM dataset in Table 2. Bio MQM contains more Critical/Major errors, and lower absolute scores on average. However, WMT MQM has more overall sentences where an error occurs. Error category distribution also diverges, notably in Fluency and Accuracy.

heading: 4 Analysis

heading: 4.1 Are fine-tuned metrics robust across domains?
text: Measuring domain robustness. The performance of a MT metric is typically measured by a certain meta-evaluation metric, such as segmentlevel Kendall’s τ correlation with human judgments. Intuitively, one could simply measure domain robustness by comparing the performance of a certain metric on domain A and domain B. This, however, is not straightforward with metaevaluations for metrics, since performance measured by those meta-evaluations is also affected by factors such as the quantity and quality of the translations included in the dataset, which is often hard to control for.
As a result, we resort to comparisons of relative performance measured against a domain-invariant baseline. To establish such comparison, we make two assumptions:
1. We assume Surface-Form metrics can serve as a domain-invariant baseline, as they are purely based on heuristics and do not involve parameters specifically tuned on a certain domain. We use average performance of BLEU,
CHRF, and TER as the baseline to minimize the impact of specific choice of heuristics.
2. We assume segment-level Kendall’s τ correlation with human judgments has a linear relationship with the objective performance of a metric. Hence, relative performance can be measured by simple linear subtraction.
Observations. Compared to Surface-Form metrics, we find that Pre-trained+Fine-tuned metrics provide a substantially smaller (sometimes even negative) improvement in human correlation in the bio domain than the WMT domain (see Figure 2). On the other hand, Pre-trained+Algorithm metrics, which have not been trained on WMT data, do not exhibit the same gap. This gap suggests that fine-tuned metrics struggle with unseen domains.
We also observe a very large performance gap for Pre-trained+Prompt metrics. Unfortunately, these metrics rely on closed-source LLMs without published training procedures, so we do not know what data the underlying LLMs were trained on.

heading: 4.2 How does fine-tuning affect domain robustness?
text: Model description. For this section, we focus on COMET (reference-based) and COMET-QE (reference-free) as they are among the most commonly used MT metrics. The COMET model works by representing the source, the hypothesis and the reference as three fixed-width vectors using a language model, such as XLM-Roberta-large (Conneau et al., 2019). These vectors and their combinations serve as an input to a simple feed-forward regressor which is fine-tuned to minimize the MSE loss with human MQM scores. A COMET model is trained in two stages, first on direct assessment (DA) quality annotations and then on MQM annotations, both from WMT shared tasks.
Setup. We limit our experiments to the En-De, Zh-En and Ru-En language directions because of WMT MQM availability. We largely followed the training recipe in the COMET Github repo5. For details, please refer to our code.
There is high inter-annotator variance in the WMT and bio MQM data. Training on the raw MQM scores is very unstable and therefore perannotator z-normalizing is necessary to replicate our setup. Note that the publicly available WMT MQM data are not z-normalized.
5github.com/Unbabel/COMET/tree/master/configs
Observation 1: Domain gap persists throughout the fine-tuning process. We would like to understand which stage among the two training stages for COMET accounts for the domain gap. To this end, we retrained COMET with varying epochs on DA/MQM data, shown in Table 3. In contrast to catastrophic forgetting (Goodfellow et al., 2013; Thompson et al., 2019a,b), where a model starts with good general-domain performance and then overfits while being adapted to a new task or domain, we do not see a sharp dropoff in the bio domain performance when training on more WMT (DA and/or MQM) data. This indicates that the model is a weak bio metric at all stages, as opposed to first learning and then forgetting.
Observation 2: In-domain data dramatically improves COMET. Generally, including bio MQM annotations in training improves COMET’s performance in the bio test set, increasing correlation
from 0.287 to 0.328 with 6k bio judgments. Indeed, just 1k judgements improves correlation to 0.313 (see Figure 3). This rules out the possibility that bio is inherently problematic for COMET’s architecture or fine-tuning strategy.

heading: 4.3 How does the pre-trained model affect domain robustness?
text: COMET and BERTSCORE are both based on XLMRoberta-large (Conneau et al., 2019), allowing us to explore how the same changes to the pre-trained model affect each metric. To see whether improving the underlying pre-trained model improves Pre-trained+Algorithm metrics built on those pretrained models, we fine-tune XLM-Roberta with data similar to the WMT and bio domain setup, respectively. Similarly, we also investigate how PRISM, another Pre-trained+Algorithm metric, is affected with changes to the pre-trained model. We use PRISM with the NLLB multilingual MT models (NLLB Team et al., 2022) as they are larger and more recent than the model released with PRISM.
Setup. Our fine-tuning data covers the four languages of interest, namely English, German, Russian, and Chinese (see Appendix E.2 for a detailed data list). Since NLLB is a translation model, we use only parallel data to fine-tune the model. For the XLM-Roberta case, note that it was finetuned with two objectives: masked language model (MLM) and translation language model (TLM). We use both parallel and monolingual data for MLM training and parallel data for TLM training.
39 44 46 47 48
0.21
0.23
K en
d a ll ’s τ
0.218
0.231
FT :B
io
60 0M
0.213
0.224
F T :B io 1. 3B
0.213 3.3B
Test:Bio
BLEU
Figure 5: Multiple NLLB MT models are used as the base model for PRISMSRC. Fine-tuning the underlying MT model improves the metric. Compute constraints preclude finetuning NLLB-3.3B.
Observations: XLM-Roberta. For both domains, improving the pre-trained model improves BERTSCORE but not COMET (see Figure 4). This indicates that the limiting factor for the poor performance of COMET on bio is the effect from its various fine-tuning stages (discussed in Section 4.2), not an underlying weakness in the pre-trained model on bio.
Observations: NLLB. Our findings are shown in Figure 5. In general, we found that improving the pre-trained models performance (as measured by BLEU on a held out test set) also improved PRISM’s performance.

heading: 5 Conclusion and Future Work
text: This paper investigated the performance of machine translation metrics across divergent domains. To this end, we introduced a new, extensive MQMannotated dataset covering 11 language pairs in the bio domain. Our analysis showed that Pretrained+Fine-tuned metrics (i.e. those that use prior human quality annotations of MT output) exhibit a larger gap between in-domain and out-ofdomain performance than Pre-trained+Algorithm metrics (like BERTSCORE). Further experiments showed that this gap can be attributed to the DA and MQM fine-tuning stage.
Despite the gap between in-domain and out-ofdomain performance, COMET is still the best performing metric on the bio domain in absolute terms. Thus, our findings suggest potential directions for future work including collecting more diverse human judgments for Pre-trained+Fine-tuned metrics and exploring ways to improve the generalization of such metrics during fine-tuning.
Limitations
Our findings are dependent on two empirical assumptions we discussed in section 4.1. To the best of our knowledge, those assumptions are necessary to achieve a fair comparison of metrics across domains, but conclusions may change if our assumptions are refuted in future studies.
We draw conclusions based on a single unseen domain (biomedical). While additional domains would have been preferable, data collection was cost prohibitive.
Context has been shown to be beneficial in machine translation evaluation (Läubli et al., 2018; Toral, 2020) and some metrics used in this work have document-level versions (Vernikos et al., 2022). However, in order to draw fair comparisons with existing metrics which do not yet have a document-level version, we only evaluated metrics at the sentence level.
We focused on segment-level evaluation and did not attempt system-level comparisons because of the limited number of system submissions to the WMT biomedical translation shared task.

heading: Acknowledgements
text: We would like to thank Georgiana Dinu, Marcello Federico, Prashant Mathur, Stefano Soatto, and other colleagues for their feedback at different stages of drafting.
Ethical Considerations
Our human annotations were conducted through a vendor. Annotators were compensated in accordance to the industry standard – specifically, in the range of $27.50 to $37.50 on an hourly basis, depending on the experience of the annotator.

heading: A Domain Overlap Between WMT and bio
text: To evaluate the overlap between the WMT and bio domains, we calculate the vocabulary overlap coefficient ( |A∩B|min(|A|,|B|)) between our new bio MQM dataset and the domains used in the WMT22 metrics shared task. The per-domain overlap matrix is shown in Figure 6. Randomly selected sentences from each domain are provided for illustration in Figure 7.

heading: B Corpus Statistics
text: Table 4 shows the size per language pair of our bio MQM dataset, as well as the WMT MQM dataset for comparison. The bio MQM dataset contains roughly 25k annotated segments, covering 11 language pairs. We split the data into test (roughly 75%) and development (roughly 25%) sets.

heading: C Translator/Annotator Qualifications
text: There were 2-4 MQM annotators for each language pair, and a total of 46 annotators. All linguists had experience in translating/post-editing/reviewing content in the bio domain. This was the main requirement to be able to work on the project. The other qualification criteria for this project were in line with the ISO standard 17100. In particular, the linguists met one or more of the following criteria: (1) A recognized higher education degree in translation; (2) Equivalent third-level degree in another subject plus a minimum of two years of doc-
umented professional translation experience; (3) A minimum of five years of documented professional translation experience; (4) Native speaker of the target language. Although linguists were experts in the bio domain, not all of them were experts in MQM annotation. For this reason, the annotators completed an MQM quiz before onboarding them to ensure they understood the guidelines and requirements.
For the translation and post-editing tasks, we used a two step process (initial post editor + reviewer). In each case the reviewer was a linguist with experience translating medical texts. There were no specific educational or vocational stipulations on that medical qualification, however they were asked to provide a medical-text-specific translation test for us to be onboarded for the project. The initial post-editor in each case was a linguistic expert, but not specifically an expert in medical translations, which is why we followed up with reviewers to ensure contents were translated accurately. Linguists had to demonstrate the following to onboard to the project: (1) At least 3+ years of professional translation experience (2) Proven proficiency in English writing skills (3) In-depth understanding and exposure to the language (4) Strong ability in translating, reviewing, adjusting, and providing adaptation for various writing styles of particular requests.

heading: D MQM Annotation Guidelines
text: Below, we reproduce the MQM annotation guidelines that we provided to the annotators.
Overview: You are asked to evaluate the translations using the guidelines below, and assign error categories and severities considering the context segments available.
Task:
1. Please identify all errors within each translated segment, up to a maximum of five.
(a) If there are more than five errors, identify only the five most severe.
(b) If it is not possible to reliably identify distinct errors because the translation is too badly garbled or is unrelated to the source, then mark a single Unintelligible error that spans the entire segment
(c) Annotate segments in natural order, as if you were reading the document. You
may return to revise previous segments.
2. To identify an error, highlight the relevant span of text.
(a) Omission and Source error should be tagged in the source text.
i. All other errors should be tagged in the target text.
(b) Unintelligible error should have an entire sentence tagged; if you think a smaller span is needed, then you should select another error category (Mistranslation, etc.).
3. Select a category/sub-category and severity level from the available options.
4. When identifying errors, please be as finegrained as possible.
(a) If a sentence contains more than one error of the same category, each one should be logged separately. For example, if a sentence contains two words that are each mistranslated, two separate mistranslation errors should be recorded.
(b) If a single stretch of text contains multiple errors, you only need to indicate the one that is most severe.
i. If all have the same severity, choose the first matching category listed in the error typology (e.g. Accuracy, then Fluency, then Terminology, etc.).
(c) For repetitive errors that appear systematically through the document: please annotate each instance with the appropriate weight.
5. Please pay particular attention to the context when annotating. You will be shown several context segments before and after the segment for evaluation. If a translation is questionable on its own but is fine in the context of the document, it should not be considered erroneous; conversely, if a translation might be acceptable in some context, but not within the current document, it should be marked as wrong.
Delivery format:
• file format: a TSV with additional columns for error categories and severity + JSON
– for multiple errors in one segment: additional row for each error + severity
– text spans will be highlighted for the annotation process and exported as tag
Error categories: Table 5
Severity (no weights, just severity): Table 6

heading: E Supplementary Information on Experiments
text: E.1 Training Steps and Compute Time for Experiments
The overall training consists of the following steps (compute times using a single A10 GPU). The times are per epoch and some experiments require training for multiple epochs.
• Language modeling → XLM-Roberta, 10hr/ep. • DA scores regression → COMETDA, 10hr/ep. • MQM scores regression → COMET, 1hr/ep.
E.2 List of Data for Fine-Tuning Pre-Trained Model
For WMT domain, we used news-commentary v18.1 dataset6 for all languages. For the bio domain, we list the data in Table 7.

heading: F Raw Scores for Figure 2
text: The segment-level correlation (Kendall’s τ ) scores used to compute improvements in Figure 2 are provided in Table 8. Note that there is no public COMET 22 MQM model.
6data.statmt.org/news-commentary/v18.1/ 7www.openagrar.de/receive/openagrar_mods_
00046540?lang=en 8huggingface.co/datasets/shibing624/medical

heading: Future Work
text: Thus, our findings suggest potential directions for future work including collecting more diverse human judgments for Pre-trained+Fine-tuned metrics and exploring ways to improve the generalization of such metrics during fine-tuning. Limitations
Our findings are dependent on two empirical assumptions we discussed in section 4. 1