Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress
Authors: Lorenzo Proietti, Stefano Perrella, Roberto Navigli
Accepted at: acl25

Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress

Lorenzo Proietti*    Stefano Perrella*    Roberto Navigli
Sapienza NLP Group, Sapienza University of Rome
{lproietti, perrella, navigli}@diag.uniroma1.it

Abstract
In Machine Translation (MT) evaluation, metric performance is assessed based on agreement with human judgments. In recent years, automatic metrics have demonstrated increasingly high levels of agreement with humans. To gain a clearer understanding of metric performance and establish an upper bound, we incorporate human baselines in the MT meta-evaluation, that is, the assessment of MT metrics’ capabilities. Our results show that human annotators are not consistently superior to automatic metrics, with state-of-the-art metrics often ranking on par with or higher than human baselines. Despite these findings suggesting human parity, we discuss several reasons for caution. Finally, we explore the broader implications of our results for the research field, asking: Can we still reliably measure improvements in MT evaluation? With this work, we aim to shed light on the limits of our ability to measure progress in the field, fostering discussion on an issue that we believe is crucial to the entire MT evaluation community.

1 Introduction and Related Work
Machine Translation (MT) evaluation is the task of assessing the quality of translated text, while MT meta-evaluation estimates the capabilities of automatic evaluation techniques, i.e., MT metrics. Historically, automatic metrics have been employed due to their low cost and fast experimentation time, whereas human evaluation is still considered the gold standard, necessary for validating automatically-derived findings. However, in recent years the MT evaluation field has seen significant advancements. Neural-based metrics have demonstrated strong correlations with human judgments, largely replacing traditional heuristic-based metrics, and becoming the de facto standard in MT evaluation (Freitag et al., 2022, 2023, 2024). More recently, LLM-based approaches to MT evaluation have emerged (Kocmi and Federmann, 2023b,a; Fernandes et al., 2023; Bavaresco et al., 2024), offering not only high correlation with human judgments but also improved interpretability. This raises the question of what is still missing in order for automatic techniques to achieve human parity, if they have not already. Indeed, unlike other Natural Language Processing tasks, MT evaluation lacks a human performance reference, making it difficult to gauge the true capabilities of MT metrics. For instance, in MT, human performance is measured by evaluating human references alongside system translations (Läubli et al., 2018; Kocmi et al., 2023, 2024a). Similarly, popular benchmarks such as HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and MT-bench (Zheng et al., 2023) report the performance of human baselines.

Since MT metrics’ performance is measured based on agreement with human annotators, we posit that agreement among the annotators themselves can serve as a reference for human performance. Previous studies have reported the Inter-annotator Agreement (IAA) in MT evaluation: Lommel et al. (2014b) used Cohen’s kappa to measure the pairwise agreement between raters; Freitag et al. (2021a) grouped raters’ assessments into seven score bins before calculating pairwise agreement; and Kocmi et al. (2024b) used Kendall correlation coefficient \( \tau_c \). However, these studies employed different measures, making direct comparisons difficult, and none contextualized IAA in relation to the performance of automatic metrics. To the best of our knowledge, Perrella et al. (2024a) were the first to assess metric and human performance jointly. Specifically, they evaluated automatic metrics and human annotators within their new evaluation framework. Nonetheless, since comparing humans and metrics was not their primary focus, they included only one human anno-
<table>
  <tr>
    <th></th>
    <th colspan="2">2020</th>
    <th colspan="2">2022</th>
    <th colspan="2">2023</th>
    <th colspan="2">2024</th>
  </tr>
  <tr>
    <th></th>
    <th>→ DE</th>
    <th>ZH →</th>
    <th>→ DE</th>
    <th>ZH →</th>
    <th>→ DE</th>
    <th>ZH →</th>
    <th>→ ES</th>
  </tr>
  <tr>
    <td>MQM</td>
    <td>3</td>
    <td>3</td>
    <td>3</td>
    <td>3</td>
    <td>4</td>
    <td>3</td>
    <td>1</td>
  </tr>
  <tr>
    <td>ESA</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>2</td>
    <td>x</td>
    <td>1</td>
  </tr>
  <tr>
    <td>pSQM</td>
    <td>3</td>
    <td>3</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>DA+SQM</td>
    <td>x</td>
    <td>x</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>x</td>
  </tr>
  <tr>
    <td>#Seg</td>
    <td>681</td>
    <td>895</td>
    <td>583</td>
    <td>1065</td>
    <td>145</td>
    <td>687</td>
    <td>449</td>
  </tr>
  <tr>
    <td>#Sys</td>
    <td>9</td>
    <td>9</td>
    <td>10</td>
    <td>13</td>
    <td>12</td>
    <td>15</td>
    <td>12</td>
  </tr>
</table>

Table 1: The four top rows indicate the number of distinct evaluators for each annotation protocol and test set. We list the studies that released these annotations in Appendix B. ‘2020’ refers to the data released by Freitag et al. (2021a), while other years correspond to the test sets from the corresponding WMT editions. The notation → x indicates that the test set contains translations from English to x, whereas x → denotes translations from x to English. The two bottom rows present the number of source segments and automatic translations per source segment present in the intersection of annotations from all human evaluators, restricted to the segments annotated by disjoint sets of raters (\S 2.1).

tation protocol – i.e., Direct Assessment + Scalar Quality Metrics (Kocmi et al., 2022a) – that exhibited very poor performance, likely due to low annotation quality, rendering it ineffective as a human performance reference for MT metrics.

In this work, we address this gap by incorporating human baselines into the metric rankings from various editions of the Metrics Shared Task of the Conference on Machine Translation (WMT). By using meta-evaluation strategies from WMT 2024 we derive a single, comprehensive ranking of MT evaluators – both human and automatic – establishing a human performance reference for MT metrics across several test sets, translation directions, and human annotation protocols, and offering a clearer understanding of the capabilities of current MT evaluation techniques. Then, given that our results suggest that automatic metrics may have reached human parity, we critically examine this claim and discuss its implications for future research in MT evaluation.¹

2 Preliminaries and Experimental Setup

In this section, we describe the human annotations, the annotation protocols, the test sets selected for our work, the meta-evaluation strategies employed, and the automatic metrics included.

²We publish the code to reproduce our results at https://github.com/SapienzaNLP/human-parity-mt-eval.

2.1 The Human Annotations

Each year, WMT conducts manual annotation campaigns to collect human judgments of translation quality. First, each test set t is created by drawing \( N_t \) segments from various sources. Segments may consist of individual sentences or entire paragraphs. Each source segment is then translated into the target language using \( M_t \) MT systems, producing \( N_t \times M_t \) translations per test set t. Finally, human raters assess translation quality (Kocmi et al., 2023, 2024a; Freitag et al., 2023, 2024).

Given the large volume of translations, non-overlapping portions of each test set are typically assigned to different raters. Consequently, the annotated test sets used in this work combine annotations from multiple raters. For simplicity, we use the term evaluator to refer to any entity that produced a set of annotations covering all segments in a test set. An evaluator can be a human rater, an MT metric, an ensemble of MT metrics, or an entity that selects annotations from different raters. For example, in the test set that we dub “2020 EN→DE” (Freitag et al., 2021a), six raters provided a total of three annotations per translation, yielding three distinct evaluators.

However, this setup introduces a problem: Distinct human evaluators may be derived from non-disjoint sets of raters. If the same rater contributes to multiple evaluators, even across non-overlapping segments, it can artificially inflate their agreement and overestimate human baseline performance. To avoid this, we restrict each test set to the largest subset of segments annotated by strictly disjoint sets of raters. Returning to the 2020 EN→DE example, we aim to partition the six raters into three groups, so that the combined annotations from raters within each group form a single evaluator. Yet, two factors prevent such a simple partitioning: i) not all raters annotated every source segment, and ii) the specific rater-to-segment assignment prevents partitioning raters such that the combined annotations from each group cover all segments. Therefore, we restrict our test set to the segments that allow such a partitioning by solving the following optimization problem: *Find the largest subset of segments and a partitioning of raters into three disjoint groups such that each group cumulatively annotated the entire subset of segments.* We apply a similar procedure to each test set with annotations of this form, reporting resulting test set sizes in Table 1. Further details are provided in Appendix A.
<table>
  <tr>
    <th colspan="2">Test set 2020</th>
    <th colspan="4">EN→DE</th>
    <th colspan="4">ZH→EN</th>
  </tr>
  <tr>
    <th>Metric</th>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sub>eq</sub></th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sub>eq</sub></th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr>
    <td>MQM-2020-2</td>
    <td></td>
    <td>1</td>
    <td>96.45</td>
    <td></td>
    <td>1</td>
    <td>58.86</td>
    <td></td>
    <td>1</td>
    <td>88.10</td>
    <td></td>
    <td>1</td>
    <td>55.70</td>
  </tr>
  <tr>
    <td>pSQM-1</td>
    <td></td>
    <td>1</td>
    <td>95.59</td>
    <td></td>
    <td>6</td>
    <td>49.41</td>
    <td></td>
    <td>1</td>
    <td>79.16</td>
    <td></td>
    <td>13</td>
    <td>43.89</td>
  </tr>
  <tr>
    <td>MQM-2020-3</td>
    <td></td>
    <td>2</td>
    <td>90.39</td>
    <td></td>
    <td>2</td>
    <td>56.84</td>
    <td></td>
    <td>1</td>
    <td>92.06</td>
    <td></td>
    <td>2</td>
    <td>52.80</td>
  </tr>
  <tr>
    <td>BLEURT-0.2</td>
    <td></td>
    <td>2</td>
    <td>86.81</td>
    <td></td>
    <td>4</td>
    <td>50.81</td>
    <td></td>
    <td>2</td>
    <td>72.59</td>
    <td></td>
    <td>3</td>
    <td>50.57</td>
  </tr>
  <tr>
    <td>pSQM-2</td>
    <td></td>
    <td>2</td>
    <td>85.87</td>
    <td></td>
    <td>9</td>
    <td>46.97</td>
    <td></td>
    <td>1</td>
    <td>89.33</td>
    <td></td>
    <td>9</td>
    <td>46.77</td>
  </tr>
  <tr>
    <td>BLEURT-20</td>
    <td></td>
    <td>2</td>
    <td>85.52</td>
    <td></td>
    <td>3</td>
    <td>51.68</td>
    <td></td>
    <td>3</td>
    <td>67.46</td>
    <td></td>
    <td>4</td>
    <td>50.12</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">Test set 2022</th>
    <th colspan="4">EN→DE</th>
    <th colspan="4">EN→ZH</th>
  </tr>
  <tr>
    <th>Metric</th>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sub>eq</sub></th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sub>eq</sub></th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr>
    <td>MetricX-23-QE-XXL*</td>
    <td></td>
    <td>1</td>
    <td>94.89</td>
    <td></td>
    <td>3</td>
    <td>57.64</td>
    <td></td>
    <td>2</td>
    <td>83.92</td>
    <td></td>
    <td>2</td>
    <td>47.43</td>
  </tr>
  <tr>
    <td>MQM-2022-2</td>
    <td></td>
    <td>1</td>
    <td>94.49</td>
    <td></td>
    <td>6</td>
    <td>55.55</td>
    <td></td>
    <td>2</td>
    <td>80.82</td>
    <td></td>
    <td>3</td>
    <td>47.05</td>
  </tr>
  <tr>
    <td>MQM-2022-3</td>
    <td></td>
    <td>1</td>
    <td>92.59</td>
    <td></td>
    <td>1</td>
    <td>61.06</td>
    <td></td>
    <td>1</td>
    <td>87.22</td>
    <td></td>
    <td>2</td>
    <td>47.56</td>
  </tr>
  <tr>
    <td>MetricX-23-XXL</td>
    <td></td>
    <td>2</td>
    <td>92.34</td>
    <td></td>
    <td>2</td>
    <td>59.27</td>
    <td></td>
    <td>1</td>
    <td>87.69</td>
    <td></td>
    <td>1</td>
    <td>48.43</td>
  </tr>
  <tr>
    <td>DA+SQM</td>
    <td></td>
    <td>6</td>
    <td>66.61</td>
    <td></td>
    <td>16</td>
    <td>46.03</td>
    <td></td>
    <td>2</td>
    <td>82.95</td>
    <td></td>
    <td>12</td>
    <td>36.26</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">Test set 2023</th>
    <th colspan="4">EN→DE</th>
    <th colspan="4">ZH→EN</th>
  </tr>
  <tr>
    <th>Metric</th>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sub>eq</sub></th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sub>eq</sub></th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr>
    <td>GEMBA-MQM*</td>
    <td></td>
    <td>1</td>
    <td>94.52</td>
    <td></td>
    <td>5</td>
    <td>58.52</td>
    <td></td>
    <td>1</td>
    <td>93.17</td>
    <td></td>
    <td>3</td>
    <td>52.80</td>
  </tr>
  <tr>
    <td>MQM-2023-3</td>
    <td></td>
    <td>1</td>
    <td>93.51</td>
    <td></td>
    <td>5</td>
    <td>58.42</td>
    <td></td>
    <td>1</td>
    <td>95.54</td>
    <td></td>
    <td>5</td>
    <td>51.65</td>
  </tr>
  <tr>
    <td>MQM-2023-2</td>
    <td></td>
    <td>1</td>
    <td>93.15</td>
    <td></td>
    <td>6</td>
    <td>57.71</td>
    <td></td>
    <td>1</td>
    <td>95.18</td>
    <td></td>
    <td>2</td>
    <td>52.90</td>
  </tr>
  <tr>
    <td>XCOMET-Ensemble</td>
    <td></td>
    <td>1</td>
    <td>92.21</td>
    <td></td>
    <td>3</td>
    <td>60.99</td>
    <td></td>
    <td>2</td>
    <td>91.15</td>
    <td></td>
    <td>1</td>
    <td>54.59</td>
  </tr>
  <tr>
    <td>MetricX-23-QE-XXL*</td>
    <td></td>
    <td>1</td>
    <td>92.12</td>
    <td></td>
    <td>1</td>
    <td>62.53</td>
    <td></td>
    <td>3</td>
    <td>88.30</td>
    <td></td>
    <td>2</td>
    <td>53.26</td>
  </tr>
  <tr>
    <td>DA+SQM</td>
    <td></td>
    <td>2</td>
    <td>91.24</td>
    <td></td>
    <td>14</td>
    <td>46.79</td>
    <td></td>
    <td>4</td>
    <td>86.28</td>
    <td></td>
    <td>22</td>
    <td>39.42</td>
  </tr>
  <tr>
    <td>ESA-1</td>
    <td></td>
    <td>2</td>
    <td>90.39</td>
    <td></td>
    <td>14</td>
    <td>46.71</td>
    <td></td>
    <td>–</td>
    <td>–</td>
    <td></td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>ESA-2</td>
    <td></td>
    <td>2</td>
    <td>89.11</td>
    <td></td>
    <td>12</td>
    <td>49.70</td>
    <td></td>
    <td>–</td>
    <td>–</td>
    <td></td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>MQM-2023-4</td>
    <td></td>
    <td>2</td>
    <td>88.93</td>
    <td></td>
    <td>14</td>
    <td>46.68</td>
    <td></td>
    <td>–</td>
    <td>–</td>
    <td></td>
    <td>–</td>
    <td>–</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">Test set 2024</th>
    <th colspan="4">EN→ES</th>
  </tr>
  <tr>
    <th>Metric</th>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sub>eq</sub></th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr>
    <td>CometKiwi-XXL*</td>
    <td></td>
    <td>1</td>
    <td>86.12</td>
    <td></td>
    <td>4</td>
    <td>67.24</td>
  </tr>
  <tr>
    <td>gemma_esa*</td>
    <td></td>
    <td>1</td>
    <td>85.72</td>
    <td></td>
    <td>3</td>
    <td>67.68</td>
  </tr>
  <tr>
    <td><b>ESA</b></td>
    <td></td>
    <td>2</td>
    <td>80.12</td>
    <td></td>
    <td>8</td>
    <td>63.84</td>
  </tr>
  <tr>
    <td>metametrics_mt_mqm_hybrid_kendall</td>
    <td></td>
    <td>2</td>
    <td>80.10</td>
    <td></td>
    <td>1</td>
    <td>68.95</td>
  </tr>
  <tr>
    <td>MetricX-24-Hybrid</td>
    <td></td>
    <td>2</td>
    <td>79.75</td>
    <td></td>
    <td>1</td>
    <td>69.20</td>
  </tr>
</table>

Table 2: Results obtained by applying the WMT 2024 Meta-Evaluation strategies to the test sets illustrated in Section 2.2. The ‘Acc.’ column contains the Meta-Evaluation accuracy, while ‘Rank’ reports clusters of statistical significance computed following Freitag et al. (2024), using the PERM-BOTH hypothesis test introduced by Deutsch et al. (2021). Starred metrics are reference-less metrics, and rows highlighted in gray are human evaluators.

2.2 Test Sets and Annotation Protocols

We estimate human performance based on the agreement among human evaluators. Specifically, we designate one human evaluator as ground truth while the others serve as human baselines. Consequently, our setup necessitates multiple human annotations for the same translations. Test sets satisfying this requirement include those released by Freitag et al. (2021a) and those from WMT editions between 2022 and 2024.

These test sets feature human annotations from at least two of the following protocols: Multidimensional Quality Metrics (MQM, Lommel et al., 2014a), Error Span Annotation (ESA, Kocmi et al., 2024b), Professional Scalar Quality Metrics (pSQM, Freitag et al., 2021a), and Direct Assess-
ments + Scalar Quality Metrics (DA+SQM, Kocmi et al., 2022a). Our work leverages these test sets, but we restrict them to source segments that simultaneously: i) were annotated by all considered human evaluators and ii) were annotated by disjoint sets of raters (as detailed in Section 2.1). Table 1 presents statistics for the test sets employed. Additionally, we illustrate all the considered annotation protocols in Appendix B.

Following standard practice in the literature (Freitag et al., 2021a,b, 2022, 2023, 2024), we designate evaluators derived from the MQM annotations released annually at WMT as the ground truth, employing the others as human baselines. Indeed, the MQM protocol relies on experienced annotators and provides a more detailed (and more expensive) evaluation compared to other protocols. Nonetheless, in Appendix F, we also investigate the effects of selecting alternative evaluators – either MQM evaluators different from those previously used or evaluators following different protocols – as the ground truth.

2.3 The MT Meta-Evaluation

We compute metric rankings using the meta-evaluation strategies employed at the WMT 2024 Metrics Shared Task:

• Soft Pairwise Accuracy (SPA) estimates evaluator performance based on the ability to rank \( MT \) systems\(^2\) in the same order as in the ranking derived from ground truth annotations (Thompson et al., 2024).

• Pairwise Accuracy with Tie Calibration (\( acc_{eq}^* \)) estimates evaluator performance based on the ability to rank translations of the same source segment in the same order as in the ranking derived from ground truth annotations (Deutsch et al., 2023).

We describe these measures in more detail in Appendix C.

2.4 Metrics

The automatic evaluators considered – i.e., the MT metrics – are those submitted to the WMT Metrics Shared Task in the 2020, 2022, 2023, and 2024 editions. Additionally, we include several state-of-the-art metrics from recent WMT editions in rankings from previous years, provided they were not trained on the corresponding test sets. Table 3 in Appendix D lists all considered metrics.

3 Results

Table 2 presents the evaluator rankings. Due to space constraints, each table includes only a subset of evaluators. A complete set of results, including all the evaluators, is provided in Appendix E.

Results vary across years and translation directions. Notably, human evaluators do not consistently rank higher than automatic metrics. Under SPA, human evaluators often share clusters of statistical significance with automatic metrics, whereas, under \( acc_{eq}^* \), they are frequently surpassed. For example, in 2020 \( \text{EN}\rightarrow\text{DE} \), BLEURT-0.2 and BLEURT-20 fall within the same statistical significance cluster as MQM-2020-3 and pSQM-2 under SPA, with pSQM-2 ranking as low as 9th under \( acc_{eq}^* \). Similarly, in 2022 \( \text{EN}\rightarrow\text{DE} \), MQM-2022-2 and MQM-2022-3 share the top cluster with MetricX-23-QE-XXL under SPA, with MQM-2022-2 ranking 6th under \( acc_{eq}^* \). Finally, in 2023 and 2024, most human evaluators rank consistently below various automatic metrics under both SPA and \( acc_{eq}^* \). Even when restricted to the human evaluators who follow the same protocol as the annotations employed as gold – i.e., MQM – they rank consistently in the top positions solely in 2020. Additionally, our findings remain valid when varying the human evaluators used as ground truth, as shown in Appendix F.

These results may indicate human-level performance in MT evaluation. Nonetheless, we argue that they are insufficient to establish equivalence between human and automatic evaluators, and elaborate our reasons in the next section.

4 Discussion

In the same spirit as Tedeschi et al. (2023), who discuss the meaning of superhuman performance in Natural Language Understanding, we outline several factors to consider before making similar claims in MT evaluation. We then discuss the broader implications of our findings, warning that measuring progress in the field may become increasingly challenging.

Meta-evaluation Certain meta-evaluation measures may be inadequate for comparing human and automatic evaluators. In particular, our results consistently rank human evaluators much lower under
acc*_{eq} than under SPA. This discrepancy may be related to the findings of Perrella et al. (2024b), who show that acc*_{eq} favors evaluators whose assessments fall within a continuous interval, whereas, as detailed in Appendix B, human evaluators produce discrete assessments.

Annotation quality Certain annotation campaigns might have produced low-quality annotations, either due to a lack of clarity in the annotation guidelines or to the ability of the raters involved. This is particularly concerning in the 2023 EN→DE test set, where, even if restricted to SPA, most human evaluators fall within the second cluster of statistical significance, alongside surface-level metrics such as BLEU.\footnote{We wish to highlight that our 2023 test set features only 145 segments annotated by all human evaluators (as reported in Table 1), which might have resulted in unreliable estimates of SPA and acc*_{eq}.}

Benchmarks difficulty Current test sets might be too easy for the MT systems whose translations are being evaluated. Supporting this hypothesis, we observe that sentinel-cand-mqm, a metric that assesses only translation fluency, ranks on par with the human evaluator ESA under SPA, and even higher under acc*_{eq} (Table 7). This suggests that the evaluated translations may differ only in minor fluency-related nuances. Arguably, to assess whether human parity has been truly achieved, future studies should compare metrics and humans in more demanding contexts. Indeed, previous research has shown that metrics may struggle in unseen domains (Zouhar et al., 2024) and lack sensitivity to specific translation errors such as incorrect number, gender (Karpinska et al., 2022), or word sense disambiguation (Martelli et al., 2025).

4.1 Can We Still Measure Improvements in MT Evaluation?

As discussed, we believe claiming human parity is premature without first addressing the issues outlined above. Nonetheless, with automatic metrics ranking the same as, or higher than, human evaluators in standard benchmarks, our results raise a critical concern about our ability to measure progress in MT evaluation: What does a higher or lower ranking truly mean?

If a metric ranks higher than a human evaluator using a non-MQM protocol, is the metric a better evaluator, or does it merely align more closely with the score distribution of the MQM protocol?

More concerning, if a metric ranks higher than an MQM evaluator, does this suggest superior evaluation capabilities, or does it simply reflect better alignment with the specific raters who produced the gold annotations? Indeed, Finkelstein et al. (2024) achieved an exceptionally high agreement with gold annotations by explicitly optimizing their metric to align with the raters themselves. More generally, we argue that in current benchmarks it is unclear whether a higher ranking – relative to either a human or an automatic evaluator – reflects genuine improvements in evaluation quality or merely closer alignment with a particular annotation protocol or rater style.

To ensure the reliability of meta-evaluation, future research should focus on exploring whether the gap between human and automatic evaluators can be restored. This could be pursued in several ways, including (but not limited to) selecting more challenging test sets, using test sets adversarial to MT metrics (e.g., from domains different from their training data), producing higher-quality human annotations, or designing new annotation protocols that yield stronger inter-annotator agreement. Additionally, greater resources could be allocated to human annotation campaigns – either by collecting multiple annotations per translation to reach a consensus among annotators or by increasing the number of segments in test sets, as suggested by Riley et al. (2024).

5 Conclusions

We incorporate human baselines into the metric rankings from previous editions of the WMT Metrics Shared Task. Our results show that MT metrics frequently rank higher than human evaluators, particularly when the latter follow annotation protocols different from MQM – the protocol used as the ground truth. While our findings suggest that metrics may have reached human-level performance, we recommend caution and highlight several issues the research community should address to assess whether human parity has been truly achieved. Finally, we discuss a critical concern arising from our findings: the limits of measuring progress in MT evaluation as automatic metrics approach human baselines. In this respect, we propose research directions to ensure that progress remains measurable or, at the very least, to extend the period during which it can be reliably tracked.
Limitations

This study required test sets annotated by multiple human evaluators. Consequently, our analysis is limited to seven test sets including four language directions.

Moreover, assessing the agreement between various human evaluators required restricting our analysis to segments annotated by all of them. As a result, some test sets contain only a small number of segments, which might reduce the reliability of the results. To mitigate this issue, our findings are supported by statistical significance analyses.

Acknowledgements

We gratefully acknowledge the support of the PNRR MUR project PE0000013-FAIR, and the CREATIVE project (CRoss-modal understanding and gEnerATIon of Vi-sual and tExtual content), which is funded by the MUR Progetti di Rilevante Interesse Nazionale programme (PRIN 2020). The authors acknowledge the CINECA award IsCb9_mttmt under the ISCRA initiative for the availability of high-performance computing resources.

This work was carried out while Lorenzo Proietti was enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome. The authors thank Anna Bavaresco for her valuable feedback on the manuscript.

References

David Anugraha, Garry Kuwanto, Lucky Susanto, Derry Tanti Wijaya, and Genta Winata. 2024. MetaMetrics-MT: Tuning meta-metrics for machine translation via human preference calibration. In Proceedings of the Ninth Conference on Machine Translation, pages 459–469, Miami, Florida, USA. Association for Computational Linguistics.

Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suggia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2024. Lms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. Preprint, arXiv:2406.18403.

Rachel Bawden, Biao Zhang, Andre Táttar, and Matt Post. 2020. ParBLEU: Augmenting metrics with automatic paraphrases for the WMT’20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 887–894, Online. Association for Computational Linguistics.

Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132–1146.

Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12914–12929, Singapore. Association for Computational Linguistics.

Sören Dreano, Derek Molloy, and Noel Murphy. 2023a. Embed_Llama: Using LLM embeddings for the metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 738–745, Singapore. Association for Computational Linguistics.

Sören Dreano, Derek Molloy, and Noel Murphy. 2023b. Tokengram_F, a fast and accurate token-based chrF++ derivative. In Proceedings of the Eighth Conference on Machine Translation, pages 730–737, Singapore. Association for Computational Linguistics.

Muhammad ElNokrashy and Tom Kočmi. 2023. eBLEU: Unexpectedly good machine translation evaluation using simple word embeddings. In Proceedings of the Eighth Conference on Machine Translation, pages 746–750, Singapore. Association for Computational Linguistics.

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Machine Translation, pages 1066–1083, Singapore. Association for Computational Linguistics.

Mara Finkelstein, Dan Deutsch, Parker Riley, Juraj Juraska, Geza Kovacs, and Markus Freitag. 2024. From jack of all trades to master of one: Specializing llm-based autoraters to a test set. Preprint, arXiv:2411.15387.

Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474.

Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian
Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Mariana Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. Are LLMs breaking MT metrics? results of the WMT24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 47–81, Miami, Florida, USA. Association for Computational Linguistics.

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, pages 578–628, Singapore. Association for Computational Linguistics.

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733–774, Online. Association for Computational Linguistics.

Thamme Gowda, Tom Kocmi, and Marcin Junczys-Dowmunt. 2023. Cometoid: Distilling strong reference-based machine translation metrics into Even stronger quality estimation metrics. In Proceedings of the Eighth Conference on Machine Translation, pages 751–755, Singapore. Association for Computational Linguistics.

Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.

Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504, Miami, Florida, USA. Association for Computational Linguistics.

Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 756–767, Singapore. Association for Computational Linguistics.

Marzena Karpinska, Nishant Raj, Katherine Thai, Yixiao Song, Ankita Gupta, and Mohit Iyyer. 2022. DEMETR: Diagnosing evaluation metrics for translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9540–9561, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117–122, Florence, Italy. Association for Computational Linguistics.

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaki Nagata, Martin Popel, Maja Popović, Mariya Shmatova, Steinhörður Steinþórsson, and Vilém Zouhar. 2024a. Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet. In Proceedings of the Ninth Conference on Machine Translation, pages 1–46, Miami, Florida, USA. Association for Computational Linguistics.

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popović, and Mariya Shmatova. 2023. Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In Proceedings of the Eighth Conference on Machine Translation, pages 1–42, Singapore. Association for Computational Linguistics.

Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022a. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1–45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Tom Kocmi and Christian Federmann. 2023a. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768–775, Singapore. Association for Computational Linguistics.

Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation.

Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478–494, Online. Association for Computational Linguistics.

Tom Kocmi, Hitokazu Matsushita, and Christian Federmann. 2022b. MS-COMET: More and better human judgements improve metric performance. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 541–548, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Minmayaa Sachan, and Mariya Shmatova. 2024b. Error span annotation: A balanced approach for human evaluation of machine translation. In Proceedings of the Ninth Conference on Machine Translation, pages 1440–1453, Miami, Florida, USA. Association for Computational Linguistics.

Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791–4796, Brussels, Belgium. Association for Computational Linguistics.

Yilun Liu, Xiaosong Qiao, Zhanglin Wu, Su Chang, Min Zhang, Yanqing Zhao, Song Peng, Shimin Tao, Hao Yang, Ying Qin, Jiaxin Guo, Minghan Wang, Yinglu Li, Peng Li, and Xiaofeng Zhao. 2022. Partial could be better than whole. HW-TSC 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 549–557, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Chi-ku Lo. 2019. YiSi - a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507–513, Florence, Italy. Association for Computational Linguistics.

Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014a. Multidimensional quality metrics (mqm) : A framework for declaring and describing translation quality metrics. Tradumática, pages 455–463.

Arle Richard Lommel, Maja Popovic, and Aljoscha Burchardt. 2014b. Assessing inter-annotator agreement for translation error annotation. In MTE: Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. International Conference on Language Resources and Evaluation (LREC-14), located at LREC 14, May 26-31, Reykjavik, Iceland. LREC.

Federico Martelli, Stefano Perrella, Niccolò Campolungo, Tina Munda, Svetla Koeva, Carole Tiberius, and Roberto Navigli. 2025. Dibimt: A gold evaluation benchmark for studying lexical ambiguity in machine translation. Computational Linguistics, pages 1–71.

Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799–2808, Florence, Italy. Association for Computational Linguistics.

Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics.

Ananya Mukherjee, Hema Ala, Manish Shrivastava, and Dipti Misra Sharma. 2020. Mee : An automatic metric for evaluation using embeddings for machine translation. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pages 292–299.

Ananya Mukherjee and Manish Shrivastava. 2022a. REUSE: REference-free UnSupervised quality estimation metric. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 564–568, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Ananya Mukherjee and Manish Shrivastava. 2022b. Unsupervised embedding-based metric for MT evaluation with improved human correlation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 558–563, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Ananya Mukherjee and Manish Shrivastava. 2023. MEE4 and XLsim : IIIT HYD’s submissions’ for WMT23 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 800–805, Singapore. Association for Computational Linguistics.

Ananya Mukherjee and Manish Shrivastava. 2024. chrF-S: Semantics is all you need. In Proceedings of the Ninth Conference on Machine Translation, pages 470–474, Miami, Florida, USA. Association for Computational Linguistics.
Subhajit Naskar, Daniel Deutsch, and Markus Freitag. 2023. Quality estimation using minimum Bayes risk. In Proceedings of the Eighth Conference on Machine Translation, pages 806–811, Singapore. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Stefano Perrella, Lorenzo Proietti, Pere-Lluís Huguet Cabot, Edoardo Barba, and Roberto Navigli. 2024a. Beyond correlation: Interpretable evaluation of machine translation metrics. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20689–20714, Miami, Florida, USA. Association for Computational Linguistics.

Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Edoardo Barba, and Roberto Navigli. 2024b. Guardians of the machine translation meta-evaluation: Sentinel metrics fall in! In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16216–16244, Bangkok, Thailand. Association for Computational Linguistics.

Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Niccolò Campolongo, and Roberto Navigli. 2022. MaTESe: Machine translation evaluation as a sequence tagging problem. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 569–577, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.

Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030–1040, Online. Association for Computational Linguistics.

Ricardo Rei, Nuno M. Guerreiro, JosÃ© Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André Martins. 2023. Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 841–848, Singapore. Association for Computational Linguistics.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020a. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020b. Unbabel’s participation in the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 911–920, Online. Association for Computational Linguistics.

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Parker Riley, Daniel Deutsch, George Foster, Viresh Ratnakar, Ali Dabirmoghaddam, and Markus Freitag. 2024. Finding replicable human evaluations via stable ranking probability. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4908–4919, Mexico City, Mexico. Association for Computational Linguistics.

Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020a. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.

Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. 2020b. Learning to evaluate translation beyond English: BLEURT submissions to the WMT metrics 2020 shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 921–927, Online. Association for Computational Linguistics.
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.

Peter Stanchev, Weiyue Wang, and Hermann Ney. 2019. EED: Extended edit distance measure for machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 514–520, Florence, Italy. Association for Computational Linguistics.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejía Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Rogers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.

Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajíč, Daniel Hershcovitch, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, and Roberto Navigli. 2023. What’s the meaning of superhuman performance in today’s NLU? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12471–12491, Toronto, Canada. Association for Computational Linguistics.

Brian Thompson, Nitika Mathur, Daniel Deutsch, and Huda Khayrallah. 2024. Improving statistical significance in human evaluation of automatic metrics via soft pairwise accuracy. In Proceedings of the Ninth Conference on Machine Translation, pages 1222–1234, Miami, Florida, USA. Association for Computational Linguistics.

Brian Thompson and Matt Post. 2020a. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 90–121, Online. Association for Computational Linguistics.

Brian Thompson and Matt Post. 2020b. Paraphrase generation as zero-shot multilingual translation: Disentangling semantic similarity from lexical and syntactic diversity. In Proceedings of the Fifth Conference on Machine Translation, pages 561–570, Online. Association for Computational Linguistics.

Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022. Embarrassingly easy document-level MT metrics: How to convert any pretrained metric into a document-level metric. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 118–128, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Vasiliy Viskov, George Kokush, Daniil Larionov, Steffen Eger, and Alexander Panchenko. 2023. Semantically-informed regressive encoder score. In Proceedings of the Eighth Conference on Machine Translation, pages 815–821, Singapore. Association for Computational Linguistics.

Yu Wan, Keqin Bao, Dayiheng Liu, Baosong Yang, Derek F. Wong, Lidia S. Chao, Wenqiang Lei, and Jun Xie. 2022a. Alibaba-translate China’s submission for WMT2022 metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 586–592, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022b. UniTE: Unified translation evaluation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8117–8127, Dublin, Ireland. Association for Computational Linguistics.

Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. CharacTer: Translation edit rate on character level. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 505–510, Berlin, Germany. Association for Computational Linguistics.

Zhanglin Wu, Yilun Liu, Min Zhang, Xiaofeng Zhao, Junhao Zhu, Ming Zhu, Xiaosong Qiao, Jingfei Zhang, Ma Miaoimiao, Zhao Yanqing, Song Peng, Shimin Tao, Han Yang, and Yanfei Jiang. 2023. Empowering a metric with LLM-assisted named entity annotation: HW-TSC’s submission to the WMT23 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 822–828, Singapore. Association for Computational Linguistics.

Jin Xu, Yinuo Guo, and Junfeng Hu. 2020. Incorporate semantic structures into machine translation evaluation via UCCA. In Proceedings of the Fifth Conference on Machine Translation, pages 934–939, Online. Association for Computational Linguistics.

Wenda Xu, Xian Qian, Mingxuan Wang, Lei Li, and William Yang Wang. 2023. SESCORE2: Learning text generation evaluation via synthesizing realistic mistakes. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5166–5183, Toronto, Canada. Association for Computational Linguistics.
Wenda Xu, Yi-Lin Tuan, Yujie Lu, Michael Saxon, Lei Li, and William Yang Wang. 2022. Not all errors are equal: Learning text generation metrics using stratified error synthesis. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6559–6574, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proceedings of the International Conference on Learning Representations.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.

Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatiana Badeka, Jenyuan Wang, and Brian Thompson. 2024. Fine-tuned machine translation metrics struggle in unseen domains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 488–500, Bangkok, Thailand. Association for Computational Linguistics.

A Fair Extraction of Evaluators from Human Annotations

The human evaluation campaigns conducted by Freitag et al. (2021a), Freitag et al. (2023), and Riley et al. (2024) produced multiple annotations for each translation. As you can see in Table 1, there are many annotations per translation for MQM and pSQM in the test sets 2020, 2022, and 2023. As discussed in Section 2.1, these annotation campaigns distributed the annotation workload among multiple raters.

Since we derive multiple evaluators from these annotations (some used as ground truth and some as human baselines), we prevent artificially inflating their agreement by not allowing the same rater to contribute to two distinct evaluators simultaneously. For example, in the 2020 EN→DE test set, six raters provided a total of three annotations per translation. We extract three human evaluators from these annotations, using one as the ground truth and the other two as human evaluators (MQM-2020-2 and MQM-2020-3 in Table 2). To achieve this, we partition the six raters into three groups, each forming one evaluator. However, not all raters annotated the entire set of source segments, and the distribution of workload did not allow for a partition that covered all annotated segments. Therefore, to retain the maximum number of segments in our test sets, we solved the following optimization problem: *Find the largest subset of segments and a partitioning of raters into three disjoint groups such that each group cumulatively annotated the entire subset of segments*.

Formally, let us define a test set \( t = \{s_1, ..., s_{N_t}\} \) as a set of \( N_t \) segments. Each segment was annotated by \( k \) out of \( R \) raters, with \( \mathcal{R} = \{r_1, ..., r_R\} \) representing the set of raters. Our goal is to determine a partition \( \Pi = \{\mathcal{R}_1, ..., \mathcal{R}_k\} \) of \( \mathcal{R} \) and a subset \( u \subseteq t \) such that \( u \) is the largest subset in which every segment has been annotated by exactly one rater from each of the \( k \) sets in the partition \( \Pi \).

To solve this optimization problem, we formulate it as an Integer Linear Programming (ILP) problem and solve it using the PuLP\(^4\) Python library. We applied this procedure to the 2020, 2022, and 2023 test sets.

B Human Annotations

We briefly illustrate how each annotation protocol considered works:

• Multidimensional Quality Metrics (MQM) requires annotators to identify error spans in the translated text, specifying error category and severity, to be selected among Neutral, Minor, Major, and Critical. A translation quality score is derived by assigning a penalty to each error span depending on severity (Lommel et al., 2014a; Freitag et al., 2021a).

• Error Span Annotation (ESA) requires annotators to identify error spans in the translated text, specify error severity, and later assign a scalar quality score from 0 to 100 to the translation (Kocmi et al., 2024b).

• Scalar Quality Metrics (SQM) requires annotators to assign a scalar quality score from 0 to 6 to the translated text. Following (Freitag et al., 2021a), we use ‘pSQM’ to refer to SQM conducted by professional annotators.\(^5\)

\(^4\)https://coin-or.github.io/pulp/.
\(^5\)In this work, we use only annotations produced by professional annotators or translators. Therefore, we exclude cSQM and Direct Assessments (DA) – which were crowdsourced – from the 2020 test sets.
• Direct Assessments + Scalar Quality Metrics (Kocmi et al., 2022a, DA+SQM) requires raters to assign a scalar quality score from 0 to 100 to the translated text. Raters are presented with seven labeled tick marks describing translation quality levels at various score thresholds, similarly to the SQM protocol.

Here, for each set of annotations employed in this work (i.e., those reported in Table 1), we indicate the reference paper that released them:

• The MQM-based and pSQM-based annotations for the test sets 2020 EN→DE and 2020 ZH→EN have been released by Freitag et al. (2021a).

• The MQM-based annotations for the test sets 2022 EN→DE and 2022 EN→ZH have been released by Freitag et al. (2022) and Riley et al. (2024).

• The DA+SQM-based annotations for the test sets 2022 EN→DE and 2022 EN→ZH have been released by Kocmi et al. (2022a).

• Three sets of MQM-based annotations for the test sets 2023 EN→DE and ZH→EN have been released by Freitag et al. (2023).

• The ESA-based annotations and the last set of MQM-based annotations (MQM-2023-4 in Table 2) for the test set 2023 EN→DE have been released by Kocmi et al. (2024b).

• The ESA-based annotations for the test set 2024 EN→ES have been released by Kocmi et al. (2024a).

• The MQM-based annotations for the test set 2024 EN→ES have been released by Freitag et al. (2024).

C Meta-Evaluation Measures

In this section, we describe the two meta-evaluation measures used in our work, as listed in Section 2.3.

C.1 Soft Pairwise Accuracy (SPA)

Thompson et al. (2024) introduced Soft Pairwise Accuracy (SPA) as an extension of Pairwise Accuracy (Kocmi et al., 2021, PA).

Given a test set \( t \), which consists of \( N_t \) source segments and \( M_t \) translations generated by the respective \( M_t \) MT systems (as described in Section 2.1), PA counts how often an evaluator \( e \) ranks system pairs in the same order as the ground truth \( g \). Let \( a_{ij} \) be 1 if evaluator \( e \) ranks systems \( i \) and \( j \) in the same order as the ground truth and 0 otherwise, where \( i, j \in \{0, ..., Mt\} \). Then, PA is defined as:

\[
PA = \left( \frac{N}{2} \right)^{-1} \sum_{i=0}^{M_t} \sum_{j=i+1}^{M_t} a_{ij}
\]

SPA extends PA by incorporating the confidence with which an evaluator and the ground truth rank two MT systems. Confidence is represented using statistical \( p \)-values. Specifically, \( p_{ij}^e \) denotes the \( p \)-value associated with the statistical hypothesis that system \( i \) is better than system \( j \) according to evaluator \( e \), while \( p_{ij}^g \) represents the corresponding \( p \)-value for the ground truth \( g \). SPA is then defined as follows:

\[
SPA = \left( \frac{N}{2} \right)^{-1} \sum_{i=0}^{M_t} \sum_{j=i+1}^{M_t} 1 - |p_{ij}^g - p_{ij}^e|
\]

Thus, SPA rewards an evaluator for expressing confidence levels similar to those of the ground truth and penalizes deviations.

C.2 Pairwise Accuracy with Tie Calibration (acc$_{eq}^*$)

Deutsch et al. (2023) introduced acc$_{eq}^*$ to account for tied scores in meta-evaluation. Unlike PA and SPA, acc$_{eq}^*$ is a segment-level measure, meaning it evaluates a metric’s ability to estimate the quality of individual translations rather than MT systems. Specifically, acc$_{eq}^*$ counts how often an evaluator \( e \) ranks pairs of translations of the same source segment in the same order as the ground truth \( g \), accounting for tied scores.

Let \( C \) be the number of translation pairs ranked in the same order by both the evaluator \( e \) and the ground truth \( g \). Similarly, let \( D \) denote the pairs ranked in the opposite order. The terms \( T_e \) and \( T_g \) represent pairs tied only in the evaluator’s scores and only in the ground truth, respectively. Lastly, \( T_{eg} \) refers to pairs tied in both the evaluator’s scores and the ground truth. acc$_{eq}^*$ is then defined as:

\[
acc_{eq}^* = \frac{C + T_{eg}}{C + D + T_e + T_g + T_{eg}}
\]

Tie Calibration Many automatic metrics produce assessments on a continuous scale, such as the real numbers in the interval [0, 1]. As a consequence, these metrics rarely, if ever, produce tied scores, resulting in \( T_e \approx 0 \) and \( T_{eg} \approx 0 \). The Tie
Calibration algorithm addresses this issue by estimating a threshold value \( \epsilon_e \) for each evaluator \( e \), such that two assessments \( e_i \) and \( e_j \) are considered tied if \( |e_i - e_j| \leq \epsilon_e \).

D  Metrics

Table 3 lists the complete set of automatic evaluators considered in this work.

E  Full Rankings

Tables 4, 5, 6, and 7 present the same rankings of Table 2, but including all tested evaluators.

F  Full Rankings Varying the Ground Truth

In this section, we examine how evaluator rankings vary depending on the choice of human evaluator used as ground truth. Specifically, we use the following evaluators as ground truth:

• pSQM-1 from Table 4.
• DA+SQM from Table 6.
• MQM-2023-2 from Table 6.
• MQM-2023-3 from Table 6.
• ESA from Table 7.

We exclude the ESA-1, ESA-2, and MQM-2023-4 evaluators from the 2023 EN→DE test set, as they annotated only a limited number of segments. This increases the number of segments in the 2023 EN→DE test set from 145 to 376. Therefore, for reference, we also report results on this test set using the same evaluator as in Tables 2 and 6. Results are presented in Tables 8, 9, 10, 11, 12, and 13.

As we can see, our findings remain valid when varying the evaluator selected as ground truth, with human evaluators consistently ranking the same as or lower than automatic metrics.
<table>
  <tr>
    <th>Metric</th>
    <th>Reference paper</th>
    <th>Metric</th>
    <th>Reference paper</th>
  </tr>
  <tr>
    <td>all-rembert-20</td>
    <td>(Mathur et al., 2020)</td>
    <td>metametrics_mt_mqm</td>
    <td>(Anugraha et al., 2024)</td>
  </tr>
  <tr>
    <td>BAQ_dyn</td>
    <td>(Mathur et al., 2020)</td>
    <td>metametrics_mt_mqm_qe</td>
    <td>(Anugraha et al., 2024)</td>
  </tr>
  <tr>
    <td>BAQ_static</td>
    <td>(Mathur et al., 2020)</td>
    <td>MetricX-23-QE-XXL</td>
    <td>(Juraska et al., 2023)</td>
  </tr>
  <tr>
    <td>BERT-base-L2</td>
    <td>(Mathur et al., 2020)</td>
    <td>MetricX-23-XXL</td>
    <td>(Juraska et al., 2023)</td>
  </tr>
  <tr>
    <td>BERT-large-L2</td>
    <td>(Mathur et al., 2020)</td>
    <td>MetricX-24-Hybrid</td>
    <td>(Juraska et al., 2024)</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>(Zhang et al., 2020)</td>
    <td>MetricX-24-Hybrid-QE</td>
    <td>(Juraska et al., 2024)</td>
  </tr>
  <tr>
    <td>BLCOM_1</td>
    <td>(Freitag et al., 2024)</td>
    <td>metricx_xxl_MQM_2020</td>
    <td>(Freitag et al., 2022)</td>
  </tr>
  <tr>
    <td>BLEU</td>
    <td>(Papineni et al., 2002)</td>
    <td>mre-score-labse-regular</td>
    <td>(Viskov et al., 2023)</td>
  </tr>
  <tr>
    <td>BLEURT</td>
    <td>(Sellam et al., 2020a)</td>
    <td>MS-COMET-22</td>
    <td>(Kocmi et al., 2022b)</td>
  </tr>
  <tr>
    <td>BLEURT-0.1-all</td>
    <td>(Mathur et al., 2020)</td>
    <td>MS-COMET-QE-22</td>
    <td>(Kocmi et al., 2022b)</td>
  </tr>
  <tr>
    <td>BLEURT-0.1-en</td>
    <td>(Mathur et al., 2020)</td>
    <td>OpenKiwi-Bert</td>
    <td>(Kepler et al., 2019)</td>
  </tr>
  <tr>
    <td>BLEURT-0.2</td>
    <td>(Mathur et al., 2020)</td>
    <td>OpenKiwi-XLMR</td>
    <td>(Kepler et al., 2019)</td>
  </tr>
  <tr>
    <td>BLEURT-20</td>
    <td>(Sellam et al., 2020a)</td>
    <td>parbleu</td>
    <td>(Bawden et al., 2020)</td>
  </tr>
  <tr>
    <td>bleurt-combi</td>
    <td>(Mathur et al., 2020)</td>
    <td>parchrf++</td>
    <td>(Bawden et al., 2020)</td>
  </tr>
  <tr>
    <td>BLEURT-extended</td>
    <td>(Sellam et al., 2020b)</td>
    <td>paresim-1</td>
    <td>(Bawden et al., 2020)</td>
  </tr>
  <tr>
    <td>bright-qe</td>
    <td>(Freitag et al., 2024)</td>
    <td>prism</td>
    <td>(Thompson and Post, 2020a)</td>
  </tr>
  <tr>
    <td>Calibri-COMET22</td>
    <td>(Freitag et al., 2023)</td>
    <td>prismRef</td>
    <td>(Thompson and Post, 2020a,b)</td>
  </tr>
  <tr>
    <td>Calibri-COMET22-QE</td>
    <td>(Freitag et al., 2023)</td>
    <td>PrismRefMedium</td>
    <td>(Thompson and Post, 2020a,b)</td>
  </tr>
  <tr>
    <td>CharacTER</td>
    <td>(Wang et al., 2016)</td>
    <td>PrismRefSmall</td>
    <td>(Thompson and Post, 2020a,b)</td>
  </tr>
  <tr>
    <td>chrF</td>
    <td>(Popović, 2015)</td>
    <td>prismSrc</td>
    <td>(Thompson and Post, 2020a,b)</td>
  </tr>
  <tr>
    <td>chrF++</td>
    <td>(Popović, 2017)</td>
    <td>Random-synname</td>
    <td>(Freitag et al., 2023)</td>
  </tr>
  <tr>
    <td>chrFS</td>
    <td>(Mukherjee and Shrivastava, 2024)</td>
    <td>REUSE</td>
    <td>(Mukherjee and Shrivastava, 2022a)</td>
  </tr>
  <tr>
    <td>COMET</td>
    <td>(Rei et al., 2020b)</td>
    <td>sentBLEU</td>
    <td>(Papineni et al., 2002)</td>
  </tr>
  <tr>
    <td>COMET-20</td>
    <td>(Rei et al., 2020a)</td>
    <td>sentinel-cand-mqm</td>
    <td>(Perrella et al., 2024b)</td>
  </tr>
  <tr>
    <td>COMET-22</td>
    <td>(Rei et al., 2022a)</td>
    <td>sentinel-ref-mqm</td>
    <td>(Perrella et al., 2024b)</td>
  </tr>
  <tr>
    <td>COMET-2R</td>
    <td>(Rei et al., 2020b)</td>
    <td>sentinel-src-mqm</td>
    <td>(Perrella et al., 2024b)</td>
  </tr>
  <tr>
    <td>COMET-HTER</td>
    <td>(Rei et al., 2020b)</td>
    <td>SESscore</td>
    <td>(Xu et al., 2022)</td>
  </tr>
  <tr>
    <td>COMET-MQM</td>
    <td>(Rei et al., 2020b)</td>
    <td>sescoreX</td>
    <td>(Xu et al., 2023)</td>
  </tr>
  <tr>
    <td>COMET-QE</td>
    <td>(Rei et al., 2021)</td>
    <td>spBLEU</td>
    <td>(Team et al., 2022)</td>
  </tr>
  <tr>
    <td>COMET-Rank</td>
    <td>(Rei et al., 2020b)</td>
    <td>SWSS+METEOR</td>
    <td>(Xu et al., 2020)</td>
  </tr>
  <tr>
    <td>COMETKiwi</td>
    <td>(Rei et al., 2022b)</td>
    <td>TER</td>
    <td>(Snover et al., 2006)</td>
  </tr>
  <tr>
    <td>CometKiwi-XL</td>
    <td>(Rei et al., 2023)</td>
    <td>tokengram_F</td>
    <td>(Dreano et al., 2023b)</td>
  </tr>
  <tr>
    <td>CometKiwi-XXL</td>
    <td>(Rei et al., 2023)</td>
    <td>UniTE</td>
    <td>(Wan et al., 2022b,a)</td>
  </tr>
  <tr>
    <td>cometoid22-wmt22</td>
    <td>(Gowda et al., 2023)</td>
    <td>UniTE-src</td>
    <td>(Wan et al., 2022b)</td>
  </tr>
  <tr>
    <td>damonmonli</td>
    <td>(Freitag et al., 2024)</td>
    <td>XCOMET</td>
    <td>(Guerrero et al., 2024)</td>
  </tr>
  <tr>
    <td>docWMT22CometDA</td>
    <td>(Vernikos et al., 2022)</td>
    <td>XCOMET-Ensemble</td>
    <td>(Guerrero et al., 2024)</td>
  </tr>
  <tr>
    <td>docWMT22CometKiwiDA</td>
    <td>(Vernikos et al., 2022)</td>
    <td>XCOMET-QE</td>
    <td>(Guerrero et al., 2024)</td>
  </tr>
  <tr>
    <td>eBLEU</td>
    <td>(ElNokrashy and Kocmi, 2023)</td>
    <td>XCOMET-QE-Ensemble</td>
    <td>(Guerrero et al., 2024)</td>
  </tr>
  <tr>
    <td>EED</td>
    <td>(Stanchev et al., 2019)</td>
    <td>XLsim</td>
    <td>(Mukherjee and Shrivastava, 2023)</td>
  </tr>
  <tr>
    <td>embed_llama</td>
    <td>(Dreano et al., 2023a)</td>
    <td>XLsimMqm</td>
    <td>(Mukherjee and Shrivastava, 2023)</td>
  </tr>
  <tr>
    <td>esim</td>
    <td>(Mathur et al., 2019)</td>
    <td>YiSi-0</td>
    <td>(Lo, 2019)</td>
  </tr>
  <tr>
    <td>f200kpBLEU</td>
    <td>(Team et al., 2022)</td>
    <td>YiSi-1</td>
    <td>(Lo, 2019)</td>
  </tr>
  <tr>
    <td>GEMBA-MQM</td>
    <td>(Kocmi and Federmann, 2023a)</td>
    <td>YiSi-2</td>
    <td>(Lo, 2019)</td>
  </tr>
  <tr>
    <td>gemba_esa</td>
    <td>(Freitag et al., 2024)</td>
    <td>YiSi-combi</td>
    <td>(Mathur et al., 2020)</td>
  </tr>
  <tr>
    <td>HWTS-Teacher-Sim</td>
    <td>(Liu et al., 2022)</td>
    <td>yisi1-translate</td>
    <td>(Mathur et al., 2020)</td>
  </tr>
  <tr>
    <td>KG-BERTScore</td>
    <td>(Wu et al., 2023)</td>
    <td>mbr-metrics-qe</td>
    <td>(Naskar et al., 2023)</td>
  </tr>
  <tr>
    <td>MaTESe</td>
    <td>(Perrella et al., 2022)</td>
    <td>mBERT-L2</td>
    <td>(Mathur et al., 2020)</td>
  </tr>
  <tr>
    <td>MaTESe-QE</td>
    <td>(Perrella et al., 2022)</td>
    <td>MEE</td>
    <td>(Mukherjee et al., 2020)</td>
  </tr>
  <tr>
    <td>MEE4</td>
    <td>(Mukherjee and Shrivastava, 2022b)</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 3: List of all automatic evaluators considered, i.e., MT metrics, associated with their reference papers. Metrics without dedicated papers cite the Metrics Shared Task results paper in which they appeared.
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="4">EN→DE</th>
    <th colspan="4">ZH→EN</th>
  </tr>
  <tr>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sup>*</sup><sub>eq</sub></th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sup>*</sup><sub>eq</sub></th>
    <th>Rank</th>
  </tr>
  <tr>
    <td>MQM-2020-2</td>
    <td></td>
    <td>1</td>
    <td>96.45</td>
    <td></td>
    <td>1</td>
    <td>58.86</td>
    <td></td>
    <td>1</td>
    <td>88.10</td>
    <td></td>
    <td>1</td>
    <td>55.70</td>
  </tr>
  <tr>
    <td>pSQM-1</td>
    <td></td>
    <td>1</td>
    <td>95.59</td>
    <td></td>
    <td>6</td>
    <td>49.41</td>
    <td></td>
    <td>1</td>
    <td>79.16</td>
    <td></td>
    <td>13</td>
    <td>43.89</td>
  </tr>
  <tr>
    <td>MQM-2020-3</td>
    <td></td>
    <td>2</td>
    <td>90.39</td>
    <td></td>
    <td>2</td>
    <td>56.84</td>
    <td></td>
    <td>1</td>
    <td>92.06</td>
    <td></td>
    <td>2</td>
    <td>52.80</td>
  </tr>
  <tr>
    <td>BLEURT-0.2</td>
    <td></td>
    <td>2</td>
    <td>86.81</td>
    <td></td>
    <td>4</td>
    <td>50.81</td>
    <td></td>
    <td>2</td>
    <td>72.59</td>
    <td></td>
    <td>3</td>
    <td>50.57</td>
  </tr>
  <tr>
    <td>pSQM-2</td>
    <td></td>
    <td>2</td>
    <td>85.87</td>
    <td></td>
    <td>9</td>
    <td>46.97</td>
    <td></td>
    <td>1</td>
    <td>89.33</td>
    <td></td>
    <td>9</td>
    <td>46.77</td>
  </tr>
  <tr>
    <td>BLEURT-20</td>
    <td></td>
    <td>2</td>
    <td>85.52</td>
    <td></td>
    <td>3</td>
    <td>51.68</td>
    <td></td>
    <td>3</td>
    <td>67.46</td>
    <td></td>
    <td>4</td>
    <td>50.12</td>
  </tr>
  <tr>
    <td>pSQM-3</td>
    <td></td>
    <td>2</td>
    <td>84.61</td>
    <td></td>
    <td>6</td>
    <td>49.38</td>
    <td></td>
    <td>1</td>
    <td>87.94</td>
    <td></td>
    <td>7</td>
    <td>47.88</td>
  </tr>
  <tr>
    <td>all-rembert-20</td>
    <td></td>
    <td>3</td>
    <td>79.19</td>
    <td></td>
    <td>4</td>
    <td>51.04</td>
    <td></td>
    <td>3</td>
    <td>66.41</td>
    <td></td>
    <td>3</td>
    <td>50.61</td>
  </tr>
  <tr>
    <td>BLEURT-extended</td>
    <td></td>
    <td>3</td>
    <td>75.55</td>
    <td></td>
    <td>5</td>
    <td>50.21</td>
    <td></td>
    <td>3</td>
    <td>64.00</td>
    <td></td>
    <td>3</td>
    <td>50.74</td>
  </tr>
  <tr>
    <td>COMET-MQM</td>
    <td></td>
    <td>4</td>
    <td>71.39</td>
    <td></td>
    <td>7</td>
    <td>48.21</td>
    <td></td>
    <td>4</td>
    <td>55.43</td>
    <td></td>
    <td>6</td>
    <td>48.49</td>
  </tr>
  <tr>
    <td>BLEURT-0.1-all</td>
    <td></td>
    <td>4</td>
    <td>71.38</td>
    <td></td>
    <td>7</td>
    <td>48.63</td>
    <td></td>
    <td>2</td>
    <td>71.04</td>
    <td></td>
    <td>5</td>
    <td>49.54</td>
  </tr>
  <tr>
    <td>COMET</td>
    <td></td>
    <td>4</td>
    <td>71.09</td>
    <td></td>
    <td>8</td>
    <td>47.36</td>
    <td></td>
    <td>4</td>
    <td>56.01</td>
    <td></td>
    <td>5</td>
    <td>49.28</td>
  </tr>
  <tr>
    <td>COMET-QE*</td>
    <td></td>
    <td>4</td>
    <td>70.59</td>
    <td></td>
    <td>8</td>
    <td>47.82</td>
    <td></td>
    <td>3</td>
    <td>58.37</td>
    <td></td>
    <td>8</td>
    <td>47.09</td>
  </tr>
  <tr>
    <td>COMET-HTER</td>
    <td></td>
    <td>5</td>
    <td>65.71</td>
    <td></td>
    <td>8</td>
    <td>47.62</td>
    <td></td>
    <td>4</td>
    <td>54.79</td>
    <td></td>
    <td>5</td>
    <td>49.30</td>
  </tr>
  <tr>
    <td>mBERT-L2</td>
    <td></td>
    <td>5</td>
    <td>65.03</td>
    <td></td>
    <td>10</td>
    <td>45.48</td>
    <td></td>
    <td>4</td>
    <td>56.49</td>
    <td></td>
    <td>6</td>
    <td>48.97</td>
  </tr>
  <tr>
    <td>COMET-2R</td>
    <td></td>
    <td>6</td>
    <td>58.12</td>
    <td></td>
    <td>9</td>
    <td>46.43</td>
    <td></td>
    <td>4</td>
    <td>55.99</td>
    <td></td>
    <td>4</td>
    <td>50.20</td>
  </tr>
  <tr>
    <td>COMET-Rank</td>
    <td></td>
    <td>6</td>
    <td>54.78</td>
    <td></td>
    <td>14</td>
    <td>41.31</td>
    <td></td>
    <td>3</td>
    <td>58.16</td>
    <td></td>
    <td>14</td>
    <td>43.57</td>
  </tr>
  <tr>
    <td>OpenKiwi-XLMR*</td>
    <td></td>
    <td>6</td>
    <td>53.25</td>
    <td></td>
    <td>11</td>
    <td>44.11</td>
    <td></td>
    <td>4</td>
    <td>53.29</td>
    <td></td>
    <td>8</td>
    <td>47.23</td>
  </tr>
  <tr>
    <td>OpenKiwi-Bert*</td>
    <td></td>
    <td>6</td>
    <td>52.01</td>
    <td></td>
    <td>16</td>
    <td>39.98</td>
    <td></td>
    <td>3</td>
    <td>59.55</td>
    <td></td>
    <td>11</td>
    <td>45.13</td>
  </tr>
  <tr>
    <td>prism</td>
    <td></td>
    <td>6</td>
    <td>51.92</td>
    <td></td>
    <td>11</td>
    <td>43.59</td>
    <td></td>
    <td>4</td>
    <td>57.88</td>
    <td></td>
    <td>8</td>
    <td>47.56</td>
  </tr>
  <tr>
    <td>Yisi-combi</td>
    <td></td>
    <td>7</td>
    <td>51.10</td>
    <td></td>
    <td>12</td>
    <td>42.63</td>
    <td></td>
    <td>–</td>
    <td>–</td>
    <td></td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>bleurt-combi</td>
    <td></td>
    <td>7</td>
    <td>51.10</td>
    <td></td>
    <td>12</td>
    <td>42.63</td>
    <td></td>
    <td>–</td>
    <td>–</td>
    <td></td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>esim</td>
    <td></td>
    <td>7</td>
    <td>50.72</td>
    <td></td>
    <td>14</td>
    <td>41.35</td>
    <td></td>
    <td>4</td>
    <td>52.90</td>
    <td></td>
    <td>10</td>
    <td>46.19</td>
  </tr>
  <tr>
    <td>chrF</td>
    <td></td>
    <td>7</td>
    <td>49.86</td>
    <td></td>
    <td>13</td>
    <td>42.05</td>
    <td></td>
    <td>5</td>
    <td>47.70</td>
    <td></td>
    <td>13</td>
    <td>44.09</td>
  </tr>
  <tr>
    <td>EED</td>
    <td></td>
    <td>7</td>
    <td>49.81</td>
    <td></td>
    <td>15</td>
    <td>40.94</td>
    <td></td>
    <td>5</td>
    <td>45.41</td>
    <td></td>
    <td>14</td>
    <td>43.64</td>
  </tr>
  <tr>
    <td>paresim-1</td>
    <td></td>
    <td>7</td>
    <td>49.54</td>
    <td></td>
    <td>14</td>
    <td>41.37</td>
    <td></td>
    <td>4</td>
    <td>53.34</td>
    <td></td>
    <td>10</td>
    <td>46.15</td>
  </tr>
  <tr>
    <td>chrF++</td>
    <td></td>
    <td>7</td>
    <td>48.87</td>
    <td></td>
    <td>13</td>
    <td>41.99</td>
    <td></td>
    <td>5</td>
    <td>48.96</td>
    <td></td>
    <td>12</td>
    <td>44.27</td>
  </tr>
  <tr>
    <td>YiSi-1</td>
    <td></td>
    <td>7</td>
    <td>48.79</td>
    <td></td>
    <td>12</td>
    <td>42.70</td>
    <td></td>
    <td>4</td>
    <td>52.74</td>
    <td></td>
    <td>7</td>
    <td>48.01</td>
  </tr>
  <tr>
    <td>CharacTER</td>
    <td></td>
    <td>7</td>
    <td>47.71</td>
    <td></td>
    <td>16</td>
    <td>40.45</td>
    <td></td>
    <td>5</td>
    <td>48.84</td>
    <td></td>
    <td>13</td>
    <td>44.01</td>
  </tr>
  <tr>
    <td>BLEURT-0.1-en</td>
    <td></td>
    <td>7</td>
    <td>47.43</td>
    <td></td>
    <td>15</td>
    <td>40.96</td>
    <td></td>
    <td>4</td>
    <td>57.29</td>
    <td></td>
    <td>7</td>
    <td>48.26</td>
  </tr>
  <tr>
    <td>YiSi-0</td>
    <td></td>
    <td>7</td>
    <td>46.23</td>
    <td></td>
    <td>17</td>
    <td>39.78</td>
    <td></td>
    <td>5</td>
    <td>46.47</td>
    <td></td>
    <td>14</td>
    <td>43.60</td>
  </tr>
  <tr>
    <td>TER</td>
    <td></td>
    <td>7</td>
    <td>45.98</td>
    <td></td>
    <td>16</td>
    <td>40.15</td>
    <td></td>
    <td>6</td>
    <td>39.68</td>
    <td></td>
    <td>15</td>
    <td>43.34</td>
  </tr>
  <tr>
    <td>parchrf++</td>
    <td></td>
    <td>7</td>
    <td>45.57</td>
    <td></td>
    <td>13</td>
    <td>42.25</td>
    <td></td>
    <td>5</td>
    <td>48.68</td>
    <td></td>
    <td>12</td>
    <td>44.25</td>
  </tr>
  <tr>
    <td>MEE</td>
    <td></td>
    <td>7</td>
    <td>45.31</td>
    <td></td>
    <td>14</td>
    <td>41.61</td>
    <td></td>
    <td>4</td>
    <td>52.91</td>
    <td></td>
    <td>13</td>
    <td>43.94</td>
  </tr>
  <tr>
    <td>sentBLEU</td>
    <td></td>
    <td>7</td>
    <td>44.41</td>
    <td></td>
    <td>15</td>
    <td>41.07</td>
    <td></td>
    <td>4</td>
    <td>50.45</td>
    <td></td>
    <td>15</td>
    <td>43.37</td>
  </tr>
  <tr>
    <td>parbleu</td>
    <td></td>
    <td>8</td>
    <td>41.38</td>
    <td></td>
    <td>15</td>
    <td>41.01</td>
    <td></td>
    <td>4</td>
    <td>50.28</td>
    <td></td>
    <td>15</td>
    <td>43.43</td>
  </tr>
  <tr>
    <td>yisi1-translate</td>
    <td></td>
    <td>8</td>
    <td>39.76</td>
    <td></td>
    <td>12</td>
    <td>42.60</td>
    <td></td>
    <td>4</td>
    <td>52.28</td>
    <td></td>
    <td>11</td>
    <td>44.70</td>
  </tr>
  <tr>
    <td>YiSi-2*</td>
    <td></td>
    <td>8</td>
    <td>38.44</td>
    <td></td>
    <td>18</td>
    <td>34.36</td>
    <td></td>
    <td>5</td>
    <td>43.35</td>
    <td></td>
    <td>12</td>
    <td>44.60</td>
  </tr>
</table>

Table 4: 2020
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="4">EN→DE</th>
    <th colspan="4">EN→ZH</th>
  </tr>
  <tr>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sup>eq</sup></th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>SPA</th>
    <th>Rank</th>
    <th>Acc.</th>
    <th>acc<sup>eq</sup></th>
    <th>Rank</th>
  </tr>
  <tr>
    <td>MetricX-23-QE-XXL*</td>
    <td>1</td>
    <td>94.89</td>
    <td>3</td>
    <td>57.64</td>
    <td>2</td>
    <td>83.92</td>
    <td>2</td>
    <td>47.43</td>
  </tr>
  <tr>
    <td>MQM-2022-2</td>
    <td>1</td>
    <td>94.49</td>
    <td>6</td>
    <td>55.55</td>
    <td>2</td>
    <td>80.82</td>
    <td>3</td>
    <td>47.05</td>
  </tr>
  <tr>
    <td>MQM-2022-3</td>
    <td>1</td>
    <td>92.59</td>
    <td>1</td>
    <td>61.06</td>
    <td>1</td>
    <td>87.22</td>
    <td>2</td>
    <td>47.56</td>
  </tr>
  <tr>
    <td>MetricX-23-XXL</td>
    <td>2</td>
    <td>92.34</td>
    <td>2</td>
    <td>59.27</td>
    <td>1</td>
    <td>87.69</td>
    <td>1</td>
    <td>48.43</td>
  </tr>
  <tr>
    <td>COMET-22</td>
    <td>2</td>
    <td>91.63</td>
    <td>5</td>
    <td>56.51</td>
    <td>2</td>
    <td>84.08</td>
    <td>3</td>
    <td>46.74</td>
  </tr>
  <tr>
    <td>COMET-20</td>
    <td>2</td>
    <td>91.28</td>
    <td>9</td>
    <td>52.42</td>
    <td>2</td>
    <td>80.56</td>
    <td>7</td>
    <td>43.81</td>
  </tr>
  <tr>
    <td>CometKiwi*</td>
    <td>2</td>
    <td>89.51</td>
    <td>7</td>
    <td>53.77</td>
    <td>3</td>
    <td>75.36</td>
    <td>8</td>
    <td>43.21</td>
  </tr>
  <tr>
    <td>BLEURT-20</td>
    <td>3</td>
    <td>88.20</td>
    <td>7</td>
    <td>53.33</td>
    <td>3</td>
    <td>77.80</td>
    <td>7</td>
    <td>43.84</td>
  </tr>
  <tr>
    <td>metricx_xxl_MQM_2020</td>
    <td>3</td>
    <td>88.10</td>
    <td>3</td>
    <td>57.43</td>
    <td>1</td>
    <td>87.04</td>
    <td>3</td>
    <td>46.89</td>
  </tr>
  <tr>
    <td>COMET-QE*</td>
    <td>3</td>
    <td>85.51</td>
    <td>10</td>
    <td>51.69</td>
    <td>3</td>
    <td>78.33</td>
    <td>7</td>
    <td>43.61</td>
  </tr>
  <tr>
    <td>MS-COMET-22</td>
    <td>3</td>
    <td>85.37</td>
    <td>8</td>
    <td>53.13</td>
    <td>1</td>
    <td>85.18</td>
    <td>6</td>
    <td>44.92</td>
  </tr>
  <tr>
    <td>CometKiwi-XXL*</td>
    <td>3</td>
    <td>84.43</td>
    <td>7</td>
    <td>53.27</td>
    <td>2</td>
    <td>81.25</td>
    <td>2</td>
    <td>47.28</td>
  </tr>
  <tr>
    <td>UniTE</td>
    <td>4</td>
    <td>82.77</td>
    <td>4</td>
    <td>57.03</td>
    <td>2</td>
    <td>83.88</td>
    <td>5</td>
    <td>45.86</td>
  </tr>
  <tr>
    <td>UniTE-src*</td>
    <td>4</td>
    <td>81.55</td>
    <td>6</td>
    <td>55.00</td>
    <td>4</td>
    <td>65.74</td>
    <td>7</td>
    <td>43.53</td>
  </tr>
  <tr>
    <td>CometKiwi-XL*</td>
    <td>4</td>
    <td>81.13</td>
    <td>8</td>
    <td>52.73</td>
    <td>2</td>
    <td>81.56</td>
    <td>4</td>
    <td>46.33</td>
  </tr>
  <tr>
    <td>YiSi-1</td>
    <td>4</td>
    <td>78.91</td>
    <td>13</td>
    <td>48.26</td>
    <td>4</td>
    <td>70.72</td>
    <td>8</td>
    <td>43.23</td>
  </tr>
  <tr>
    <td>MATESE</td>
    <td>5</td>
    <td>78.03</td>
    <td>7</td>
    <td>53.48</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>5</td>
    <td>75.61</td>
    <td>14</td>
    <td>47.57</td>
    <td>4</td>
    <td>70.69</td>
    <td>8</td>
    <td>43.28</td>
  </tr>
  <tr>
    <td>SESscore</td>
    <td>5</td>
    <td>75.16</td>
    <td>12</td>
    <td>50.45</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>MS-COMET-QE-22*</td>
    <td>5</td>
    <td>74.44</td>
    <td>12</td>
    <td>50.37</td>
    <td>2</td>
    <td>78.84</td>
    <td>9</td>
    <td>42.51</td>
  </tr>
  <tr>
    <td>MEE4</td>
    <td>5</td>
    <td>74.19</td>
    <td>15</td>
    <td>46.81</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>chrF</td>
    <td>5</td>
    <td>73.05</td>
    <td>16</td>
    <td>46.38</td>
    <td>3</td>
    <td>72.67</td>
    <td>10</td>
    <td>41.87</td>
  </tr>
  <tr>
    <td>f200spBLEU</td>
    <td>5</td>
    <td>71.04</td>
    <td>15</td>
    <td>46.84</td>
    <td>4</td>
    <td>71.76</td>
    <td>10</td>
    <td>41.85</td>
  </tr>
  <tr>
    <td>HWTSC-Teacher-Sim*</td>
    <td>5</td>
    <td>69.68</td>
    <td>13</td>
    <td>48.10</td>
    <td>4</td>
    <td>68.43</td>
    <td>11</td>
    <td>40.53</td>
  </tr>
  <tr>
    <td><b>DA+SQM</b></td>
    <td>6</td>
    <td>66.61</td>
    <td>16</td>
    <td>46.03</td>
    <td>2</td>
    <td>82.95</td>
    <td>12</td>
    <td>36.26</td>
  </tr>
  <tr>
    <td>MATESE-QE*</td>
    <td>6</td>
    <td>65.42</td>
    <td>11</td>
    <td>51.06</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>BLEU</td>
    <td>6</td>
    <td>65.00</td>
    <td>15</td>
    <td>46.51</td>
    <td>4</td>
    <td>67.31</td>
    <td>13</td>
    <td>34.28</td>
  </tr>
  <tr>
    <td>REUSE*</td>
    <td>7</td>
    <td>37.95</td>
    <td>17</td>
    <td>43.58</td>
    <td>5</td>
    <td>33.46</td>
    <td>12</td>
    <td>35.89</td>
  </tr>
</table>

Table 5: 2022
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="4">EN→DE</th>
    <th colspan="4">ZH→EN</th>
  </tr>
  <tr>
    <th>SPA</th><th>Rank</th><th>Acc.</th><th>acc<sub>eq</sub></th><th>Rank</th><th>Acc.</th><th>SPA</th><th>Rank</th><th>Acc.</th><th>acc<sub>eq</sub></th><th>Rank</th><th>Acc.</th>
  </tr>
  <tr><td>GEMBA-MQM*</td><td>1</td><td>94.52</td><td>5</td><td>58.52</td><td>1</td><td>93.17</td><td>3</td><td>52.80</td></tr>
  <tr><td>MQM-2023-3</td><td>1</td><td>93.51</td><td>5</td><td>58.42</td><td>1</td><td>95.54</td><td>5</td><td>51.65</td></tr>
  <tr><td>CometKiwi-XXL*</td><td>1</td><td>93.22</td><td>5</td><td>58.46</td><td>1</td><td>92.86</td><td>6</td><td>50.94</td></tr>
  <tr><td>MQM-2023-2</td><td>1</td><td>93.15</td><td>6</td><td>57.71</td><td>1</td><td>95.18</td><td>2</td><td>52.90</td></tr>
  <tr><td>CometKiwi-XL*</td><td>1</td><td>93.11</td><td>6</td><td>57.38</td><td>2</td><td>92.02</td><td>6</td><td>50.71</td></tr>
  <tr><td>MetricX-23-XXL</td><td>1</td><td>92.57</td><td>2</td><td>61.82</td><td>2</td><td>91.58</td><td>2</td><td>53.13</td></tr>
  <tr><td>XCOMET-QE-Ensemble*</td><td>1</td><td>92.48</td><td>4</td><td>59.89</td><td>2</td><td>90.54</td><td>3</td><td>52.87</td></tr>
  <tr><td>cometoid22-wmt22*</td><td>1</td><td>92.43</td><td>5</td><td>58.09</td><td>2</td><td>90.09</td><td>7</td><td>50.23</td></tr>
  <tr><td>COMET</td><td>1</td><td>92.33</td><td>7</td><td>56.65</td><td>4</td><td>87.18</td><td>9</td><td>48.42</td></tr>
  <tr><td>XCOMET-Ensemble</td><td>1</td><td>92.21</td><td>3</td><td>60.99</td><td>2</td><td>91.15</td><td>1</td><td>54.59</td></tr>
  <tr><td>MetricX-23-QE-XXL*</td><td>1</td><td>92.12</td><td>1</td><td>62.53</td><td>3</td><td>88.30</td><td>2</td><td>53.26</td></tr>
  <tr><td>Calibri-COMET22</td><td>1</td><td>92.01</td><td>10</td><td>51.26</td><td>4</td><td>87.02</td><td>16</td><td>44.57</td></tr>
  <tr><td>docWMT22CometDA</td><td>1</td><td>91.76</td><td>8</td><td>54.71</td><td>4</td><td>87.41</td><td>13</td><td>46.15</td></tr>
  <tr><td>sescoreX</td><td>1</td><td>91.66</td><td>8</td><td>54.76</td><td>4</td><td>85.73</td><td>13</td><td>46.39</td></tr>
  <tr><td>DA+SQM</td><td>2</td><td>91.24</td><td>14</td><td>46.79</td><td>4</td><td>86.28</td><td>22</td><td>39.42</td></tr>
  <tr><td>Calibri-COMET22-QE*</td><td>2</td><td>90.80</td><td>11</td><td>50.33</td><td>4</td><td>87.59</td><td>11</td><td>47.24</td></tr>
  <tr><td>ESA-1</td><td>2</td><td>90.39</td><td>14</td><td>46.71</td><td>–</td><td>–</td><td>–</td><td>–</td></tr>
  <tr><td>BLEURT-20</td><td>2</td><td>90.35</td><td>8</td><td>55.19</td><td>4</td><td>87.36</td><td>9</td><td>48.63</td></tr>
  <tr><td>mbr-metricx-qe*</td><td>2</td><td>89.98</td><td>5</td><td>58.75</td><td>3</td><td>88.55</td><td>4</td><td>52.04</td></tr>
  <tr><td>prismRef</td><td>2</td><td>89.92</td><td>11</td><td>50.72</td><td>5</td><td>82.50</td><td>14</td><td>46.06</td></tr>
  <tr><td>docWMT22CometKiwiDA*</td><td>2</td><td>89.92</td><td>8</td><td>55.30</td><td>2</td><td>90.95</td><td>10</td><td>47.83</td></tr>
  <tr><td>MS-COMET-QE-22*</td><td>2</td><td>89.85</td><td>8</td><td>54.55</td><td>4</td><td>87.59</td><td>10</td><td>47.81</td></tr>
  <tr><td>f200spBLEU</td><td>2</td><td>89.24</td><td>11</td><td>50.54</td><td>5</td><td>81.12</td><td>18</td><td>43.33</td></tr>
  <tr><td>CometKiwi*</td><td>2</td><td>89.23</td><td>6</td><td>57.74</td><td>3</td><td>89.37</td><td>5</td><td>51.74</td></tr>
  <tr><td>mre-score-labse-regular</td><td>2</td><td>89.14</td><td>10</td><td>51.12</td><td>4</td><td>87.14</td><td>17</td><td>43.80</td></tr>
  <tr><td>ESA-2</td><td>2</td><td>89.11</td><td>12</td><td>49.70</td><td>–</td><td>–</td><td>–</td><td>–</td></tr>
  <tr><td>YiSi-1</td><td>2</td><td>88.96</td><td>9</td><td>53.15</td><td>4</td><td>85.70</td><td>12</td><td>46.68</td></tr>
  <tr><td>MQM-2023-4</td><td>2</td><td>88.93</td><td>14</td><td>46.68</td><td>–</td><td>–</td><td>–</td><td>–</td></tr>
  <tr><td>KG-BERTScore*</td><td>2</td><td>88.79</td><td>7</td><td>56.98</td><td>3</td><td>89.31</td><td>8</td><td>49.75</td></tr>
  <tr><td>MaTESc</td><td>2</td><td>88.40</td><td>9</td><td>53.36</td><td>2</td><td>92.06</td><td>7</td><td>50.34</td></tr>
  <tr><td>BLEU</td><td>2</td><td>88.02</td><td>12</td><td>50.06</td><td>6</td><td>80.92</td><td>19</td><td>43.13</td></tr>
  <tr><td>BERTscore</td><td>2</td><td>87.33</td><td>11</td><td>50.88</td><td>5</td><td>84.68</td><td>15</td><td>45.79</td></tr>
  <tr><td>MEE4</td><td>2</td><td>87.07</td><td>10</td><td>51.62</td><td>6</td><td>80.51</td><td>19</td><td>42.94</td></tr>
  <tr><td>XLsim</td><td>2</td><td>86.58</td><td>10</td><td>51.01</td><td>6</td><td>81.00</td><td>19</td><td>42.84</td></tr>
  <tr><td>tokengram_F</td><td>3</td><td>85.60</td><td>12</td><td>49.72</td><td>5</td><td>81.01</td><td>18</td><td>43.52</td></tr>
  <tr><td>chrF</td><td>4</td><td>84.25</td><td>12</td><td>49.54</td><td>5</td><td>81.47</td><td>17</td><td>43.72</td></tr>
  <tr><td>eBLEU</td><td>4</td><td>83.87</td><td>13</td><td>48.96</td><td>6</td><td>80.44</td><td>20</td><td>42.55</td></tr>
  <tr><td>embed_llama</td><td>4</td><td>81.33</td><td>14</td><td>47.12</td><td>4</td><td>84.84</td><td>21</td><td>41.05</td></tr>
  <tr><td>Random-sysname*</td><td>5</td><td>59.47</td><td>16</td><td>39.07</td><td>7</td><td>54.34</td><td>23</td><td>34.49</td></tr>
  <tr><td>prismSrc*</td><td>6</td><td>30.03</td><td>15</td><td>40.89</td><td>8</td><td>35.54</td><td>22</td><td>39.28</td></tr>
</table>

Table 6: 2023
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="2">SPA</th>
    <th colspan="2">acc<sub>eq</sub></th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>Acc.</th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr>
    <td>CometKiwi-XXL*</td>
    <td>1</td>
    <td>86.12</td>
    <td>4</td>
    <td>67.24</td>
  </tr>
  <tr>
    <td>gemba_esa*</td>
    <td>1</td>
    <td>85.72</td>
    <td>3</td>
    <td>67.68</td>
  </tr>
  <tr>
    <td>COMET-22</td>
    <td>1</td>
    <td>82.37</td>
    <td>5</td>
    <td>66.60</td>
  </tr>
  <tr>
    <td>bright-qe*</td>
    <td>1</td>
    <td>81.77</td>
    <td>4</td>
    <td>67.39</td>
  </tr>
  <tr>
    <td><b>ESA</b></td>
    <td><b>2</b></td>
    <td><b>80.12</b></td>
    <td><b>8</b></td>
    <td><b>63.84</b></td>
  </tr>
  <tr>
    <td>XCOMET-QE*</td>
    <td>2</td>
    <td>80.10</td>
    <td>3</td>
    <td>67.99</td>
  </tr>
  <tr>
    <td>metametrics_mt_mqm_hybrid_kendall</td>
    <td>2</td>
    <td>80.10</td>
    <td>1</td>
    <td>68.95</td>
  </tr>
  <tr>
    <td>XCOMET</td>
    <td>2</td>
    <td>79.96</td>
    <td>2</td>
    <td>68.67</td>
  </tr>
  <tr>
    <td>MetricX-24-Hybrid</td>
    <td>2</td>
    <td>79.75</td>
    <td>1</td>
    <td>69.20</td>
  </tr>
  <tr>
    <td>BLCOM_1</td>
    <td>2</td>
    <td>79.17</td>
    <td>6</td>
    <td>65.02</td>
  </tr>
  <tr>
    <td>MetricX-24-Hybrid-QE*</td>
    <td>2</td>
    <td>79.09</td>
    <td>2</td>
    <td>68.92</td>
  </tr>
  <tr>
    <td>sentinel-cand-mqm*</td>
    <td>2</td>
    <td>78.54</td>
    <td>5</td>
    <td>66.39</td>
  </tr>
  <tr>
    <td>BLEURT-20</td>
    <td>2</td>
    <td>75.96</td>
    <td>7</td>
    <td>64.48</td>
  </tr>
  <tr>
    <td>metametrics_mt_mqm_qe_kendall.seg.s*</td>
    <td>3</td>
    <td>73.29</td>
    <td>4</td>
    <td>67.49</td>
  </tr>
  <tr>
    <td>CometKiwi*</td>
    <td>3</td>
    <td>71.74</td>
    <td>5</td>
    <td>66.51</td>
  </tr>
  <tr>
    <td>PrismRefMedium</td>
    <td>3</td>
    <td>70.93</td>
    <td>11</td>
    <td>61.39</td>
  </tr>
  <tr>
    <td>PrismRefSmall</td>
    <td>3</td>
    <td>70.52</td>
    <td>10</td>
    <td>61.51</td>
  </tr>
  <tr>
    <td>YiSi-1</td>
    <td>3</td>
    <td>70.51</td>
    <td>11</td>
    <td>61.44</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>3</td>
    <td>67.75</td>
    <td>11</td>
    <td>61.41</td>
  </tr>
  <tr>
    <td>chrF</td>
    <td>3</td>
    <td>66.73</td>
    <td>13</td>
    <td>61.05</td>
  </tr>
  <tr>
    <td>damonmonli</td>
    <td>3</td>
    <td>66.37</td>
    <td>9</td>
    <td>62.10</td>
  </tr>
  <tr>
    <td>chrFS</td>
    <td>4</td>
    <td>64.31</td>
    <td>11</td>
    <td>61.37</td>
  </tr>
  <tr>
    <td>spBLEU</td>
    <td>4</td>
    <td>63.19</td>
    <td>12</td>
    <td>61.08</td>
  </tr>
  <tr>
    <td>BLEU</td>
    <td>5</td>
    <td>60.67</td>
    <td>13</td>
    <td>61.04</td>
  </tr>
  <tr>
    <td>MEE4</td>
    <td>5</td>
    <td>60.36</td>
    <td>10</td>
    <td>61.57</td>
  </tr>
  <tr>
    <td>sentinel-ref-mqm</td>
    <td>6</td>
    <td>44.19</td>
    <td>13</td>
    <td>61.04</td>
  </tr>
  <tr>
    <td>sentinel-src-mqm*</td>
    <td>6</td>
    <td>44.19</td>
    <td>13</td>
    <td>61.04</td>
  </tr>
  <tr>
    <td>XLsimMqm*</td>
    <td>6</td>
    <td>39.25</td>
    <td>12</td>
    <td>61.11</td>
  </tr>
</table>

Table 7: 2024
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="2">SPA</th>
    <th colspan="2">acc<sup>eq</sup></th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>Acc.</th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr><td>pSQM-3</td><td>1</td><td>83.42</td><td>6</td><td>65.23</td></tr>
  <tr><td>MQM-2020-2</td><td>1</td><td>80.56</td><td>6</td><td>65.22</td></tr>
  <tr><td>MQM-2020-3</td><td>1</td><td>80.34</td><td>6</td><td>65.22</td></tr>
  <tr><td>MQM-2020-1</td><td>1</td><td>79.55</td><td>6</td><td>65.22</td></tr>
  <tr><td>pSQM-2</td><td>1</td><td>74.02</td><td>4</td><td>65.26</td></tr>
  <tr><td>BERT-large-L2</td><td>1</td><td>70.19</td><td>3</td><td>65.38</td></tr>
  <tr><td>COMET</td><td>1</td><td>68.58</td><td>1</td><td>65.64</td></tr>
  <tr><td>SWSS+METEOR</td><td>2</td><td>67.26</td><td>4</td><td>65.26</td></tr>
  <tr><td>MEE</td><td>2</td><td>67.07</td><td>6</td><td>65.22</td></tr>
  <tr><td>prism</td><td>2</td><td>66.01</td><td>5</td><td>65.25</td></tr>
  <tr><td>sentBLEU</td><td>2</td><td>65.70</td><td>5</td><td>65.25</td></tr>
  <tr><td>parbleu</td><td>2</td><td>65.63</td><td>6</td><td>65.23</td></tr>
  <tr><td>BLEURT</td><td>2</td><td>65.28</td><td>2</td><td>65.49</td></tr>
  <tr><td>YiSi-1</td><td>2</td><td>64.71</td><td>6</td><td>65.22</td></tr>
  <tr><td>yisi1-translate</td><td>2</td><td>64.32</td><td>6</td><td>65.23</td></tr>
  <tr><td>CharacTER</td><td>2</td><td>63.99</td><td>6</td><td>65.23</td></tr>
  <tr><td>all-rembert-20</td><td>2</td><td>63.48</td><td>2</td><td>65.47</td></tr>
  <tr><td>BLEURT-20</td><td>2</td><td>63.10</td><td>3</td><td>65.38</td></tr>
  <tr><td>paresim-1</td><td>2</td><td>62.79</td><td>4</td><td>65.30</td></tr>
  <tr><td>esim</td><td>2</td><td>62.42</td><td>4</td><td>65.30</td></tr>
  <tr><td>chrF++</td><td>3</td><td>62.38</td><td>6</td><td>65.23</td></tr>
  <tr><td>BLEURT-0.1-en</td><td>3</td><td>62.22</td><td>2</td><td>65.47</td></tr>
  <tr><td>COMET-2R</td><td>3</td><td>62.10</td><td>1</td><td>65.61</td></tr>
  <tr><td>parchrF++</td><td>3</td><td>61.92</td><td>5</td><td>65.24</td></tr>
  <tr><td>EED</td><td>3</td><td>60.91</td><td>6</td><td>65.23</td></tr>
  <tr><td>mBERT-L2</td><td>3</td><td>60.84</td><td>2</td><td>65.43</td></tr>
  <tr><td>chrF</td><td>3</td><td>60.81</td><td>4</td><td>65.25</td></tr>
  <tr><td>YiSi-0</td><td>3</td><td>60.55</td><td>5</td><td>65.25</td></tr>
  <tr><td>BLEURT-0.2</td><td>3</td><td>60.54</td><td>3</td><td>65.34</td></tr>
  <tr><td>COMET-Rank</td><td>3</td><td>59.97</td><td>6</td><td>65.22</td></tr>
  <tr><td>BLEURT-extended</td><td>3</td><td>59.90</td><td>3</td><td>65.36</td></tr>
  <tr><td>BAQ_static</td><td>3</td><td>59.89</td><td>6</td><td>65.22</td></tr>
  <tr><td>BLEURT-0.1-all</td><td>3</td><td>59.80</td><td>4</td><td>65.25</td></tr>
  <tr><td>BERT-base-L2</td><td>3</td><td>59.57</td><td>2</td><td>65.49</td></tr>
  <tr><td>COMET-HTER</td><td>3</td><td>58.88</td><td>2</td><td>65.42</td></tr>
  <tr><td>COMET-MQM</td><td>3</td><td>58.27</td><td>5</td><td>65.25</td></tr>
  <tr><td>BAQ_dyn</td><td>3</td><td>58.12</td><td>6</td><td>65.22</td></tr>
  <tr><td>COMET-QE*</td><td>3</td><td>57.83</td><td>4</td><td>65.26</td></tr>
  <tr><td>TER</td><td>3</td><td>56.12</td><td>5</td><td>65.25</td></tr>
  <tr><td>OpenKiwi-Bert*</td><td>3</td><td>55.35</td><td>5</td><td>65.25</td></tr>
  <tr><td>OpenKiwi-XLMR*</td><td>3</td><td>50.98</td><td>4</td><td>65.30</td></tr>
  <tr><td>YiSi-2*</td><td>4</td><td>48.35</td><td>4</td><td>65.27</td></tr>
</table>

Table 8: The test set is 2020 ZH→EN. The evaluator selected as the ground truth follows the pSQM protocol (pSQM-1 in Table 4).
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="2">SPA</th>
    <th colspan="2">acc*<sub>eq</sub></th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>Acc.</th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr><td>MetricX-23-QE-XXL*</td><td>1</td><td>95.11</td><td>3</td><td>58.41</td></tr>
  <tr><td>GEMBA-MQM*</td><td>1</td><td>94.89</td><td>14</td><td>43.06</td></tr>
  <tr><td>CometKiwi*</td><td>1</td><td>94.83</td><td>3</td><td>58.21</td></tr>
  <tr><td>KG-BERTScore*</td><td>1</td><td>94.57</td><td>5</td><td>57.16</td></tr>
  <tr><td><b>MQM-2023-3</b></td><td>1</td><td>94.24</td><td>13</td><td>46.99</td></tr>
  <tr><td>docWMT22CometKiwiDA*</td><td>1</td><td>94.22</td><td>2</td><td>58.60</td></tr>
  <tr><td>CometKiwi-XL*</td><td>1</td><td>94.16</td><td>2</td><td>58.80</td></tr>
  <tr><td>MS-COMET-QE-22*</td><td>1</td><td>94.04</td><td>7</td><td>55.66</td></tr>
  <tr><td>CometKiwi-XXL*</td><td>1</td><td>93.98</td><td>1</td><td>59.66</td></tr>
  <tr><td>MetricX-23-XXL</td><td>2</td><td>93.31</td><td>3</td><td>58.06</td></tr>
  <tr><td>COMET</td><td>2</td><td>92.80</td><td>3</td><td>58.42</td></tr>
  <tr><td>docWMT22CometDA</td><td>2</td><td>92.45</td><td>2</td><td>58.95</td></tr>
  <tr><td>mre-score-labse-regular</td><td>2</td><td>92.30</td><td>8</td><td>54.75</td></tr>
  <tr><td><b>MQM-2023-1</b></td><td>2</td><td>92.10</td><td>15</td><td>42.20</td></tr>
  <tr><td>Calibri-COMET22</td><td>2</td><td>92.02</td><td>3</td><td>58.17</td></tr>
  <tr><td>mbr-metricx-qe*</td><td>2</td><td>91.94</td><td>3</td><td>58.26</td></tr>
  <tr><td><b>MQM-2023-2</b></td><td>2</td><td>91.40</td><td>11</td><td>48.91</td></tr>
  <tr><td>cometoid22-wmt22*</td><td>2</td><td>91.34</td><td>5</td><td>57.06</td></tr>
  <tr><td>sescoreX</td><td>2</td><td>91.22</td><td>4</td><td>57.41</td></tr>
  <tr><td>BLEURT-20</td><td>3</td><td>90.66</td><td>5</td><td>57.26</td></tr>
  <tr><td>prismRef</td><td>3</td><td>89.58</td><td>10</td><td>54.16</td></tr>
  <tr><td>Calibri-COMET22-QE*</td><td>3</td><td>89.55</td><td>8</td><td>55.22</td></tr>
  <tr><td>YiSi-1</td><td>3</td><td>89.41</td><td>6</td><td>56.64</td></tr>
  <tr><td>XLsim</td><td>3</td><td>87.93</td><td>6</td><td>56.47</td></tr>
  <tr><td>XCOMET-Ensemble</td><td>4</td><td>87.77</td><td>4</td><td>57.82</td></tr>
  <tr><td>XCOMET-QE-Ensemble*</td><td>4</td><td>87.34</td><td>6</td><td>56.40</td></tr>
  <tr><td>eBLEU</td><td>4</td><td>87.33</td><td>10</td><td>53.91</td></tr>
  <tr><td>BERTscore</td><td>4</td><td>86.83</td><td>7</td><td>56.13</td></tr>
  <tr><td>f200spBLEU</td><td>4</td><td>86.82</td><td>8</td><td>54.97</td></tr>
  <tr><td>MaTSE</td><td>4</td><td>86.69</td><td>16</td><td>37.35</td></tr>
  <tr><td>MEE4</td><td>4</td><td>86.12</td><td>7</td><td>55.93</td></tr>
  <tr><td>BLEU</td><td>5</td><td>84.45</td><td>10</td><td>53.63</td></tr>
  <tr><td>tokengram_F</td><td>5</td><td>83.15</td><td>8</td><td>54.98</td></tr>
  <tr><td>chrF</td><td>5</td><td>82.45</td><td>9</td><td>54.74</td></tr>
  <tr><td>embed_llama</td><td>5</td><td>81.27</td><td>9</td><td>54.28</td></tr>
  <tr><td>Random-sysname*</td><td>6</td><td>60.55</td><td>12</td><td>47.94</td></tr>
  <tr><td>prismSrc*</td><td>7</td><td>28.59</td><td>11</td><td>48.54</td></tr>
</table>

Table 9: The test set is 2023 EN→DE. The evaluator selected as the ground truth follows the DA+SQM protocol (DA+SQM in Table 6). Different from Tables 2 and 6, we exclude the evaluators ESA-1, ESA-2, and MQM-2023-4, because they annotated a limited number of translations. This way, we increase the number of segments in the test set from 145 to 376.
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="2">SPA</th>
    <th colspan="2">acc<sup>eq</sup></th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>Acc.</th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr>
    <td><b>MQM-2023-3</b></td>
    <td>1</td>
    <td>97.09</td>
    <td>7</td>
    <td>56.62</td>
  </tr>
  <tr>
    <td>GEMBA-MQM*</td>
    <td>1</td>
    <td>97.09</td>
    <td>5</td>
    <td>59.18</td>
  </tr>
  <tr>
    <td>CometKiwi-XXL*</td>
    <td>1</td>
    <td>95.96</td>
    <td>7</td>
    <td>56.43</td>
  </tr>
  <tr>
    <td>CometKiwi-XL*</td>
    <td>1</td>
    <td>95.47</td>
    <td>6</td>
    <td>57.33</td>
  </tr>
  <tr>
    <td><b>MQM-2023-2</b></td>
    <td>1</td>
    <td>95.20</td>
    <td>4</td>
    <td>60.04</td>
  </tr>
  <tr>
    <td>docWMT22CometDA</td>
    <td>1</td>
    <td>94.99</td>
    <td>10</td>
    <td>53.86</td>
  </tr>
  <tr>
    <td>MetricX-23-XXL</td>
    <td>2</td>
    <td>94.93</td>
    <td>2</td>
    <td>61.65</td>
  </tr>
  <tr>
    <td>XCOMET-Ensemble</td>
    <td>2</td>
    <td>94.58</td>
    <td>1</td>
    <td>62.23</td>
  </tr>
  <tr>
    <td>MetricX-23-QE-XXL*</td>
    <td>2</td>
    <td>94.52</td>
    <td>3</td>
    <td>61.14</td>
  </tr>
  <tr>
    <td>COMET</td>
    <td>2</td>
    <td>94.39</td>
    <td>8</td>
    <td>55.71</td>
  </tr>
  <tr>
    <td>XCOMET-QE-Ensemble*</td>
    <td>2</td>
    <td>94.14</td>
    <td>4</td>
    <td>59.89</td>
  </tr>
  <tr>
    <td>docWMT22CometKiwiDA*</td>
    <td>2</td>
    <td>93.70</td>
    <td>10</td>
    <td>53.55</td>
  </tr>
  <tr>
    <td>BLEURT-20</td>
    <td>2</td>
    <td>93.35</td>
    <td>9</td>
    <td>54.83</td>
  </tr>
  <tr>
    <td>Calibri-COMET22-QE*</td>
    <td>2</td>
    <td>93.25</td>
    <td>14</td>
    <td>49.62</td>
  </tr>
  <tr>
    <td>cometoid22-wmt22*</td>
    <td>2</td>
    <td>92.43</td>
    <td>7</td>
    <td>56.73</td>
  </tr>
  <tr>
    <td>CometKiwi*</td>
    <td>3</td>
    <td>92.27</td>
    <td>7</td>
    <td>56.76</td>
  </tr>
  <tr>
    <td><b>DA+SQM</b></td>
    <td>3</td>
    <td>92.10</td>
    <td>18</td>
    <td>43.68</td>
  </tr>
  <tr>
    <td>KG-BERTScore*</td>
    <td>3</td>
    <td>92.05</td>
    <td>9</td>
    <td>54.40</td>
  </tr>
  <tr>
    <td>sescoreX</td>
    <td>3</td>
    <td>91.99</td>
    <td>10</td>
    <td>53.99</td>
  </tr>
  <tr>
    <td>YiSi-1</td>
    <td>3</td>
    <td>91.28</td>
    <td>11</td>
    <td>51.66</td>
  </tr>
  <tr>
    <td>mbr-metricx-qe*</td>
    <td>3</td>
    <td>91.22</td>
    <td>8</td>
    <td>56.19</td>
  </tr>
  <tr>
    <td>MS-COMET-QE-22*</td>
    <td>3</td>
    <td>90.70</td>
    <td>10</td>
    <td>53.51</td>
  </tr>
  <tr>
    <td>prismRef</td>
    <td>3</td>
    <td>90.22</td>
    <td>14</td>
    <td>49.74</td>
  </tr>
  <tr>
    <td>Calibri-COMET22</td>
    <td>3</td>
    <td>88.28</td>
    <td>14</td>
    <td>49.54</td>
  </tr>
  <tr>
    <td>XLSim</td>
    <td>4</td>
    <td>88.01</td>
    <td>13</td>
    <td>50.31</td>
  </tr>
  <tr>
    <td>mre-score-labse-regular</td>
    <td>4</td>
    <td>87.47</td>
    <td>13</td>
    <td>49.85</td>
  </tr>
  <tr>
    <td>BERTscore</td>
    <td>4</td>
    <td>87.01</td>
    <td>12</td>
    <td>50.48</td>
  </tr>
  <tr>
    <td>f200spBLEU</td>
    <td>4</td>
    <td>86.90</td>
    <td>13</td>
    <td>50.37</td>
  </tr>
  <tr>
    <td>MaTESe</td>
    <td>4</td>
    <td>86.73</td>
    <td>7</td>
    <td>56.23</td>
  </tr>
  <tr>
    <td>eBLEU</td>
    <td>4</td>
    <td>86.24</td>
    <td>16</td>
    <td>48.30</td>
  </tr>
  <tr>
    <td>MEE4</td>
    <td>4</td>
    <td>86.05</td>
    <td>12</td>
    <td>50.81</td>
  </tr>
  <tr>
    <td>BLEU</td>
    <td>5</td>
    <td>84.48</td>
    <td>15</td>
    <td>49.18</td>
  </tr>
  <tr>
    <td>tokengram_F</td>
    <td>5</td>
    <td>83.85</td>
    <td>14</td>
    <td>49.51</td>
  </tr>
  <tr>
    <td>chrF</td>
    <td>5</td>
    <td>83.34</td>
    <td>14</td>
    <td>49.44</td>
  </tr>
  <tr>
    <td>embed_llama</td>
    <td>5</td>
    <td>80.09</td>
    <td>17</td>
    <td>45.05</td>
  </tr>
  <tr>
    <td>Random-sysname*</td>
    <td>6</td>
    <td>61.26</td>
    <td>20</td>
    <td>38.19</td>
  </tr>
  <tr>
    <td>prismSrc*</td>
    <td>7</td>
    <td>28.88</td>
    <td>19</td>
    <td>40.02</td>
  </tr>
</table>

Table 10: The test set is 2023 EN→DE. The evaluator selected as the ground truth follows the MQM protocol (it is the evaluator selected as ground truth in Table 6). Different from Tables 2 and 6, we exclude the evaluators ESA-1, ESA-2, and MQM-2023-4, because they annotated a limited number of translations. This way, we increase the number of segments in the test set from 145 to 376.
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="2">SPA</th>
    <th colspan="2">acc<sup>eq</sup></th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>Acc.</th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr><td>BLEURT-20</td><td>1</td><td>97.12</td><td>6</td><td>60.66</td></tr>
  <tr><td>MetricX-23-XXL</td><td>1</td><td>96.51</td><td>1</td><td>64.61</td></tr>
  <tr><td>docWMT22CometDA</td><td>1</td><td>96.41</td><td>8</td><td>59.16</td></tr>
  <tr><td>CometKiwi-XXL*</td><td>1</td><td>96.40</td><td>4</td><td>61.57</td></tr>
  <tr><td>GEMBA-MQM*</td><td>1</td><td>96.19</td><td>8</td><td>58.96</td></tr>
  <tr><td>CometKiwi-XL*</td><td>1</td><td>95.88</td><td>6</td><td>60.45</td></tr>
  <tr><td>COMET</td><td>1</td><td>95.82</td><td>5</td><td>61.31</td></tr>
  <tr><td style="background-color:#e0e0e0;">MQM-2023-3</td><td>1</td><td>95.50</td><td>8</td><td>59.25</td></tr>
  <tr><td style="background-color:#e0e0e0;">MQM-2023-1</td><td>1</td><td>95.20</td><td>7</td><td>60.03</td></tr>
  <tr><td>mbr-metricx-qe*</td><td>1</td><td>95.12</td><td>3</td><td>62.39</td></tr>
  <tr><td>MetricX-23-QE-XXL*</td><td>2</td><td>94.99</td><td>2</td><td>63.51</td></tr>
  <tr><td>Calibri-COMET22-QE*</td><td>2</td><td>94.48</td><td>15</td><td>51.95</td></tr>
  <tr><td>docWMT22CometKiwiDA*</td><td>2</td><td>94.47</td><td>9</td><td>57.58</td></tr>
  <tr><td>XCOMET-Ensemble</td><td>2</td><td>93.81</td><td>1</td><td>64.20</td></tr>
  <tr><td>sescoreX</td><td>2</td><td>93.51</td><td>7</td><td>59.94</td></tr>
  <tr><td>CometKiwi*</td><td>2</td><td>93.04</td><td>6</td><td>60.24</td></tr>
  <tr><td>KG-BERTScore*</td><td>2</td><td>92.78</td><td>8</td><td>59.06</td></tr>
  <tr><td>XCOMET-QE-Ensemble*</td><td>3</td><td>92.71</td><td>4</td><td>61.92</td></tr>
  <tr><td style="background-color:#e0e0e0;">DA+SQM</td><td>3</td><td>91.40</td><td>16</td><td>48.92</td></tr>
  <tr><td>MS-COMET-QE-22*</td><td>3</td><td>91.34</td><td>9</td><td>57.54</td></tr>
  <tr><td>cometoid22-wmt22*</td><td>3</td><td>91.22</td><td>5</td><td>61.07</td></tr>
  <tr><td>YiSi-1</td><td>3</td><td>90.47</td><td>9</td><td>57.50</td></tr>
  <tr><td>mre-score-labse-regular</td><td>3</td><td>90.02</td><td>10</td><td>56.50</td></tr>
  <tr><td>prismRef</td><td>3</td><td>89.92</td><td>12</td><td>55.16</td></tr>
  <tr><td>f200spBLEU</td><td>3</td><td>89.09</td><td>11</td><td>55.81</td></tr>
  <tr><td>XLsim</td><td>4</td><td>88.77</td><td>11</td><td>55.49</td></tr>
  <tr><td>Calibri-COMET22</td><td>4</td><td>88.37</td><td>11</td><td>55.67</td></tr>
  <tr><td>eBLEU</td><td>4</td><td>88.19</td><td>14</td><td>54.13</td></tr>
  <tr><td>BERTscore</td><td>4</td><td>88.09</td><td>10</td><td>56.27</td></tr>
  <tr><td>BLEU</td><td>4</td><td>87.09</td><td>13</td><td>54.61</td></tr>
  <tr><td>MaTESe</td><td>4</td><td>86.49</td><td>13</td><td>54.19</td></tr>
  <tr><td>MEE4</td><td>4</td><td>86.46</td><td>10</td><td>56.41</td></tr>
  <tr><td>tokengram_F</td><td>4</td><td>86.12</td><td>11</td><td>55.66</td></tr>
  <tr><td>chrF</td><td>4</td><td>85.53</td><td>11</td><td>55.50</td></tr>
  <tr><td>embed_llama</td><td>5</td><td>80.95</td><td>15</td><td>51.39</td></tr>
  <tr><td>Random-sysname*</td><td>6</td><td>59.64</td><td>18</td><td>42.43</td></tr>
  <tr><td>prismSrc*</td><td>7</td><td>25.26</td><td>17</td><td>43.73</td></tr>
</table>

Table 11: The test set is 2023 EN→DE. The evaluator selected as the ground truth follows the MQM protocol (MQM-2023-2 in Table 6). Different from Tables 2 and 6, we exclude the evaluators ESA-1, ESA-2, and MQM-2023-4, because they annotated a limited number of translations. This way, we increase the number of segments in the test set from 145 to 376.
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="2">SPA</th>
    <th colspan="2">acc<sup>eq</sup></th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>Acc.</th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr><td>GEMBA-MQM*</td><td>1</td><td>97.98</td><td>7</td><td>55.71</td></tr>
  <tr><td>CometKiwi-XXL*</td><td>1</td><td>97.96</td><td>4</td><td>58.36</td></tr>
  <tr><td>CometKiwi-XL*</td><td>1</td><td>97.49</td><td>5</td><td>57.78</td></tr>
  <tr><td><b>MQM-2023-1</b></td><td>1</td><td>97.09</td><td>6</td><td>56.46</td></tr>
  <tr><td>MetricX-23-XXL</td><td>1</td><td>96.86</td><td>1</td><td>61.26</td></tr>
  <tr><td>docWMT22CometDA</td><td>1</td><td>96.82</td><td>6</td><td>56.98</td></tr>
  <tr><td>MetricX-23-QE-XXL*</td><td>1</td><td>96.47</td><td>2</td><td>60.55</td></tr>
  <tr><td>COMET</td><td>2</td><td>96.30</td><td>4</td><td>58.63</td></tr>
  <tr><td>docWMT22CometKiwiDA*</td><td>2</td><td>96.09</td><td>8</td><td>54.86</td></tr>
  <tr><td><b>MQM-2023-2</b></td><td>2</td><td>95.50</td><td>3</td><td>59.25</td></tr>
  <tr><td>CometKiwi*</td><td>2</td><td>94.67</td><td>5</td><td>57.83</td></tr>
  <tr><td>KG-BERTScore*</td><td>2</td><td>94.45</td><td>6</td><td>56.67</td></tr>
  <tr><td>sescoreX</td><td>2</td><td>94.31</td><td>5</td><td>57.58</td></tr>
  <tr><td><b>DA+SQM</b></td><td>2</td><td>94.24</td><td>14</td><td>46.99</td></tr>
  <tr><td>BLEURT-20</td><td>3</td><td>94.18</td><td>4</td><td>58.68</td></tr>
  <tr><td>Calibri-COMET22-QE*</td><td>3</td><td>94.01</td><td>13</td><td>49.98</td></tr>
  <tr><td>YiSi-1</td><td>3</td><td>93.19</td><td>7</td><td>55.64</td></tr>
  <tr><td>MS-COMET-QE-22*</td><td>3</td><td>93.17</td><td>7</td><td>55.46</td></tr>
  <tr><td>mbr-metricx-qe*</td><td>3</td><td>93.06</td><td>3</td><td>59.30</td></tr>
  <tr><td>XCOMET-Ensemble</td><td>3</td><td>92.64</td><td>2</td><td>60.61</td></tr>
  <tr><td>prismRef</td><td>3</td><td>92.58</td><td>10</td><td>53.57</td></tr>
  <tr><td>cometoid22-wmt22*</td><td>3</td><td>92.55</td><td>4</td><td>58.50</td></tr>
  <tr><td>XCOMET-QE-Ensemble*</td><td>3</td><td>92.14</td><td>3</td><td>59.43</td></tr>
  <tr><td>Calibri-COMET22</td><td>4</td><td>90.86</td><td>10</td><td>53.18</td></tr>
  <tr><td>XLSim</td><td>4</td><td>90.86</td><td>8</td><td>54.63</td></tr>
  <tr><td>BERTscore</td><td>4</td><td>89.67</td><td>9</td><td>54.30</td></tr>
  <tr><td>f200spBLEU</td><td>4</td><td>89.56</td><td>9</td><td>54.06</td></tr>
  <tr><td>mre-score-labse-regular</td><td>4</td><td>89.14</td><td>8</td><td>54.47</td></tr>
  <tr><td>MEE4</td><td>4</td><td>88.72</td><td>8</td><td>54.40</td></tr>
  <tr><td>eBLEU</td><td>4</td><td>88.67</td><td>11</td><td>53.06</td></tr>
  <tr><td>BLEU</td><td>5</td><td>87.13</td><td>11</td><td>52.76</td></tr>
  <tr><td>tokengram_F</td><td>5</td><td>86.17</td><td>10</td><td>53.62</td></tr>
  <tr><td>chrF</td><td>5</td><td>85.58</td><td>10</td><td>53.51</td></tr>
  <tr><td>MaTSESe</td><td>5</td><td>84.75</td><td>12</td><td>52.12</td></tr>
  <tr><td>embed_llama</td><td>5</td><td>82.93</td><td>13</td><td>50.10</td></tr>
  <tr><td>Random-sysname*</td><td>6</td><td>58.48</td><td>16</td><td>41.27</td></tr>
  <tr><td>prismSrc*</td><td>7</td><td>28.67</td><td>15</td><td>44.52</td></tr>
</table>

Table 12: The test set is 2023 EN→DE. The evaluator selected as the ground truth follows the MQM protocol (MQM-2023-3 in Table 6). Different from Tables 2 and 6, we exclude the evaluators ESA-1, ESA-2, and MQM-2023-4, because they annotated a limited number of translations. This way, we increase the number of segments in the test set from 145 to 376.
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="2">SPA</th>
    <th colspan="2">EN→ES acc<sub>eq</sub></th>
  </tr>
  <tr>
    <th>Rank</th>
    <th>Acc.</th>
    <th>Rank</th>
    <th>Acc.</th>
  </tr>
  <tr><td>COMET-22</td><td>1</td><td>86.90</td><td>2</td><td>53.11</td></tr>
  <tr><td>BLCOM_1</td><td>1</td><td>86.12</td><td>2</td><td>53.00</td></tr>
  <tr><td>XCOMET</td><td>1</td><td>84.88</td><td>3</td><td>52.35</td></tr>
  <tr><td>metametrics_mt_mqm_hybrid_kendall</td><td>1</td><td>84.80</td><td>1</td><td>53.92</td></tr>
  <tr><td>XCOMET-QE*</td><td>1</td><td>83.67</td><td>4</td><td>51.01</td></tr>
  <tr><td>PrismRefMedium</td><td>1</td><td>83.07</td><td>5</td><td>50.62</td></tr>
  <tr><td>MetricX-24-Hybrid</td><td>2</td><td>82.96</td><td>2</td><td>53.13</td></tr>
  <tr><td>BLEURT-20</td><td>2</td><td>82.19</td><td>2</td><td>52.93</td></tr>
  <tr><td>gemb_a_esa*</td><td>2</td><td>81.37</td><td>10</td><td>40.86</td></tr>
  <tr><td>MetricX-24-Hybrid-QE*</td><td>2</td><td>80.93</td><td>4</td><td>51.35</td></tr>
  <tr><td>PrismRefSmall</td><td>2</td><td>80.84</td><td>4</td><td>51.23</td></tr>
  <tr><td><b>MQM-2024</b></td><td><b>2</b></td><td><b>80.13</b></td><td><b>11</b></td><td><b>34.61</b></td></tr>
  <tr><td>sentinel-cand-mqm*</td><td>2</td><td>79.06</td><td>6</td><td>50.23</td></tr>
  <tr><td>YiSi-1</td><td>2</td><td>78.76</td><td>3</td><td>51.88</td></tr>
  <tr><td>BERTScore</td><td>2</td><td>78.28</td><td>4</td><td>50.88</td></tr>
  <tr><td>metametrics_mt_mqm_qe_kendall.seg.s*</td><td>3</td><td>77.05</td><td>7</td><td>49.17</td></tr>
  <tr><td>CometKiwi-XXL*</td><td>3</td><td>76.51</td><td>5</td><td>50.86</td></tr>
  <tr><td>bright-qe*</td><td>3</td><td>75.73</td><td>9</td><td>43.46</td></tr>
  <tr><td>MEE4</td><td>3</td><td>75.20</td><td>4</td><td>51.03</td></tr>
  <tr><td>CometKiwi*</td><td>3</td><td>74.39</td><td>5</td><td>50.47</td></tr>
  <tr><td>chrFS</td><td>3</td><td>73.90</td><td>4</td><td>51.05</td></tr>
  <tr><td>chrF</td><td>4</td><td>70.94</td><td>5</td><td>50.69</td></tr>
  <tr><td>spBLEU</td><td>4</td><td>70.77</td><td>6</td><td>49.81</td></tr>
  <tr><td>BLEU</td><td>4</td><td>69.37</td><td>7</td><td>49.43</td></tr>
  <tr><td>damonmonli</td><td>4</td><td>63.34</td><td>8</td><td>48.14</td></tr>
  <tr><td>sentinel-ref-mqm</td><td>5</td><td>54.36</td><td>12</td><td>15.38</td></tr>
  <tr><td>sentinel-src-mqm*</td><td>5</td><td>54.36</td><td>12</td><td>15.38</td></tr>
  <tr><td>XLsimMqm*</td><td>5</td><td>39.32</td><td>9</td><td>43.02</td></tr>
</table>

Table 13: The test set is 2024. The evaluator selected as the ground truth follows the ESA protocol (ESA in Table 7).

Reviewer paper summaries: This study examines whether automated metrics can perform as well as human evaluators in machine translation. Researchers compared machine systems with human evaluators using datasets from the WMT Metrics Shared Tasks (2020–2024) and several human annotation methods like MQM, ESA, and SQM. With strategies such as Soft Pairwise Accuracy (SPA, system-level) and Pairwise Accuracy with Tie Calibration (segment-level), they found that top metrics (e.g., MetricX-23-QE-XXL, BLEURT-20) often score as well as or even better than human evaluators when using non-MQM methods. However, the study warns that this does not fully prove "human parity" due to issues like inconsistent human judgments, simple test scenarios, and potential biases in the evaluation methods.
Summary of strengths: 1. The paper integrates human baselines into metric rankings, showing that automatic metrics can sometimes match or exceed human evaluation.  
2. It also discusses the limits of current benchmarks and offers ideas for future improvements in MT evaluation.
Summary of weaknesses: 1. One limitation of this paper is that most automatic metrics are only evaluated on a single test set. It would be more convincing to assess each metric across multiple test sets to better understand their generalization abilities. Evaluating on just one WMT year leaves open the possibility that model-based metrics may be overfitting, because they can be well fine-tuned on the provided validation set.

2. Another weakness is the lack of in-depth analysis on why human evaluations underperform. Including some qualitative or quantitative analysis on annotation quality, inter-annotator agreement, or task difficulty would strengthen the credibility of the conclusions.

3. The paper does not provide a clear discussion on the performance of LLM-based metrics. Given the recent shift toward LLMs in evaluation, it would be helpful to analyze how they compare against traditional heuristic metrics (e.g., BLEU) and representation-learning metrics (e.g., COMET).
Comments, suggestions and typos: It’s unclear how the ground-truth (MQM) relates to evaluators like MQM-2020-2 or MQM-2020-3 (see Table 2). The paper doesn’t explain how these are built or chosen, which makes the setup and results hard to understand.
Overall assessment scores: 2.5
Metareview score: 4
Metareview: The metric contributes to the discussion of human and automatic evaluation of machine translation systems. It demonstrates that the leading automatic metrics can outperform human evaluation when measures with the same standard.
Summary of reasons to publish: The paper makes a concise contribution to the body of knowledge on human and automatic evaluation, specifically on machine translation. It makes an important discovery and provides valuable analysis.