Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In! (2024)

Abstract: Annually, at the Conference of Machine Translation (WMT), the Metrics Shared Task organizers conduct the meta-evaluation of Machine Translation (MT) metrics, ranking them according to their correlation with human judgments. Their results guide researchers toward enhancing the next generation of metrics and MT systems. With the recent introduction of neural metrics, the field has witnessed notable advancements. Nevertheless, the inherent opacity of these metrics has posed substantial challenges to the meta-evaluation process. This work highlights two issues with the metaevaluation framework currently employed in WMT, and assesses their impact on the metrics rankings. To do this, we introduce the concept of sentinel metrics, which are designed explicitly to scrutinize the meta-evaluation process’s accuracy, robustness, and fairness. By employing sentinel metrics, we aim to validate our findings, and shed light on and monitor the potential biases or inconsistencies in the rankings. We discover that the present metaevaluation framework favors two categories of metrics: i) those explicitly trained to mimic human quality assessments, and ii) continuous metrics. Finally, we raise concerns regarding the evaluation capabilities of state-of-the-art metrics, emphasizing that they might be basing their assessments on spurious correlations found in their training data.

heading: 1 Introduction
text: Over the past few years, the Machine Translation (MT) field has witnessed significant advancements, largely driven by the advent of neural architectures, with the Transformer (Vaswani et al., 2017) being the most notable. Modern MT systems deliver mostly fluent and accurate translations, posing a challenge for their quality evaluation – even when conducted by human annotators, especially those who lack professional training (Freitag
*Equal contribution.
et al., 2021a). Under these circumstances, shallow overlap-based metrics are gradually being replaced by neural-based metrics, which demonstrate a better correlation with human judgments (Freitag et al., 2022). However, a significant limitation is that most neural-based metrics are black-box systems trained to predict human judgments in the form of scalar scores, and typically do not provide justifications for their assessments. Besides rendering them challenging to interpret, such opacity also complicates their meta-evaluation. In this respect, we found that certain strategies for the assessment of MT metrics’ capabilities – which have recently been employed in the context of the Metrics Shared Task at the Conference on Machine Translation (WMT)1 – favor specific metric categories and potentially encourage undesirable metrics behavior. To demonstrate these problems, we introduce the concept of sentinel metrics, i.e., a suite of metrics serving as a probe to identify pitfalls in the meta-evaluation process. Sentinel metrics are either trained with incomplete information – which makes them inherently unable to evaluate the quality of machine-translated text properly – or consist of variations of existing metrics – which have been devised to expose specific issues in the meta-evaluation.
As an example, in Table 1, we present the segment-level ranking of WMT23 with the inclusion of a sentinel metric. As can be seen, SENTINELCAND ranks in the upper half. SENTINELCAND is a sentinel metric designed to assess the quality of a candidate translation based solely on the translation itself, without accessing its source sentence or any reference translation. Arguably, such a metric should only be capable of evaluating a translation’s fluency, but not its ad-
1With its first edition in 2006 (Koehn and Monz, 2006), "WMT is the main event for machine translation and machine translation research." (https://machinetranslate. org/wmt).
16216
equacy in conveying the original message, and a fair assessment should rank it at lower positions. Notably, SENTINELCAND is above strong baselines such as COMET (Rei et al., 2020) and BLEURT20 (Sellam et al., 2020), suggesting that there might be some issues with the segment-level metaevaluation methods used in WMT23.
In this work, we: i) illustrate the issues that affect the segment-level meta-evaluation measures
used in WMT23, demonstrating their impact experimentally with the help of sentinel metrics; ii) propose solutions for addressing these issues; iii) raise concerns regarding the reliability of state-ofthe-art MT metrics. We publish the code to reproduce our work and the weights of the sentinel metrics at https://github.com/SapienzaNLP/ guardians-mt-eval.

heading: 2 The Meta-evaluation of MT Metrics
text: Yearly, the WMT Metrics Shared Task organizes a competition among metrics, including participants’ submissions and baselines, to identify the metric that most closely aligns with human judgments. Historically, the organizers have employed correlation with human judgment as a meta-evaluation strategy. Recently, significant efforts have been made to refine the meta-evaluation process, encompassing the adoption of new measures, such as those proposed by Kocmi et al. (2021) and Deutsch et al. (2023), and the introduction of the challenge sets sub-task (Freitag et al., 2021b, 2022), among other initiatives. In this section, we provide an overview of WMT’s official meta-evaluation setting.
First, multiple MT systems are employed to translate source segments found in one or more test datasets.2 Consequently, test datasets contain several translations of the same source segment. Second, a manual evaluation campaign is carried out to assess the quality of all translations. Finally, metrics’ capabilities are assessed based on their alignment with human judgments, which are in the form of scalar scores. Such alignment is typically estimated using correlation and accuracy measures. Specifically, metrics are evaluated at two granularity levels:
• at the segment level, metrics assign a score to every translation, and they are ranked according to their ability to discern between higherand lower-quality translations;
• at the system level, metrics assign a score to each MT system,3 and they are ranked according to their ability to discern between superior and inferior systems.
2A segment typically refers to a single sentence, but can also include multiple sentences. For instance, at WMT23, a segment represents an entire paragraph rather than a single sentence for the English-to-German translation direction.
3Typically, the score of a system is calculated as the mean of the scores given to its translations.
At both granularity levels, metrics can be evaluated using several statistical methods, such as the Kendall τ and Pearson ρ correlation coefficients, which have traditionally been applied at the segment and system levels, respectively. A final metrics ranking is derived by aggregating results from all the chosen statistics. For example, at WMT23, the final ranking was computed from the following three statistics:
1. System-level pairwise ranking accuracy (Kocmi et al., 2021), which evaluates metrics based on their ability to rank systems in the same order as human judgments.
2. System- and segment-level Pearson correlation, which measures the degree to which metric scores and human scores are correlated linearly.
3. Segment-level pairwise ranking accuracy with tie calibration (Deutsch et al., 2023), which evaluates metrics based on their ability to rank segments in the same order as human judgments, or their ability to predict ties correctly.
In this work, we identify two critical issues related to the second and third statistics, and provide the following recommendations to address them:
• Translations should be grouped by their source segment before calculating segmentlevel correlations (Section 3).
• Tie calibration should not be conducted on the test set (Section 4).
In the following two sections, we provide an overview of some of the aforementioned statistics, illustrate their flaws, and demonstrate their impact by leveraging our sentinel metrics.

heading: 3 To Group or Not to Group?
text: At early editions of the WMT Metrics Shared Task (Macháček and Bojar, 2013, 2014; Stanojević et al., 2015; Bojar et al., 2016), human assessments were collected in the form of Relative Rankings (RR). Specifically, the annotators were tasked to rank up to 5 translations of the same source sentence, produced by different MT systems. From each ranking, up to 10 pairwise comparisons were extracted. Despite metrics assessments being scalar scores – which theoretically enabled the comparison of all pairs of translated segments – correlation
was measured only on those pairs of translations for which RR annotations were available. Therefore, only translations of the same source sentence were compared. Later on, at subsequent editions of WMT, new techniques for human evaluation were adopted: first, Direct Assessments (Graham et al., 2013, DA) – where annotators rate individual translations on a scale from 0 to 100 – then, Multidimensional Quality Metrics (Lommel et al., 2014, MQM) – where annotators tag the spans of a translation that contain errors, specifying their category and severity. With both the new annotation schemas, each translated segment was assigned a scalar quality score independently of the other translations,4 which made it possible to compare all translations, not only those of the same source sentence. This new possibility raised doubts regarding the best way to compute the correlation between metrics and human assessments. Indeed, it could be computed using all translations at once – No Grouping – or by first grouping translations based on either their source segment – Segment Grouping – or the system that produced them – System Grouping – and then returning the average correlation of these groups.
At the WMT21 Metrics Shared Task, Freitag et al. (2021b) chose the No Grouping strategy, arguing that the other options would provide only a partial view of the overall picture. At WMT22, all three grouping strategies were used (Freitag et al., 2022), and later at WMT23, Freitag et al. (2023) chose No Grouping again. Although No Grouping is the only strategy that assesses the MT metrics’ ability to discern between higher- and lower-quality translations in absolute terms, irrespective of the source segment or MT system, we show that both No Grouping and System Grouping may introduce unfairness and favor trained metrics over the rest.

heading: 3.1 The Relation Between Spurious Correlations and Grouping Strategies
text: Most neural-based metrics are trained with a regression objective to approximate human judgments. They are expected to infer by pattern-matching the relation between human judgments and various phenomena, such as omissions, additions, or other translation errors. However, this mechanism might inadvertently lead to the detection of patterns that are not in a causal relation with the concept of
4In MQM, a final score is obtained by applying a specific weighting to each combination of the detected spans’ category and severity.
translation quality, but are instead spurious correlations, e.g., the length of a translation, or the number of named entities in it, among others. Arguably, the meta-evaluation should not reward metrics for basing their assessments on spurious correlations between the features of the source, translation, or reference, and the human judgments. However, our intuition is that No Grouping and System Grouping strategies might be doing so by allowing the comparison of translations from different sources. To simplify, consider a metric that unfairly penalizes a translation solely because it contains many named entities. Using No Grouping or System Grouping, such a metric might have a non-negative correlation with human judgments if, on average, translating sentences containing many named entities is more challenging than translating other sentences, because MT systems would be making more mistakes in translating them. Therefore, exploiting such a pattern might be beneficial even though it is not causally related to the quality of a translation. In contrast, when using Segment Grouping, such a pattern would be ineffective, as different translations of the same source sentence should contain the same amount of named entities. More generally, we would expect Segment Grouping to lessen the impact of most spurious correlations derived from features shared by a source sentence and its translations.
To assess the extent of this issue, we incorporate three sentinel metrics into the current metaevaluation framework and re-compute the metrics’ rankings using all grouping strategies. Crucially, we find that the impact of spurious correlations when No Grouping and System Grouping strategies are employed is substantial – favoring trained metrics over the rest5 – and is significantly reduced with Segment Grouping.

heading: 3.2 The Sentinel Metrics
text: This section describes the three sentinel metrics employed to measure the impact of grouping strategies on the meta-evaluation process:
1. SENTINELCAND, which assesses the quality of a translation without taking its source or reference as input.
5Indeed, overlap-based metrics such as BLEU (Papineni et al., 2002) and chrF (Popović, 2015), or LLM-based metrics such as GEMBA-MQM (Kocmi and Federmann, 2023), were not trained to mimic human assessments and should not be able to leverage spurious correlations.
2. SENTINELSRC, which predicts the quality of a translation solely based on its source.
3. SENTINELREF, which predicts the quality of a translation solely based on its reference.
Having no information regarding the translation to evaluate, SENTINELSRC and SENTINELREF can only learn spurious correlations between the features of the source and reference sentences, respectively, and the human judgments. SENTINELCAND, instead, is a metric with partial information. Indeed, it is possible to evaluate a translation’s fluency and grammatical correctness without comparing it with its source or reference sentences, but not its adequacy. Nonetheless, we expect SENTINELCAND to base its assessments on spurious correlations also.

heading: 3.3 Experimental Setup
text: Sentinel metrics employ XLM-RoBERTa large (Conneau et al., 2020) as their backbone model, with a multi-layer fully-connected neural network on top of the [CLS] token, which is used to output predictions in the form of scalar scores. We train sentinel metrics to minimize the Mean Squared Error (MSE) between their predicted scores and human judgments. Our dataset comprises a selection of data from WMT spanning 2017 to 2022, incorporating Direct Assessments (DA) and Multidimensional Quality Metrics (MQM) scores. Following Rei et al. (2022a), we train sentinel metrics for a single epoch using DA from 2017 to 2020 and finetune them for a further epoch using MQM data. Additional details regarding the training process are reported in Appendix B.

heading: 3.4 Results
text: In Table 2, we report the ranking derived from the segment-level Pearson correlation of the primary submissions to the Metrics Shared Task of WMT23, with the inclusion of sentinel metrics, in the language direction ZH → EN, and with all three grouping strategies. We report in Appendix C the rankings alongside the correlation values for all the official translation directions of the Metrics Shared Task, i.e., ZH → EN, EN → DE and HE → EN. As can be seen, SENTINELSRC ranks fourth and third when the grouping strategies are No Grouping and System Grouping, respectively, surpassing strong baselines like COMET or BLEURT20, and even state-of-the-art metrics like GEMBAMQM. The only metrics that are not surpassed are
large regression-based systems such as XCOMETEnsemble (Guerreiro et al., 2023) and MetricX-23 (Juraska et al., 2023), which might have learned the same spurious correlations leveraged by the sentinel metrics, in addition to non-spurious patterns (cf. Section 3.4.1). Conversely, when grouping by segment, SENTINELSRC and SENTINELREF are correctly positioned at the bottom of the ranking,6 and SENTINELCAND ranks 11th, compared to 3rd and 2nd with No Grouping and System Grouping, respectively. A notable difference between the grouping strategies is the positioning of GEMBA-MQM, which is ranked 7th and 9th with No Grouping and System Grouping, respectively, and becomes first with Segment Grouping. We hypothesize that this is due to GEMBA-MQM being based on GPT-4, which has not been explicitly fine-tuned on human assessments and is less likely to leverage spurious correlations such as those described in Section 3.1. Interestingly, with grouping strategies other than Segment Grouping, GEMBA-MQM is surpassed by all the sentinel metrics.
SENTINELCAND is the only sentinel metric that does not rank at the very bottom with Segment Grouping, outperforming prismSrc (Thompson and Post, 2020) and embed_llama (Dreano et al., 2023), and positioning itself within the same cluster of statistical significance as BLEU. This suggests that focusing solely on the candidate translation – specifically, its fluency and grammatical correctness – may be sufficient to exceed the performance of some less effective metrics, at least in terms of Pearson correlation with human judgments. Furthermore, we highlight that our results may provide an answer to the open question left at WMT23 regarding the inconsistency of segment-level and system-level correlations for prismSrc. Freitag et al. (2023) noticed that, despite displaying a moderate correlation at the segment level, prismSrc was showing negative correlation values at the system level. As can be seen from Table 2, prismSrc ranks 15th out of 24 with No Grouping but 13th out of 14 with Segment Grouping (i.e., it is in the second to last significance cluster, close to the sentinel metrics). This result is consistent with prismSrc’s negative correlation at the system level.
In Appendix C, we also report the rankings and correlations obtained using the Kendall τ correlation coefficient for each grouping strategy, to show
6This had to be expected, given that both these metrics return the same assessment for all translations of the same source segment.
that our findings are independent of the correlation measure, at least among those typically employed at WMT, i.e., Pearson ρ and Kendall τ .

heading: 3.4.1 Are MT metrics learning from spurious correlations?
text: We hypothesize that some of the trained metrics may be basing their assessments on the same spurious correlations as those leveraged by the sentinel metrics. To delve deeper into this, we measure their segment-level Pearson correlation with the sentinel metrics using No Grouping. Surprisingly, XCOMET-Ensemble, XCOMETQE-Ensemble, MetricX-23, and MetricX-23-QE, which are the only metrics that surpass the sentinels in Table 2, display a high correlation with
all three sentinel metrics. Interestingly, their correlation with SENTINELSRC is 0.750, 0.736, 0.690, and 0.712 (Figure 4), respectively, while their correlation with human judgment is 0.650, 0.647, 0.625, and 0.647, respectively (Table 5). We recognize that these metrics share many similarities with our sentinels, as both are neural transformerbased systems and both were trained with the same regression-based objective, using largely the same data. This similarity likely contributes to the high correlation values observed. However, with access limited to only the source segment, SENTINELSRC relies exclusively on spurious correlations to conduct the evaluation. For this reason, we argue that these results raise concerns about the reliability of state-of-the-art MT metrics, which may be learning to exploit spurious correlations to minimize the Mean Squared Error with human judgments during training. To further support our hypothesis, we plot in Figure 1 the relation between the assessments of XCOMET-Ensemble and translation length, which serves as a simple spurious correlate of translation quality.7 We also plot the distribution of MQM human judgments over translation length. As we can see from the figure, XCOMET-Ensemble scores decrease at increasing candidate lengths, with the metric almost never assigning scores higher than 0.9 to translations longer than 400 characters. However, the distribution of human judgments shows that human annotators rated many of those translations as perfect or near-perfect, indicating that XCOMETEnsemble might be biased to assign lower scores to longer translations, irrespective of their quality. Furthermore, the least-squares regression lines show that, on average, and as expected, longer translations contain more errors than shorter ones, and therefore are assigned lower scores by human annotators. This suggests that detecting biases of this type might be particularly complex without datasets crafted specifically for it.
We leave the investigation of these phenomena to future work and, for further details, we direct readers to Appendix D, where we report the pairwise correlation between most of the considered metrics and sentinel metrics, and to appendix E, where we report the relation between such metrics’ assessments and translation length.
7We expect that longer sentences are, on average, more challenging to translate. Therefore, we anticipate that MT metrics might have learned to assign lower scores to longer translations, despite the length and quality of translations not being causally related.

heading: 4 The Evaluation of Ties
text: In this Section, we focus on the third statistic among those described in Section 2, i.e., the segment-level pairwise ranking accuracy with tie calibration, dubbed acceq by Deutsch et al. (2023). Prior to WMT23, the organizers of the Metrics Shared Task used to employ the Kendall τ coefficient – which is a statistic used to estimate the rank-based agreement between two sets of measurements (Kendall, 1945) – to measure the correlation between metrics and human judgments at the segment level. Deutsch et al. (2023) pointed out that the Kendall τ coefficient does not account for metrics correctly predicting ties,8 and introduced acceq to address this issue. Unfortunately, our analysis indicates that acceq inadvertently compromises evaluation fairness in order to accommodate ties, ultimately biasing the results in favor of continuous metrics9 over discrete ones.
8Given a pair of translations whose quality has been assessed by human annotators, the pair is tied if both translations were assigned with the same score.
9By continuous, we refer to those metrics whose assessments can take on any value within a given range, as opposed

heading: 4.1 The Kendall τ
text: In this section, we define the Kendall τ coefficient as employed by the organizers of the Metrics Shared Task of WMT21 and WMT22.10 Let m,h be the vectors of metric and human assessments, respectively. Concordant pairs are the pairs of metric assessments that have been ranked in the same order by humans; discordant pairs are those ranked in a different order. We define C and D as the number of concordant and discordant pairs, respectively. We also define Th as the number of pairs only tied in the gold scores, Tm as the number of pairs only tied in the metric scores, and Thm as the number of pairs tied both in gold and metric scores, i.e., the number of correctly predicted ties. The Kendall τ correlation coefficient is defined as follows (Kendall, 1945):
τ = C −D√
(C +D + Th)(C +D + Tm) . (1)

heading: 4.2 The acceq
text: As noted by Deutsch et al. (2023), Kendall τ penalizes the prediction of ties, but never rewards them, as Tm and Th are in the denominator, and Thm is not used. This issue was not prominent in the earliest editions of the Metrics Shared Task, where ties in human scores were disregarded, and older metrics rarely produced ties. Currently, instead, it is essential to consider the prediction of ties, especially since human MQM annotations contain a lot of them,11 and some recently-proposed metrics are designed to output evaluation assessments that resemble MQM (Perrella et al., 2022; Kocmi and Federmann, 2023). For this reason, Deutsch et al. (2023) proposed a measure that mimics the τ coefficient in the way it is computed, but also accounts for correctly predicting ties:
acceq = C + Thm
C +D + Th + Tm + Thm . (2)
Differently from Kendall τ , acceq includes Thm in the numerator, and the denominator encompasses the total number of pairs. Notably, discordant pairs
to discrete metrics, which can take on a limited set of values. Metrics from the COMET family such as COMET, XCOMETEnsemble, and CometKiwi (Rei et al., 2022b) are continuous, whereas GEMBA-MQM (Kocmi and Federmann, 2023) and MaTESe (Perrella et al., 2022) are examples of discrete metrics.
10This is τb in Deutsch et al. (2023). 11This is also due to the increasing quality of automatic translation, as perfect translations are assigned the same maximum score.
are not subtracted from the numerator, rendering this metric a measure of accuracy, with scores ranging between 0 and 1. In Appendix F, we provide a numerical example of the computation of both Kendall τ and acceq from the vectors m and h.
The acceq measure, as it stands, would unfairly disadvantage continuous metrics. Indeed, it is extremely infrequent for such metrics to assign the same score to two different translations, meaning that they never predict ties. To address this issue, Deutsch et al. (2023) propose the tie calibration algorithm. In the following section, we briefly illustrate this algorithm and explain why it should not be conducted on the same test set used for the meta-evaluation.

heading: 4.3 Tie Calibration
text: The tie calibration algorithm determines, for each metric, a threshold ϵ such that, given two metric assessments m1 and m2, they are tied if |m1 − m2| ≤ ϵ. Deutsch et al. (2023) propose selecting the ϵ that maximizes acceq on the same test set used for the metrics meta-evaluation, enabling metrics to output the number of tied scores that best fits the distribution of human ties in the considered test set. This distribution is not stable across test sets (Table 11), and Deutsch et al. (2023) show that ϵ values are not stable either. Nonetheless, they argue that this would not impact the fairness of the evaluation. Unfortunately, our analysis shows that this is not the case. Specifically, despite all metrics’ ϵ values being selected on the same test data, we demonstrate that continuous metrics are more flexible to best fit the underlying distribution of human ties, compared to discrete ones, leading to unfairly higher acceq values.

heading: 4.4 Two New Sentinel Metrics
text: To demonstrate the impact of this phenomenon, we introduce two additional sentinel metrics, i.e., SENTINELGEMBA and SENTINELMATESE. GEMBAMQM (Kocmi and Federmann, 2023) and MaTESe (Perrella et al., 2022) are MT metrics that output discrete scores in the form of MQM quality assessments and participated in WMT23. SENTINELGEMBA and SENTINELMATESE are perturbed versions of GEMBA-MQM and MaTESe, respectively, obtained by adding Gaussian noise – N (0, 0.0001) – to their predictions. By making their output continuous in the neighborhood of discrete values, we partially fill their gap with continuous metrics, while preventing any two dif-
ferent discrete assessments from inverting their ordering. That is, if two GEMBA-MQM’s assessments m1,m2 are such that m1 > m2, this relation is preserved by SENTINELGEMBA. In general, we expect a fair meta-evaluation to rank these sentinels on par or below their discrete counterparts. Furthermore, we wish to remark that this solution is sub-optimal compared to metrics that are continuous by design. Indeed, due to the addition of Gaussian noise, the ordering of all SENTINELGEMBA and SENTINELMATESE’s assessments in the neighborhood of discrete values is randomized.
To demonstrate that SENTINELGEMBA and SENTINELMATESE can better fit the distribution of human ties compared to their discrete counterparts, we modify such a distribution in the test data. Specifically, we repeatedly sub-sample the test data, such that for each pair of tied human assessments we remove that pair from the test data with a certain probability pt, and do the same for non-tied pairs, which are removed with probability pn. We extract 13 samples by assigning various values to pt and pn and report the chosen values in Table 12 in Appendix G. As a consequence, each pair (pt, pn) represents a different sub-sample of test data, with a different percentage of tied human pairs. Then, for each metric, we select the best ϵ and compute acceq on each of these samples.

heading: 4.5 Results
text: In Figure 2 (left), we present the acceq results for a subset of continuous metrics, together with GEMBA-MQM, MaTESe, SENTINELGEMBA, and SENTINELMATESE. We discuss our results on the WMT23 ZH → EN test set, and report results concerning the other language directions, i.e., EN → DE and HE → EN, in Appendix G. At first glance, it is evident that discrete metrics exhibit a distinct acceq pattern compared to continuous and sentinel metrics. Notably, at lower percentages of tied human pairs, SENTINELGEMBA and SENTINELMATESE significantly outperform GEMBA-MQM and MaTESe.12 This discrepancy arises because the tie calibration algorithm selects very small ϵ values, close to 0 for every metric, allowing the number of ties predicted by continuous metrics to potentially drop to 0. Conversely, metrics that yield discrete scores inherently produce a certain number of ties, placing them at a disadvantage, and thus ranking conceptually identical metrics like SENTINELGEMBA and GEMBA-MQM at significantly different positions. Interestingly, in the hypothetical scenario in which there are no tied
12It is important to highlight that the range of human tie percentages explored in our analysis is similar to that found in the WMT test sets. Indeed, as shown in Table 11, such percentages range from a minimum of 15.14% to a maximum of 53.35%, observed in the WMT22 EN → DE test set.
human pairs in the dataset, SENTINELGEMBA would rank second (despite several of its assessments having a random ordering), whereas GEMBA-MQM would be second to last. At increasing percentages of gold ties, instead, the acceq values obtained by SENTINELGEMBA and SENTINELMATESE converge to those of their discrete counterparts. However, this is a limitation of these sentinels’ design and does not imply that the evaluation is fair at higher percentages of human ties.
To better investigate the source of unfairness, in Figure 2 (right) we show how the optimal ϵ changes at varying percentages of human ties. As can be seen, continuous metrics’ ϵ is dynamically adjusted with heightened sensitivity, contrary to what happens for discrete metrics. Specifically, their ϵ is exactly 0 until the percentage of human ties over all pairs is 39%. Additionally, for MaTESe, it remains constant between 44% and 56%, and between 61% and 68%, and the same happens for GEMBA-MQM between 47% and 51% and between 56% and 68%. In contrast, the values change for all the other metrics in the same intervals, enabling them to better fit the distribution of gold ties found in the test set.
4.5.1 Can we use a held-out set for tie calibration?
We have demonstrated that conducting the tie calibration on the same test set used for the evaluation favors continuous metrics over discrete ones. Nonetheless, this does not necessarily mean using a held-out dataset would ensure a fair metaevaluation. Indeed, our experiments show that unfairness stems from the different levels of adaptability between continuous and discrete metrics to the distribution of human ties found in the dataset used for tie calibration. Therefore, we expect that using a held-out dataset would still advantage continuous metrics if the distribution of human ties in the held-out resembled that of the test set, and disadvantage them if such a distribution differed from that of the test set. In both cases, continuous metrics’ increased adaptability compared to discrete metrics would impair the fairness of the evaluation. To investigate this further, we compute a 80-20 split of the test set to obtain an evaluation set for tie calibration. Then, we repeatedly sub-sample such an evaluation set to modify its distribution of human ties and compute acceq on the new test set. The results are shown in Figure 3. We observe that the ranking is unstable at varying percentages of
human ties, putting continuous metrics at a disadvantage if the proportion of ties in the evaluation set deviates significantly from that in the test set.

heading: 5 Conclusion
text: In this work, we identified two issues with the current meta-evaluation of Machine Translation, as conducted at the Metrics Shared Task of the Conference on Machine Translation. We proposed a suite of sentinel metrics designed to highlight these issues and demonstrate their impact on the metrics rankings, revealing that certain metric categories are unfairly advantaged. Indeed, the None Grouping and System Grouping strategies favor trained metrics over overlap- and LLM-based ones and the algorithm of tie calibration favors continuous metrics over discrete ones, or vice versa, depending on the percentage of tied assessments in the dataset used for it. Specifically, continuous metrics are favored if the tie calibration is conducted on the same test set used for the evaluation. Finally, we observed a notably high correlation between sentinel metrics and state-of-the-art metrics, raising concerns about their reliability and suggesting that their assessments might be based on spurious correlations present in the training data.

heading: Acknowledgements
text: We gratefully acknowledge the support of the PNRR MUR project PE0000013-FAIR, and the CREATIVE project (CRoss-modal understanding and gEnerATIon of Visual and tExtual content), which is funded by the MUR Progetti di Rilevante Interesse Nazionale programme (PRIN 2020).
This work has been carried out while Lorenzo Proietti and Alessandro Scirè were enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome.
Limitations
Our analysis recommends grouping translations by their source segment before computing segmentlevel correlations with human judgments, showing that the rankings derived from the No Grouping and System Grouping strategies favor certain metric categories and potentially reward metrics for leveraging spurious correlations. However, we recognize that the Segment Grouping strategy does not evaluate the ability of metrics to distinguish between higher- and lower-quality translations in absolute terms, that is, independently of their source sentence. We believe this aspect should play a role in the meta-evaluation process, and leave to future work the development of fairer methods to fill this gap. Furthermore, we acknowledge that, due to Segment Grouping, each correlation measure is computed on a limited number of data points, i.e., as many as the MT systems that translated each source segment. In this respect, we argue that it would be necessary to investigate the metrics’ ranking stability with varying numbers of MT systems, similar to the work of Riley et al. (2024), where they explored MT systems’ ranking stability in designing human evaluation studies.
Finally, we acknowledge that we did not provide a clear recommendation regarding a fair option for conducting the tie calibration algorithm. We demonstrated that continuous metrics are favored if selecting the optimal ϵ on the same test set used for the meta-evaluation and that using a held-out dataset would not be fair either. Nonetheless, using a held-out set would at least prevent the distribution of human ties used for tie calibration from being identical to that of the test set, and therefore it should be preferred. In general, we believe that
a promising approach might involve studying the meaning of the score deltas of continuous metrics (akin to the work of Kocmi et al. (2024) regarding system-level assessments) and treating as tied all assessments within pre-defined score ranges derived from such deltas. This approach would also enhance the interpretability of MT metrics’ assessments.

heading: A Official Ranking
text: In Table 3, we report the official segment-level ranking of WMT23 Metrics Shared Task, including sentinel metrics.

heading: B Training the Sentinel Metrics
text: The input for the sentinel metrics consists of either the source text (SENTINELSRC), candidate translation (SENTINELCAND), or reference translation (SENTINELREF). Each sentence is tokenized and passed to the XLM-RoBERTa large model, which serves as a feature extractor. Then, we pass the embedding of the [CLS] token to a multi-layer, fully-connected neural network, which outputs the final scalar score. More formally, considering t as
the input text for a sentinel metric:
et = XLM-R(t)
h (1) t = Dropout
( Tanh ( W
(1) h et + b (1) h
))
h (2) t = Dropout
( Tanh ( W
(2) h h (1) t + b (2) h
))
st = Woh (2) t + bo
Where:
• t is the tokenized input sentence.
• et is the [CLS] token embedding at the output of XLM-RoBERTa large.
• h(i)t represents the output of the i th layer of
the fully-connected neural network. Each layer consists of a linear transformation, using weight matrix W (i)h and bias vector b (i) h , followed by a Tanh activation function and a dropout layer.
• Wo and bo are the output layer’s weight matrix and bias vector, respectively.
• st is the output scalar score assigned to sentence t.
Both training phases (i.e., the first, using DA-based human judgments, and the second, using MQMbased ones) employ the same set of hyperparameters, detailed in Table 4.

heading: C Grouping Strategies
text: In Tables 5, 6, 7, we report the complete set of rankings and Pearson correlations, at the segment level, of the primary submissions to the WMT23 Metrics Shared Task, with sentinel metrics. Sentinel metrics are consistently ranked lower with Segment Grouping. Furthermore, in Tables 8, 9, 10, we report the complete set of rankings and Kendall τ correlation coefficients, at the segment level, of the primary submissions to the WMT23 Metrics Shared Task, with sentinel metrics. With Kendall τ as well, sentinel metrics rank lower when Segment Grouping is employed. We wish to note that Segment Grouping requires the estimation of multiple correlation coefficients, which are then averaged. Consequently, each correlation is measured on a substantially smaller number of data points, compared to No Grouping and System Grouping. As a result, the number of clusters of statistical significance is reduced. Therefore, one should not
focus on the absolute values of the ranks but on their value relative to that of the other metrics. For instance, in Table 9, SENTINELCAND is ranked 5th out of 19 with No Grouping, and 4th out of 11 with
Segment Grouping. While the absolute value of the rank is lower, in terms of correlation it has moved from the 8th to the 17th position.

heading: D Metrics Pairwise Correlations
text: In Figures 4, 5, 6, we report the pairwise correlation between a subset of the primary submissions and baselines of WMT23, with the inclusion of sentinel metrics. We use Pearson correlation coefficient with No Grouping. State-of-the-art regressionbased metrics display a notably high correlation with sentinels. Specifically, the highest correlations are reported by XCOMET-Ensemble, MetricX-23, and their reference-less counterparts. Moderate correlation is also reported between sentinels and baseline metrics such as CometKiwi, COMET, and BLEURT-20. As expected, instead, lexical-based metrics such as BLEU and chrF display close to
no correlation with sentinels. Similarly, GEMBAMQM, a state-of-the-art LLM-based metric that has not been fine-tuned on human assessments, shows lower levels of correlation with the sentinel metrics, compared to the other state-of-the-art metrics.

heading: E Length Bias
text: In Figures 7,8 we report the relation between metrics assessments and the length of the candidate translation. We concatenate the data from all the three language directions used in the MQM-based evaluation of WMT23, i.e., ZH → EN, EN → DE, and HE → EN. We wish to remind the reader that the meta-evaluation of WMT23 was conducted at the paragraph level for EN → DE, and therefore, the reported candidate lengths are much larger than those in Figure 1, which comprises only ZH → EN. As we can see from the figures, most regression-
based metrics, sentinels included, almost never assign very high scores to long translations, even if they are correct. This is in marked contrast to metrics trained with different objectives, such as MaTESe, or not fine-tuned to mimic the human judgment, such as GEMBA-MQM. Indeed, both these metrics assign their highest score to several translations longer than 1200 characters. Notably, there are several metrics whose assessments converge to a very narrow range of values as length increases. For example, BLEURT-20’s assessments seem to be confined between approximately 0.4 and 0.8 for translations longer than 1000 characters, and a similar pattern is observed for COMET.

heading: F Kendall τ and acceq Computation Example
text: In this section, we provide an example of the computation of Kendall τ and acceq from two vectors of human and metric scores, i.e., h and m in the following table:
m 0.6 0.5 0.4 0.4 h 5 3 5 5
For each vector, there are six pairs of assessments. In particular, the pairs of metric assessments are (m1,m2), (m1,m3), (m1,m4), (m2,m3), (m2,m4), (m3,m4).
In Equations 1 and 2, C = 1, since the only concordant pair is (m1,m2). Indeed, m1 > m2 and h1 > h2. D = 2, since the pairs (m2,m3), (m2,m4) are discordant. Tm = 0, since
there are no pairs tied only in the metric scores. Th = 2, since the pairs (h1, h3), (h1, h4) are tied only in the human scores. Thm = 1, since the remaining pair, i.e., (m3,m4), is tied in both human and metric scores. In this example, τ = −0.258 and acceq = 0.333.

heading: G Ties
text: In Table 11, we report the percentage of tied human pairs in the datasets used in recent editions of WMT.
In Tables 12, 13, 14, we report the values of pt and pn used to sub-sample the ZH → EN, EN → DE, and HE → EN test sets, respectively, to conduct the experiment described in Section 4.4. We also report the corresponding percentage of human ties and total number of pairs, for each sample.
In Figures 9, 10, 11, we report the acceq and
optimal ϵ for each considered metric, in all three language directions considered at WMT 2023.
In Figure 12, we report the acceq values of the considered metrics, as computed on a 80% split of the test set. ϵ values have been estimated using a held-out set derived as a 20% split of the entire test set. The held-out set is repeatedly sub-sampled to vary its percentage of tied human scores. Different percentage values are reported on the x-axis.

heading: Future Work
text: We believe this aspect should play a role in the meta-evaluation process, and leave to future work the development of fairer methods to fill this gap. Furthermore, we acknowledge that, due to Segment Grouping, each correlation measure is computed on a limited number of data points, i. e. , as many as the MT systems that translated each source segment. In this respect, we argue that it would be necessary to investigate the metrics’ ranking stability with varying numbers of MT systems, similar to the work of Riley et al. (2024), where they explored MT systems’ ranking stability in designing human evaluation studies. Finally, we acknowledge that we did not provide a clear recommendation regarding a fair option for conducting the tie calibration algorithm. We demonstrated that continuous metrics are favored if selecting the optimal ϵ on the same test set used for the meta-evaluation and that using a held-out dataset would not be fair either. Nonetheless, using a held-out set would at least prevent the distribution of human ties used for tie calibration from being identical to that of the test set, and therefore it should be preferred. In general, we believe that
a promising approach might involve studying the meaning of the score deltas of continuous metrics (akin to the work of Kocmi et al. (2024) regarding system-level assessments) and treating as tied all assessments within pre-defined score ranges derived from such deltas. This approach would also enhance the interpretability of MT metrics’ assessments