Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration
Authors: Daniel Deutsch
George Foster
Markus Freitag
Venue: EMNLP 2023 Main (2023)
Keywords: meta-evaluation
metrics
kendall's tau
TL;DR: This work demonstrates the flaws of Kendall's tau with respect to ties for meta-evaluating evaluation metrics and proposes a pairwise accuracy statistic that fixes Kendall's shortcomings.

Abstract: Kendall's tau is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations.  Its focus on pairwise score comparisons is intuitive but raises the question of how ties should be handled, a gray area that has motivated different variants in the literature. We demonstrate that, in settings like modern MT meta-evaluation, existing variants have weaknesses arising from their handling of ties, and in some situations can even be gamed. We propose instead to meta-evaluate metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties, in combination with a tie calibration procedure that automatically introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. We argue and provide experimental evidence that these modifications lead to fairer ranking-based assessments of metric performance.

Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration
Daniel Deutsch, George Foster, and Markus Freitag Google
(dandeutsch, fosterg, freitag)
Abstract
deals with ties ($4). First, MT systems get better, they (1) produce more "perfect" outputs, which get assigned the same score by human raters; and (2) necessitate error-basee analyses such as MQM (Lommel et al., 2014; Freitag et al., 2021a), which often produce tied scores due to integer error counts. Second, on the metric side, the use of recently- LLM--based metries (Koemi and Federmann, 2023) and metrics that model MQM annotations (Perrella al., 2022) can also lead small and discrete score ranges that assign many tics.
Kendall's J is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations. Its focus on pairwise score comparisons is intuitive but raises the question of how ties should be han- dled, a gray that has motivated different variants in the literature. We demonstrate that, in settings like modern MT neta-evaluution, existing variants have weaknesses arising from their handling of ties, and in some situations can even be gamed. We propose instead to meta-evaluate metrics with version of pair- wise accuracy that gives metrics credit cor- rectly predicting ties, in combination with tie calibration procedure that automatically in- troduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. We argue and provide experi- mental evidence that these modifications lead to fairer ranking-based assessments of metric performance.'
In this paper, examine the problems caused by ties in Kendall's т, using data from the WMT metrics tasks. We first show that there are simple phenomena that are not handled properly by any of the existing Kendall variants, which mostly differ in how they treat ties (65.1). We also demonstrate the possibility of gaming the meta meta-evaluation by exploiting how ties are handled by existing 7 r's, re- sulting in large improvements in certain evaluation settings ($5.2).
1 Introduction
We propose instead to meta-evaluate metrics with a version of pairwise accuracy that is robust to these problems, assigning proper credit for cor- rectly predicting ties ($6). Although there is a modification 7 that is closely related to pairwise accuracy, we argue that the accuracy formulation is easier to interpret, being just the proportion of correctly ranked pairs (including tied pairs).
Kendall's т is a widely used correlation statistic (Kendall, 1938). It is easy grasp intuitively, be- ing based on pairwise rank ordering. This makes complementary other well-known statistics such as Pearson or Spearman.
In the context of machine translation (MT), Kendall plays key role assessing the performance of evaluation metrics, a process known as meta- evaluation: it has been the main statistic for measur- ing a metric's ability to segment-level trans- lations in the Workshop on Machine Translation (WMT) metrics shared tasks over the years (Freitag et al., 2022b, inter alia).
However, pairwise accuracy comes with its own problem, namely that it can discriminate against metrics that rarely assign ties. To counter this, we also propose an algorithm called tie calibration that automatically introduces ties into metric scores in order to optimize its correlation ($7). We argue, and show empirically, that these two modifications result in fairer assessment of MT metric perfor- mance ($8.1).
Several recent developments in MT common to other areas of generative have highlighted an important weakness in Kendall's namely how
'The code to run our proposed methods and repro- duce our results can be found at wtteeeeeeeeerccttteeeereeree
Finally, we analyze different aspects of pairwise accuracy and tie calibration, including assessing
com/

---

2.2 Metric Neta-Evaluation
15
metricl: p= 0.94, To 0.78
Metric Score
metric2: 0.71, P= 1.00, To 1.00
For over 10 years, the Workshop on Machine Trans- lation (WMT) has run a metrics shared task that meta-evaluates automatic metrics. Mcta-evaluation quantifies metric's performance by calculating the agreement correlation between the metric's scores and human-annotated scores large num- ber of translations. In WMT, metrics meta- evaluated at either the system- or segment-level, as follows.
5-
0
2
6
8
Human Score
Figure 1: Pearson's 's Spearman's p. and Kendall's 's ть calculated between hypothetical human scores and met- ric scores. Lines between data points are shown for vi- sualization purposes.
First, metric and human scores are collected for translations produced by N systems for M source segments, System-level correlations are calculated between the N metric and human scores per sys- tem, typically calculated by averaging over the M Segment scores. In WMT, the system-level correla- tion is often Pearson, or more recently, ranking- based pairwise agreement that is similar to our pro- posed statistic ($6), except that it does not need to account for ties since ties are very rare at the system-level (Kocmi et al., 2021).
the generalization of tie calibration across datasets ($8.2), the score ranges where ties are introduced ($8.3), and how more fine-graid aneegrained statistics can be used to better understand metric behavior ($8.4). While experimental setting is limited MT metrics, our work should be applicable to meta- evaluation for other generative Al metries with similar characteristics.
Segment-level correlations evaluate metric scores on individual translations rather than ag- gregated system scores. They can be calculated in several different ways (see Appendix A for equa- tion definitions):
2 Background & Related Work
We begin by justifying our exclusive focus on ranking-based statistics, like Kendall's then provide some background MT metric meta- evaluation, and finally contextualize our work by discussing Kendall variants.
No-Grouping: Calculate the correlation be- tween the N M translation scores
2.1 Why not Pearson or Spearman?
Group-by-Item: Calculate the average cor- relation between the translation scores grouped by source segment
Pearson's s r and Spearman's p are two other widely- used correlation coefficients. The Pearson coeffi- cient captures linear correspondence between two input vectors, defined their covariance divided by the product of their variances. Spearman equivalent to Pearson applied to the ranks of the inputs. As shown in Figure 1, Pearson is comple- mentary to Kendall; it assigns much higher score to the noisy but globally linear metricl, but much lower score the perfectly-orderee but non-lincat metric2. Spearman is compromise, siding with Pearson for metricl and for Kendall for metric2.
Group-by-System: Calculate the average correlation between the M translation scores grouped by system
Segment-level correlations are better than system- level correlations discriminating between metrics (Freitag al., 2022b), and they are more closely related applications where metries can be used improve generation, such as Minimum Bayes Risk decoding (Freitag et al., 2022a; Fernandes et al., 2022).
For applications where linear correspondence with a gold standard and correct ranking decisions are both important, it is advisable to measure both Pearson and Kendall, as is typically done in the MT evaluations described b
Historically, WMT has evaluated metrics at the segment-level using the group-by-item method, however no-grouping
used in WMT'21 and all three were used in WMT'22. The standard cor- relation function that is used is some variant of Kendall's т, described next.
Although we focus on problems with Kendall here, it worth noting that Pearson has problems of its own, notably sensitivity to outliers (Mathur et al., 2020). For instance, adding the point (100, 100) to metriel produces an almost perfect correlation of 0.99 compared to 0.82 for Kendall.
"Item" used to keep the terminology generic so it can he applied to other generation tasks. Here, "item" refers to the source segment.

---

Definition
Proposed By
WMT Shared Task Years
(C DD/DD+CC+D0 +T m Tam)
Kendall (1938)
7h = +
D+TTm
Kendall (1945)
2021 2022
(C (D))
Stuart (1953)
Fia [C-D-Ta)(C+++DD
Callison-Buree al. (2010)
2010-2012, 2017-2012,0111-202211 2017-2000*
TJ3 (C [DD]((C DD)
Machášek and Bujar (2013)
2013
TI4 CC -DD)(C + Tm
Macháček and Bojar (2014)
2014-2016
Feg (C Tham D- Tm)/(C + Tm Thm)
This work ($6)
BCCay (C +C + Ta + Tm + Thm)
This work ($6)
Table 1:
Each variant of handles ties differently, and the WMT metrics shared task has not consistently used the same 7 over the years. The acCeq and Teg statistics are proposed this work ($6). See Table 2 the notation definition for this table.
Symbol
Description
fact that the metrics shared task organizers either did not want to penalize small differences in met- ric scores when the human score is tied (Callison- Burch et al., 2010) or only evaluated on pairs that had a large difference in DA score in order to en- sure the pair's 's ranking was reliable (Bojar et al., 2017).
n
number of inputs
vector human scores
vector
scures
number of concordantt pairs
number of discordant pairs
The number of pairs tied only
Tm
The number of pairs tied only
Thm
The numher of puirs tied in both b and
k
The minimum of the number of unique values in it
Overall, none of the r's directly rewards the met- ric for correctly predicting ties in the human score. We view our work as a next step in updating the meta-evaluation to account for properties of today's metrics and human scores.
Table 2: The notation used for defining different r's.
2.3 The Landscape of Kendall's
Kendall's r is a ranking-based correlation coeffi- cient. Although there are many different variants of intuitively, it counts how frequently the metric and human scores agree (concordant) or disagree (discordant) the ranking of all possible pairs of translations. Importantly, there cannot be a tie in either the metric or human score for a pair to be con- sidered concordant or discordant. Each 7 ranges from 1 to 1, with the extremes resulting from the metric and human scores being perfectly discor- dant/concordant and 0 meaning random chance. Some variants of generic and included in li- braries like SciPy, whereas others were proposed by WMT metrics shared task organizers and tailored to the application of MT metric meta-evaluation Ta- ble shows the definitions of the different variants of using the notation in Table 2.
3 Analysis Setup
Datasets Our analysis is performed on the Mul- tidimensional Quality Metrics (MQM; Lommel et al., 2014; Freitag et al., 2021a) ratings collected by the WMT'22 metrics shared task (Freitag et al., 2022b) for three language pairs: en-*de, zh-ten, and en->ru. We use the MQM scores the ground- truth human scores that the automatic metrics' scores are evaluated against. The language pairs have 13-15 systems and around 1300-1900 seg- ments per system with MQM ratings.
Automatic Metrics We explore how the choice of meta-evaluation statistic affects the rankings of the primary metric submissions the WMT'22 shared task, in addition to the recently proposed GEMBA metrics (Koemi and Federmann, 2023).
The main differences between the variants how they handle ties. The standard variants, ть and Te, are modifications of Ta designed ensure the values can reach -1 and 1 in the presence of ties. In contrast to our proposal, the versions proposed by WMT do not include ties in the human scores, and penalize ties in the metric scores. This is due to the
We also discuss and examine various different met- rics in more detail, including the top 2 perform- ing metrics in the WMT'22 shared task, Metric-X and COMET-22 (Rei al., 2022), in addition to BLEURT-20 (Sellam et al., 2020), MaTESe (Per- rella al., 2022), and GEMBA. The former three metrics are regression based metrics that predict floating point translation quality scores. MaTESc predicts span-level errors that are combined into an
"oe note about the error in the WMT' 17 report in the WMT' 18 report (Ma et al., 2018).

---

Group-by-
Metric
en-de
zh-en
en-ru
LP
#Segments
#Pairs
#Ties
#Zero-Ties
Metric-X
0.7%
0.2%
0.5%
COMET-22
1.3%
0.1%
0.5%
en-de
18k
120k
64k (53%)
48k (40%)
MaTESe
71.9%
39.6%
80.8%
zh-cn
28k
197k
82k (42%)
63k (32%)
GEMBA-GPT-335
60.3%
56.6%
50.9%
en-ru
20k
138k
61k (44%)
40k (29%)
GEMBA-GPT- 69.6%
46.9%
60.1%
Table 3: The number of segments, pairs, tied pairs, and pairs tied at MQM=0 (error free) across the dif- ferent WMT language pairs for group-by-item cor- relations. The statistics for other segment-lever correla- tions can be found in Appendix C.
Table 4: The percent of pairs that are tied when group- ing by source segment is drastically different for regres- sion metrics (Metric-X and COMET) versus metrics that effectively act multi-class classifiers (MaTESe and GEMBA).
h = 0,,0, O, 1,
overall score based on an error severity weighting. The GEMBA metrics predict quality scores using 0-shot prompting with GPT-3.5 and GPT-4 (Brown et al., 2020). Importantly, the predicted scores from MaTESe and GEMBA tend come from small set of values rather than a large range of possible floating point scores, which has significant impli- cations for the number of ties they predict (see $4) and how they are treated by different variants of T.
mi 0,
m2
[0, 1, 4,5
Metric
Ta
Tb
To
710
T13
714
Tog
aCCog
m1
.47
.78
.29
.78
.78
.87
.93
m2
.60
.77
.38
1.0
1.0
1.0
.20
.60
Figure When considering ties, my only incorrectly ranks 1 out of the (2) pairs, whereas my incorrectly how each handles ties, only accay and Tog strongly prefer my m2. Notably, Tin- T13- and T14 are unable to distinguish a perfect metric (m = h)) from m2. The acCey and Tey statistics are proposed in this work ($6).
ranks 6. However,
4 Why Ties are Important
There are several motivations for incorporating ties into ranking-based meta-evaluation statistic like Kendall's T.
First, ties in human scores from recent WMT shared tasks are much more trustworthy than they were previously. Since WMT'20, the human scores are MQM scores instead of direct assessment (DA) scores. The MQM scores come from expert trans- lators and are more reliable than the crowdsourced DA scores. As such, ties (or minor differences) between scores are more likely representative of actual ties (or minor differences) in translation qual- ity.
els (GEMBA) predict large number of ties (see Table 4). These metrics should be directly rewarded for correctly predicting ties in the human scores, which is not the case with existing Kendall variants.
5 Shortcomings of Kendall's Variants
The way ties are handled by existing variants of Kendall's т introduces blind spots in the meta- evaluation and opens the door for metrics to ex- ploit r-specific properties to improve correlations. We demonstrate the shortcomings of existing r's through motivational example and experimental analysis.
Second, ties in MQM scores are very common. For instance, up to 53% of possible pairs in en-de have tied MQM scores (see Table 3), the majority of which have MQM scores of O, meaning there are no errors in the translations. As the quality of MT systems improves, the number of tied transla- tions likely to increase since there will be fewer differences between systems. If ties in the MQM scores are removed from the meta-evaluation (as is done by some Kendall variants), we throw away a valuable metric quality signal and lose the ability to discriminate between metries that reliably detect ties and those that do not (see next section).
5.1 A Motivating Example
Due to how existing r's handle ties, they are unable to discriminate between metrics that accurately pre- diet ties and those that do not. Figure 2 contains an example of such instance.
When considering ties, metric mi only incor- rectly ranks out of the 15 possible pairs, whereas m2 incorrectly ranks 6 pairs. However, because existing r's do not give credit to metrics for cor- rectly predicting ties, the correlation coefficients
Finally, recently proposed metrics, such as MaTESe or those based on large language mod-

---

to integers by assigning each score to an equal- width bucket. This bucketing results in ties in chal- lenging pairs because similar quality translations likely have close metric scores, so when the scores are converted to integer buckets, their scores be- come the same value. Figure plots the group-by- item ть (the coefficient used in WMT'22) and the number of non-NaN groups function of the number of buckets.
Group-by-tten To
0.350
Bucketed Scores Original Scores
0.325
0.30 3.000
0.275
ANon-NaN Groups
1000
750
500
When the number of buckets small, the num- ber of non-NaN segments is reduced, and the result- ing correlations improve over the original values by very large margins. Because the correlations with different numbers of buckets are computed different non-NaN subsets of the full dataset, their values are not fairly comparable. Indeed, in $8, we demonstrate that WMT'22 metrics submis- sions were evaluated on different non-NaN groups. and directly comparing their correlations leads to erroneous conclusions.
250
logro(Number of Buckets)
Figure 3: Dividing the Metric-X scores into equal width buckets can increase the group-by-item correla- tion by a large margin. However, at the same time, the number of groups used in the correlation (with non- NaN scores) decreases, meaning the corresponding cor- relations are not fairly comparable since they are com- puted on different sets of data.
A metric could have taken advantage of the NaN problem in order game the WMT metrics shared task since the number of non-NaN seg- ments not taken into account in the metric meta- evaluation. A method for handling ties that made correlations for constant vectors well defined would close this loophole.
either consider my to be either approximately equal or worse than mg. This blind spot of existing r's means they are inadequate for evallanag met- rics in the presence of ties.
5.2 The NaN Problem
Another consequence of how the т correlations handle ties is what we refer to the "NaN prob- lem." In the event that either the metric or human scores are a constant vector (therefore, all pairs are tied), many of the 7 values are not defined, or NaN. When the segment-level correlation is calculated by grouping by either item or system and one of the groups' correlations is NaN, the correlation is re- moved from the average in practice. This happens most often when grouping by item because the size of the input vectors the number of systems, N, which is generally rather small (s=15).
Evaluating with Pairwise Accuracy
Instead of using Kendall's as the ranking-based meta-evaluation statistic, we propose use ver- sion of pairwise accuracy that includes ties. We define the pairwise accuracy to be the proportion of all pairs that the metric either ranks correctly or correctly predicts are tied. The equation for our proposal, denoted accey ("eq" for equality, as ties), is included in Table 1. This statistic now di- rectly incorporates ties in the human and metric scores.
A metric could take advantage of this property of the segment-level correlation by introducing ties for difficult-to-scom groups, resulting in NaN scores. This has the effect of removing the chal- lenging groups from the meta-evaluation, resulting in higher correlations. 5.6 Indeed, we find that this is possible.
Although there is a modification of Kendall's T that corresponds to acCeq (denoted Tey in Table 1), we advocate for reporting accuracy instead. Accu- racy is more intuitive than t since its value is be- tween 0 and and it can be read as the proportion of pairs that the metric correctly ranks/predicts as ties. This stands in contrast to that is between -1 and 1, which does not have an easy-to-communicate interpretation. Pairwise accuracy has the additional benefit of aligning how metrics are meta-evaluated at the system- and segment-levels (Kocmi et al., 2021). The results related to metric rankings in this
To introduce ties, we mapped Metric-X's scores
'Another possibility would be assign a neutral cor- However. this has the disadvantage penalizing metrics that assign ties when human scores are also tied or are close to being tied.
relation value
"Noee that both Pearson and Spearman are also NaN for constant vectors and therefore are also susceptible to gaming.

---

Metric
Definition
it prevents a fair comparison between metrics that do and do not predict ties.
tiéSprecision
Thm /Thm + Tm) Thm /TTh/TThmm + T C/(C + +D+++++ Ihn C/(C+ )
To address this shortcoming, we propose an algo- rithm called tie calibration for automatically intro- ducing ties into metric scores so that metrics that do and do not predict ties can be fairly compared. The algorithm is based on the intuition that, although regression metrics do not frequently predict ties, the difference between two translations' scores is sometimes small enough to be considered tie.
tieSrecall
correct-rankppecision
correct-rankecall
Table 5: Definitions of precision and recall on correctly predicting tied pairs or the correct ranking of non-tied pairs. See Table 2 for the notation definition.
Tie calibration searches for an e value that maxi- mizes a rank-based correlation statistic (e.g., or acCoy) such that any two translations with differ- ence in score less than is considered to be tic.? Our implementation considers all possible differ- ences between the (2) pairs of translations as can- didates for e and selects the one that maximizes the desired ranking- statistic. The algorithm runs in (n² log n),, where n is the number of transla- tions. 8 Detailed psuedocode for tie calibration is included in Appendix D.
work apply equally aCCey and Teq.
Pairwise accuracy (and Teq) does not suffer from the same issues as the r's that were presented in $5: acCeq strongly prefers m.. the metric with fewer incorrectly ranked pairs (Figure 2; $5.1). Because its value is never NaN, it does not suffer from the NaN problem ($5.2); all examples are always used for evaluation.
6.1 Evaluating Ties and Non-Ties
Pairwise accuracy effectively evaluates the auto- matic metrics as 3-way classifiers that decide be- tween predicting a tie or one of the two possible rankings for each pair. This formulation nicely al- lows for further decomposition into class-specific precision, recall, and F,, which can be used to fur- ther understand metric performance. Class-specific evaluations help to address a potential class imbal- ance problem between tied and non-tied pairs that may be hidden by accuracy.
Because tie calibration introduces an optimal number of tie predictions, metrics are not penalized for mnder-predicting ties, and therefore metrics that do and do not predict ties can be fairly compared. An added benefit of tie calibration is that the result- ing optimal e improves the interpretability of metric scores, Its value can be understood as the threshold for which difference in metric scores should be considered significant (at least with respect to specific dataset; see $8.2).
Table 5 contains the definitions of precision and recall with respect to "ties" and "correct ranking." " The "ties" statistics calculate the precision of the metric when predicting a tie and its recall of hu- man ties. The "correct ranking" statistics calculate the proportion of correctly ranked pairs out of all pairs it predicts are not tied and the proportion of all human non-tied pairs correctly ranked by the metric. These additional statistics help provide more holistic view of metric performance.
Henceforth we use to denote a statistic that has been calculated with tie calibration (e.g., acc and Ex the optimal tie threshold found by the algorithm.
Discussion. In principle, tie calibration can be used to find an optimal value of any correlation statistic in which the presence of ties changes the value, acCey being one of them. However, care needs to be taken to ensure that the statistic handles ties in a desirable way. For example, 713 omits all ties from its formula, so tie calibration could convert a discordant pair into tie to improve the value of T13, which, if the human scores are not tied, is undesirable (accey would not reward this change). The combination of tie calibration and a
7 Tie Calibration
Although we argue that acCeq properly addresses ties in human and metric scores, some metrics do not frequently predict exact ties between transla- tions. Regression metrics, such BLEURT (Sel- lam al., 2020) and COMET (Rei et al., 2020), practically never predict tied scores for two differ- ent translations (see Table 4), so they will not able to correctly predict a tie in the human score, putting them at a disadvantage. This is undesirable because
We experimented with relative differences between scores and found little difference compared to absolute differ- ences.
'In practice, when n is large, we downsample the number of pairs to consider when searching for which significantly improves runtime. Experimentally. this appears to be rather gond approximation (sce Appendix E).

---

Metric
MaTESe and GEMBA, the two metrics that output the most ties.
MetticXX
0.270 4)
0.381 1)
0.605 1)
0.04
UniTE
0.278 3)
0.322
3)
0.595
2)
0.14
20MET-22
0,258
0.366
0.594
3)
Under Tb. GEMBA-GPT- and MaTESe are the top ranked metrics. However, this result can be partially explained by the NaN problem ($5.2).
MaTESe
0.281
0.459 (16)
0.582 4)
0.00
UniTE-sre
0.205 9)
0.221 8)
0,582 5)
0.12
GEMRA-GPT4
0.322 !)
-0.367 (15)
0.573 6
4.00
MaTESc-QR
0.234
-0.573 (17)
0.572 7)
MaTESe's correlation is calculated on 773 non-
COMRTKiwi
0.181
0.254
0.572
0.16
NaN segments, compared 1133 for Metric-X.
BLEURT-20
0.254
0.289 4)
0.568 9)
0.09
MS-COMET-22
(13)
0.241 7)
0,565 (10)
4.65
When both metrics are evaluated on the same 773 segments, Metric-X's correlation is higher (0.296
COMET-QE
0.182 (14)
0.179 (10)
0,555 (11)
SEScore
0.269
0.554 (12)
1.30
MS-COMMET-QE-22
0,080 (16)
(12)
0.550 (13)
6,50
versus 0.281). 1. This result highlights how correla-
HWTSC-Teachere
0.106 (15)
0.123 (11)
0,545 (14)
0.34
tions calculated on different source segments can- not be fairly compared.
GEMBA-GPT335
0.209 8)
-0.344 (14)
0,545 (15)
15.00
MEE4
(11)
0.539 (16)
REUSE
0.074 (18)
-0.134 (13)
0.534 (17)
0.47
Constant-Metric Metrie
3.000 -1,000 (18)
0.534 (18)
0.00
If T10 is used to rank the metrics, MaTESe and GEMBA fall to the bottom at of the ranking. This result can be explained by the fact that Tio is sys- tematically biased against metrics that output a large number of ties because ties are penalized as if they are discordant pairs. Predicting a tie can only decrease 710 correlation. In fact, the T10 values of MaTESe and GEMBA can be improved by large margins simply by randomly breaking all ties since around half the pairs will now become concor- dant, while the other half remain penalized as they were before. For example, randomly breaking ties improves MaTESe and GEMBA-GPT-33, corre- lations by around 0.5-0.6 points (MaTESe: - -0.159 to ==0 15, GEMBA: -0.344 05.15). In contrast, COMET-22's correlation only improves by =0.005 due the fact that it predicts few ties (see Table 4).
Table 6: The correlations (and ranks) of the metrics evaluated by The Tio, and acCaq with tie calibration, denoted acc/y. using the group-by-item segment-level correlation on the WMT'22 en-de dataset. e" the op- timal threshold found by tie calibration.
statistic that does not properly handle ties may lead to unexpected results.
8 Analysis
In this section, we analyze several different aspects related to our proposal of pairwise accuracy and tie calibration. We address the following questions:
$8.1: I: How does the choice of meta- evaluuataa statistic affect metric ranking?
In contrast, when the metrics are ranked by acceq with tie calibration, denoted acc MaTESe and the GEMBA metrics ranked 4th, 6th, and 15th. Because Teg and acCey are never NaN, all values are fairly comparable. Further, there no systematic bias for or against ties; Randomly breaking or in- troducing ties runs the risk changing a correct prediction of a tie or concordant pair into a discor- dant pair or an incorrect tie prediction. Clearly, the choice of correlation statistic matters, and we ar- gue that acct, is the most fair and reliable method compared to the variants.
$8.2: How does the selected value of e gener- alize across datasets?
Does the selected e value introduce ties uniformly across score values for a metric?
$8.3:
What insights be drawn from eval- uating metrics on predicting tied versus non- tied pairs?
$8.4:
8.1 Comparing Metric Rankings
Table shows group-by-itee correlations calcu- lated with various 7's and pairwise accuracy. We also report the performance of a "constant metric" that predicts a tie for every pair as a baseline com- parison. From the existing 7's, we report πь and Tio since they are the most recent and used most frequently by WMT.9 Clearly, the choice of meta- evaluation statistic significantly affects the metric rankings, with the largest changes happening to
8.2 Generalization of Epsilon
The previous analysis selected e* on the same dataset that is used to rank the metrics. Here, we examine what happens the e value is selected on a held-out dataset. For this analysis, the MQM rat- ings from the WMT'21 metrics shared task (Freitag et al., 2021b) are used as a held-out set.
Figure 4 shows the different ex and acc values for BLEURT-22 when é is selected on one dataset
"See Appendix C for the results for cach language pair, typo of segment-level correlation, and correlation statistic.

---

en-de
Number of Pairs
All Pairs
0.5
WMT21
5000
Tidd Pairs
Pairwise Accuracy
WMT22
Accuracy
0.4
Best Epsilon
8.0
0.2
0.4
0.5
zh-en
Average Pair Score
0.4
Figure 5: The distribution of average pair scores where ties are introduced for Metric-X on WMT zh-en us- ing as the threshold is skewed right with respect to the distribution of all pairs, suggesting the is biased toward introducing ties to predict perfect translations.
0.3
0.0
0.2
0.4
0.5
0.8
1.0
Epsilon
Figure 4: The generalization of the selected 6* (dashed line) across datasets appears to depend on specific prop- erties of the datasets. We suspect if the number of ties in the datasets is very different (as in zh-en), the is less likely to generalize well.
0.6-
Accuracy
0.4
F1
Correct Rank F1 Best Epsilon
and applied to the other for en-de and zh-en. For en- de, the epsilon value changes by 0.03, and the acCoq calculated the held-out changes by relative 2%; suggesting the results rather stable.
0.0
0.00
0.25
0.50
0.75
1.25 Epsilon
1.00
1.50
1.75
2.00
Figure 6: The F, scores for predicting ties or cor- rect pair rankings for COMET-22 on WMT'22 en-de demonstrate the metric is better at predicting ties than correct pair rankings.
However, zh-en behaves quite differently. From the plots, clear that for WMT'21, there al- most never an incentive to predict a tic, as evi- denced by the very low e", and the corresponding e* does not generalize well to WMT'22 (or vice versa). Our hypothesis is that this result is due to the fact that the WMT'21 zh-en data has far fewer ties than the WMT data (23% versus 41 %)..
parison to the distribution of all pairs' average scores, the tied distribution is skewed toward higher predicted scores. Since Metric-X has a relatively strong correlation to MQM scores, this suggests that the newly introduced ties mostly predict per- fect translations, which are assigned high scores according the metric. An extension of our tie cal- ibration procedure could first identify threshold to predict a perfect translation, then run tie calibration on the remaining pairs.
These results indicate that e" values are likely to generalize across dissimilar datasets under cur- rent metrics. Such property would be desirable- and an interesting challenge for metric developers since it would make score differences more inter- pretable. However, we argue that treating e* as a latent variable calibrated on the current test set allows for fair comparisons of metrics even in the absence of this property. Other evaluation proto- cols have also involved optimizations on the test set, for example using an oracle sentence segmenter to evaluate MT for speech (Matusov et al., 2005).
8.4 Class-Specific Statistics
Figure 6 plots the ties-F1, correct-rank-F; (see $6.1), 1), and pairwise accuracy for COMET-22 on en-de. The ties-F, is much higher than the correct- rank-F1 for almost every e, demonstrating that the metric more reliably predicts tied pairs than the cor- rect rank for non-tied pairs. This likely due to the fact that the number of perfect translations is large, and the values are biased toward introducing ties to predict perfect translations ($8.3).
8.3 Where are Ties Introduced?
Since most of the human score ties are for error free translations (see Table 3), it is worth understanding if the tie threshold introduces ties for high scoring translations predict error- free translation ties or if the ties are introduced more uniformly across the score distribution.
If statistic other than pairwise accuracy is better aligned to how metric is being used practice, the tie calibration procedure can be used to select an e that strikes the desired balance performance with respect to the class-specific statistics.
Figure plots the distribution of the average score per pair where ties are introduced by e* for Metric-X on the W2 zh-en dataset. In com-

---

9 Conclusion
References
Ondřej Bojar, Yeette Graham, and Amir Kamran. 2017. Results of the WMT17 Metries Shared Task. In Proceedings of the Second Conference on Machine Translation, pages 489-513, Copenhagen, Denmark. Association for Computational Linguisties.
In this work, we demonstrated the importance of taking ties into account when calculating rank- based correlation statistics. We argued existing variants of Kendall's rent state of eell-evalu We advocated to instead use pairwise accuracy, which rewards met- ries for both predicting correct pair rankings and correctly predicting ties, combination with tie calibration procedure that allows for comparing metrics that do and do not predict ties. Although our experiments were specific to MT, the methods proposed are generally applicable to any metric meta-evaluation in NLP.
inadequate for the cur-
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. Advances in neural information process- ing systems, 13:1877-1901.
Chris Callison-Burch, Philipp Kochn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Ma- chine Translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 17-53, Uppsala, Sweden, As- sociation for Computational Linguistics.
Limitations
The tie calibration algorithm introduced in $7 makes an assumption that absolute differences in metric scores reflect the same amount of change in quality for any value of the metric. That is, the difference in predicted quality between translations with scores 0.2 and 0.1 is the same as with scores 100.2 and 100.1. An alternative version of the tie calibration algorithm could introduce ties based on relative differences between scores instead of ab- solute differences. We experimented with relative differences and did not see a significunt different in results. However, it may be that a metric that we did not experiment with performs better with relative instead of an absolute difference.
Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-Aware Decoding for Neural Machine Translation. the 2022 Conference of the North American Chap- of the Association for Computational Linguis- ties: Human Language Technologies. pages 1396- 1412, Seattle, United States. Association for Com- putational Linguistics.
Proceedings of
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Transla- tion. Transactions of the Association for Computa- tional Linguistics, 9: 9:1460-1474.
Since the tie decision operates the pair level, the e value does not induce a global ordering of translations. For example, if there are scores 1, 2. and 3 with < = 1, pairs (1, 2) and (2, 3) are tied but (1, 3) is not. A similar limitation can also be observed in pairwise statistical significance testing. Finally, although we argue that our meta- evaluation proposal is more fair, we are unaware of any way to prove that this is true. Instead, we rely on experimental results and the fact that our proposals are not susceptible to known issues with existing methodologies.
Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022a. High Quality Rather than High Model Probability: Minimum Bayes Risk De- coding with Neural Metrics. Transactions of the Association for Computational Linguistics, 10:811- 1- 825.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi- kiu Lo. Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André E. T. Martins. 20226. Results of WMT22 Metrics Shared Task: Stop Using BLEU Neural Metries Are Bet- ter and More Robust. In Proceedings of the Seventh Conference on Machine Trunslation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Acknowledgments
Markus Freitag. Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondfej Bojar. 2021b. Results of the WMT21 Met- rics Shared Task: Evaluating Metrics with Expert- based Human Evaluations on TED and News Do- main. In Proceedings of the Sixth Conference Machine Translation, pages 733-774, Online, Asso- ciation for Computational Linguistics.
The authors would like thank Juraj Juraska, Mara Finkelstein, Ricardo Rei, Chi-kiu Lo, Tom Koemi, and Alon Lavie for their helpful discussions and feedback related to this work.

---

Maurice G Kendall.
1938. A New Measure of Rank Correlation. Biometrika, 30(1/2):81-93
Ricardo Rei. José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Mar- tins. 2022. COMET-22: Unbabel-IST 2022 Submis- sion for the Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Trunslation (WMT), pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguisties.
Maurice Kendall. 1945. The Treatment of Ties in Ranking Problems. Biometrika, 33(3):239-22 251.
Tom Koemi and Christian Federmann. 2023, Large Language Models Are State-of-the-Art Evaluators of Translation Quality.
Tom Kocmi, Christian Federmann, Roman Grund- kiewicz, Marcin mmczyy-Dowmmun Hitokazu Mat- sushita, and Arul Menezes. 2021. To Ship Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478-494, Online, Association for Computa- tional Linguisties.
Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Associa- tion for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning Robust Metrics for Text Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7881-7892, Online. Association for Computational Linguistics.
Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional Quality Metrics (MQM): Framework for Declaring and Describing Transla- tion Quality Metrics. Tradumâtica, (12):0455-463.
Qingsong Ma, Ondfej Bojar, and Yvette Graham. 2018. Results of the WMT18 Metries Shared Task: Both characters and embeddings achieve good perfor- mance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671-688, Belgium, Brussels. Association for Com- putational Linguistics.
Alan Stuart. 1953. The Estimation and Comparison of Strengths of Association in Contingency Tables. Biometrika, 40 1051/12
A Correlation Definitions
This section more explicitly defines the three dif- ferent types of segment-level correlations.
Matouš Macháček and Ondfej Bojar. 2013. Results of the WMTI3 Metries Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Transla- sion, pages 45-51, Sofia, Bulgaria. Association for Computational Linguistics.
Let hij and maj denote the human and met- ric scores for the translation produced by system E I,...,N on source segment j E 1,..., M. Define Corr(-) to be a correlation coefficient, such as Pearson's iassonn' r, Spearman's 's p, Kendall's т, or any such function that calculates an agreement score a set of paired observations, like the pairwise accuracy statistic proposed in this work. There are three different segment-level correlations that be computed.
N
Matouš Macháček and Ondiej Bojar. 2014. Results of' the WMT14 Metries Shared Task. In Proceedings of the Ninth Workshop on Statistical Machine Trans- lation, pages 293-301, Baltimore, Maryland, USA. Association for Computational Linguistics.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in bleu: Reevaluating the eval- uation of automatic machine translation evaluation metries. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4984-4997,
1. No-Grouping
Corr miy)) N.M
(I)
Evgeny Matusov, Gregor Leusch, Oliver Bender, and Hermann Ney. 2005. Evaluating Machine Transla-
tion Output with Automatic Sentence Segmentation. In Proceedings of the Second International Work- shop Spoken Language Translation, Pittsburgh. Pennsylvania, USA.
2. Group-by-Tten
Σ
Cor (()))))
(2)
Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Niccolò Campolungo, and Roberto Navigli. 2022. MaTESe;
M
Machine Translation Evaluation as a Se- quence Tagging Problem. In Proceedings of the
3. Group-by-Systend
Seventh Conference on Machine Translation (WMT), pages 569-577, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguis- tics.
1 N
Corr ([hhig. miy))
(3)
N

---

Metric
Metric
Notation
Tey
Human
C
Tm
D
Human
11
-1
T
Thm
Th
1
-1
T
C
Table A mapping between the notation this paper and the tabular notation from WMT" 14.
Metric
aCCey
Metric
Human
0
1 0
710
0 1
Human
11
X
X
X
Table The tabular versions of Tog and acCoq.
1
-1
C Additional Results
Metric
713
Table 10 contains more statistics related to the num- ber of tied pairs in the WMT'22 MQM scores, in- cluding the number of pairs that are tied with score of 0 (i.e., error free translation).
Human
X
-1
X
X
X
X
1
The full correlation results and metric ranks ac- cording to the different 7s across different language pairs and segment- correlations is included this section. See Table 11 for the listing of the individual tables.
Metric
Human
0
-1
X
X
0
1
D Tie Calibration Psuedocode
Table 8: The tabular versions Tio, Ti3 and 714-
Algorithm contains the pscudocode for the tic cal- ibration procedure ($7) when applied to two vectors of human and metric scores. The algorithm runs in On log n) where n is the number of scored translations. The bottleneck sorting all the (?) possible pairs. When (2) is too large, we approxi- mate the search for the optimal by downsampling the number of pairs. See Appendix E for an analy- sis of how lossy this approximation is.
B WMT Tabular Notation
WMT' 14 (Macháčck and Bojar, 2014) developed a tabular notation to describe how Kendall's 7 was calculated. For completeness, we include a map- ping of the notation from this work in Table 2 to the tabular notation in Table 7. The tabular versions of Tio, T13, and T14 reproduced in Table 8. The tabular versions of Teg and acceq are included in Table 9.
In practice, the tie calibration is applied to ma- trices of human and metric scores, where each row corresponds to a group (see $2). The algorithm is very similar to Algorithm except there addi- tional bookkeeping required to match each (i, pair to the group that it came from. The extra book- keeping only adds an O(1) overhead.
Re-using the notation from WMT 14, a value can be computed using the tabular notation via the following equation:
Ch
h.me (<) Ch.MXX
E Epsilon Search Approximation
(4)
Sh
Finding the exact value that maximizes pair- wise accuracy requires considering all possible (*) choices of e. For specific segment-level correla- tions, such as the no-grouping variant, (?) can be prohibitively large, on the order of hundreds millions of pairs (see Table 3).
hmm[(s,,))
Cnm#X
Ch defined as the coefficient in the tabular notation and Sr.m is the number of pairs that fall into the corresponding bucket.

---

No Grouping
Group-by-Item
Group-by-Systen
LP
#Translations
#Pairs
#Ties
#Zero-Ties
#Pairs
#Ties
#Zero-Ties
#Pairs
#Ties
Fiero-Ties
en-de
58m (34%)
(29%)
120k
64k (53%)
48k (40%)
4mn (35%)
dmn (29%)
zh-en
28k
395m
103m (26%)
87m (22%)
197k
82k (42%)
63k (32%)
26m
7m (26%)
6m (22%)
cu-ru
20k
195m
44m (22%)
35m (18%)
138k
61k (44%)
40k (29%)
3m (23%)
2m (19%)
Table 10: The numher translations, pairs, tied pairs, and pairs tied MQM=0 (perfect translations) across the different WMT language pairs and segment-level correlations.
LP
Correlation
Table
Algorithm An On log n) algorithm that intro-
duces metric ties to select an optimal T value.
No-Grouping
Table 12
1: function TIECALIBRATIONO. m.
en-de
Group-by-Iten
Table 13
C. Thy Tm, 7hm SUFFSTATS(H, m,0)
Group-by-System
Table 14
3:
p= [[. :1, 1....,,.333
Sort P Pby |m, myl
No-Grouping
Table 15
20
en-ru
Group-by-Tice
Table 16
Teur
TC, Th, Tma Thm)
Group-by-Systen
Table 17
Goum
for
(t, E PPPdd do
if [m, myl # Caur then
No-Grouping
Table 18
10:
I' t- max(T" .Terr)
zh-en Group-by-Ttent Table 19
11:
Remove (i. from C,D.T, T Tam
Group-by-System Table 20
h; h; then Tam Tam
13:
else
Table I1:
Pointers to the full correlation and metric ranking results under different тS for each language pair and lype of segment-level correlation.
15:
+1
16: Tour + D, Th,, Tmi Tim
17: Gair f- m, myl
max(T"" Toun)
return 7
When the number of pairs is too large, we in- stead find an approximate best by sampling from
20:
function
21:
function SUFFSTATS(H, m. m)
22:
C.D. Dr, Tm. hmm <- 0,0,0
all possible pairs. Figure 7 plots the e* values cal- culated subset of the data and the value of acCeg for those e* values. Even with as little as 10% of the possible pairs, the approximations are
23:
for (f.j) E ((i, i, l....,.....
j] do
24:
ht: = hú and [m; - m,l S then
25:
Thm + Tim +1
26:
else h, h, then
27:
T, <- Tw +1
quite precise. The largest observed differences over
28:
else m: -myl I c then
29:
Te. < T. +1
30 iterations for e" were 2.5e-3 and for acCeq were
30:
clse Sign h; - hy) = Sign(m, my) then
4.3e-5. Overall, downsampling appears to be safe
31:
C C +1
approximation to improve the run time of the tie calibration algorithm.
32:
else
33:
1) 1-1+1
34:
return C, D,D, Tmn Thm
35:
end function
F Unbabel Normalization
The experiments in this paper calculate MQM scores for translations using the normalization tech- nique advocated for by Google: translation's 's MQM score is the sum of the weights of each of the errors. The alternative method used by Un- babel normalizes the sum of error weights by the length of the translation. The Unbabel normaliza- tion will thus result in fewer human score ties than the Google normalization.
for the group-by souup-by-ite -ttm segment-level correlation are shown in Table 21.
Overall, the fewer ties did not make a significant impact on whether or not it was possible to demon- strate that the meta-evaluation statistics are biased toward or against ties. For instance, ть favors met- rics with ties, such as GEMBA-GPTE and Tio is still biased against metries that predict ties. We sus- pect this is due the fact that the majority of ties occur for perfect translations, which will remain tied in either normalization method. Further, the number of non non- perfect ties (MQM score O) only decreased by 7% (from 44% to 37%). Therefore,
We repeated the analysis from $8.1 using the Unbabel normalization method and calculated the rankings of the different metries under acCeq and T variants for the en-ru language pair. The results

---

Metric
73
Tr
T10
118
714
a0Cay
ace og
Metric-X
0.289 ( 2)
0.356 ( 2)
0.293 2)
0.440 ( 2)
0.440 4)
0.440
2)
0.473 4)
0.525 1)
0.05
UniTE
0.288 < 3)
0.356 ( 3)
0.292 ( 3)
0.438
3)
0.438 5)
0,438
3)
0.473 5)
0.519
2)
0.11
COMET-22
0.292 ( 1)
0.361 ( 1)
0.296 ( 1)
0.445 1)
0.445 3)
0,445 1)
0.475 3)
0.518
3)
0.14
MaTESe
(.170
0.323 6)
0.181 (14)
-0.241 (16)
0,516 2)
0,258 (14)
0,500
0,494 4)
0.00
GEMBA-GPT4
0.199
0.347 4)
-0.153 (15) (11)
0.555 1)
0.302 (11)
0,481
2)
0,493 5)
4,00
BLEURT-20
0.274 4)
0.338 5)
0.278
4)
0.417
0,417 8)
0.417 4)
0.466 6)
0,490
0.06
DS-CO2ET222
0.225 (
0.277 (10)
0.228
7)
0.342
8)
0.342 (11)
0.342
0,441
0.487
1.21
UniTE-sre
0.229 6)
0.283 9)
0.232
6)
0.349
6)
0.349 (10)
0.349 6)
0.444 (10)
0.479
0.10
COMETKiwi
0.230 5) 0.283 8)
0,233
5)
0.349 5)
0.349 9)
0.349 5)
0.444 9)
0.473 9)
GEMBA-GPT-355
0,208 (10) 0.301 7)
0.222 9)
0,058 (14)
0.426 6)
0.316 (10)
0.452 8)
0.461 (10)
5.00
COMET-QE
0.225 ( 8)
0.277 (11)
0.228 8)
0,342 ( 7)
(12)
0.342 8)
0.441 (12)
0.457 (11)
0.01
MrTESe-QQE QE
(16)
0.242 (13)
0.129 (15)
.0.391 (17)
0.422 7)
0.181 (16)
0.457 7)
0.456 (12)
0.00
MS-COMET-QQ-22
0.184 (13)
0.186 (13) (15)
0.279 (11)
0.279 (15)
0.279 (13)
0,421 (15)
0.456 (13)
1.46
SFScore
0.211 ( 9)
0.261 (12)
0.214 (10)
0.322 9)
0.322 (13)
0.322 9)
0.435 (13)
0.452 (14)
0.39
MEE4
0.191 (12)
0.236 (14)
0.194 (12)
0.290 (10)
0.291 (14)
0,290 (12)
0.425 (14)
0,429 (15)
0.01
HWTSC-Teacche-Sii
0.122 (15)
0.150 (16)
0.123 (16)
0,185 (12)
0.185 (16)
0,185 (15)
0,390 (16)
0,403 (16)
0.15
REUSE
0.046 (17)
0.057 (17)
0.047 (17)
0,070 (13)
0.070 (17)
0.070 (17)
0.352 (17)
0.354 (17)
0.01
Constant-Metric
0.000 (18) 0.000 (18)
0.000 (18) -1.000 (18)
0.000 (18) 0.000 (18) 0.342 (18)
0.339 (18)
0.00
Table 12: The correlations (and metric ranks) for the no-grouping correlation on the WMT' en-de dataset.
Metric
F1o
F14
a0Cag
acCgg
Metric-X
0.174
0,270
0.269 1)
0.381 (
0.385
0.384
0.325 (11) 0.605
0.04
UniTB
0.162 3)
0.278
0.255 2)
0.322
0,382 4)
0.368
0.425 6)
0.595 (
COMET-22
0.163 2)
0.258
0.255 3)
0.366 (
0,371
0.370
0.325 (12)
0.594
MaTESe
0.080 (12)
0.281
0.207 6)
-0.459 (16)
0,391
0.171 (13)
0.582 1)
0.582
0.00
UniTE-src
0.123 5)
0,205
0.190 7)
0,221
0,277 9)
0.266
0.406
0.582
0.12
GEMBA-GPTA
0.106 9)
0.322
0.241 4)
-0.367 (15)
0.487 1)
0.237
0.567 ()
0.573
4,00
MaTESe-QE
0.059 (15)
0,234
0.175 (10)
0.320 (17)
0.121 (15)
0.572 2)
0.572
0.00
COMETKiwi
0.116 6)
0,181
0.178 9)
0.254
0,259 (11)
0.259
0.301 (14)
0.572
0.16
BLEURT-20
0.149 4)
0.254
0.233 5)
0.289
0.344 ( 6)
0.334
0.419 7)
0.568
0.09
MS-COMET-22
0.107 8)
0.169 (13)
0.166 (11)
0.241
0,244 (13)
0.244
0.294 (15)
0.565 (10)
4,65
COMET-QF:
0.094
0.138 (14)
0.144 (14)
0.179 (10)
0.182 (14)
0.182 (12)
0.285 (17)
0.555
SEScore
0.114 7)
0,182 (10)
0.180 8)
0.269
0,270 (10)
0.270
0.291 (16)
0.554
1.30
MS- MS-COMET-QE-22
0.051 (16)
0.080 (16)
0.076 (16)
0.116
0,118 (16)
0.118 (16)
0,264 (18)
(13)
6.50
HWTSC-TTecher--im
0.067 (14)
0.106 (15)
0.103 (15)
0.123 (11)
0,149 (15)
0.147 (14)
0,328 (10)
0.545 (14)
0.34
GEMBA-GPT-335
0.078 (13)
0.209 (
0.151 (13)
-0.344 (14)
0.324 7)
(11)
0.509 5)
0.545 (15)
15.00
MEE4
0.100 (10)
0.182 (11)
0.157 (12)
0.201 (
0,252 (12)
0.246 8)
0.394 9)
0.539 (16)
0.13
REUSE
-0.052 (18)
-0.074 (18)
-0.081 (18)
-0.134 (13)
-0.093 (18)
-0.087 (18)
0,319 (13)
0.534 (17)
0.47
Constant-Metric
(17)
0.000 (17)
0.000 (17)
(18)
(17)
0.000 (17) 0.534 4)
0.534 (18)
0.00
Table 13: The correlations (and metric ranks) for the group-by- bt-m correlation on the WM en-de dataset.
we argue that the results presented in this work apply to either normalization technique, but larger changes will likely be observed under using the Google method due to the increase number of ties.
0.055
Epsilon
0.050
0.045
Accuracy
0.52475
0.52450
0.52425
0
8
10
Sampling Percent
Figure 7: The plots show the best e" found to maxi- mize acCeq when the (?) pairs downsampled (per- cent shown on a-axis). The e" is then used to calculate accay on all (%) pairs. Fach sampling rate was run 30 times and the maximum, minimum, and mean values
are shown in the plots. Even with as little as 10% of the possible pairs, the approximation very good.

---

Metric
To
T18
714
BCCcy
Metrie-X
0.285 ( 3)
0.351 ( 3)
0.293 ( 3)
0.434 ( 3)
0.434 5)
0.434
3)
0.469 5)
0.524
0.05
COMET-22
0.290 1)
0.357 ( 1)
0.299 1)
0.441 1)
0.441
( 3)
0.441
1)
0.472 3)
0.515 2)
0.16
UniTE
0.286 ( 2)
0.352 2)
(
0.434 2)
0.434
4)
0.434
2)
0.470 4)
0.511 3]
0.14
MrTESe
0.167 (14)
0.314 6)
(14)
-0,255 (16)
0.508
2)
0.253
(14)
0.499
1)
0,494 4)
0.00
GEMBA-GPT-4
0.19
0.338 (338(4) 4)
(11)
-0.168 (15)
0.542 1)
0.293 (11)
0,480 2)
0,486 5)
4,00
BLEURT-20
0.272 ( 4)
( 5)
0.280 4)
0.413 4)
0,414 8)
0.414 4)
0.463 6)
0,486 6)
0.04
MS-COMET-22
0.223 ( 8)
0.230 (117
9)
0,338 ( 8)
0.338 (12)
0.338
8)
0.439 (12)
0,482 7)
1.30
UniTE-sre
0.229 ( 6) 0.282 9)
0.235
6)
0,347 ( 6)
0.347 (10)
0.347
6)
0,441 (10)
0,473 8)
0.09
COMETKiwi
0.229 5) 0.282 8)
0.236 5)
0.347 5)
0.347 9)
0.347
5)
0.442
0.471
0.12
COMET-QE
0.225 ( () 0.276 (10)
0.231 7)
0.341
7)
0.341 (LI)
0.341 7)
0.439 (1L)
0.459 (10)
0.02
MaTESe-QE
0.117 (16)
0.236((3)
0.131 (15)
-0,401 (17)
0,155 7)
0.177 (16)
0.457 7)
0.456((1)
GEMBA-GPT-3.5
0,207 (10)
0.299 7)
0.231 8)
0,056 (14)
0.424 6)
0.315 (10)
0.451 ( 8)
0.453 (12)
5.00
SEScore
0,209 ( 9)
0.258 (12)
0.215 (10)
0,318 ( 9)
0.318 (13)
0.318 9)
0.432 (13)
0.448 (13)
0.43
MS. COMET-QR-22
0.185 (13)
0.228 (15)
(13)
0.280 (11)
0.280 (15)
0.280 (13)
0.419 (15)
0.445 (14)
1.33
MEE4
0.190 (12)
0.234 (14)
0.195 (12)
0.288 (10)
0.288 (14)
0,288 (12)
0.423 (14)
0.432 (15)
0.01
HWTSC- TSC-Teacher-SSim
0.121 (15)
0.149
0.125 (16)
(12)
0.184 (16)
0.184 (15)
0.388 (16)
0.405 (16)
0.15
REUSE
0.057 (17)
0.070
0.058 (17)
0.086 (13)
0.086 (17)
0.086 (17)
0.355 (17)
0,356 (17)
0.00
Constant-Metric
0,000
0.000 (18)
(18)
-1.000 (18)
0.000 (18) 0.000 (18) 0.346 (18)
0,343 (18)
0.00
Table 14: The correlations (and metric ranks) for the group-by-systend correlation on the WMT 22 en-de dataset.
Metric
%a
73
Tc
T10
+13
714
aCCpg
acc
Metric-X
0.370 ( 1)
0.420 ( 1)
0.372 (
1)
0.477 ( 1)
0.477
4)
0.477
1)
0.573 1)
0.579 1)
0.02
COMET-22
0.352 ( 2)
0.400 ( 2)
0.354 ( 2)
0.454 ( 2)
0.454
( 5)
0.454
2)
0.564
2)
0.563 2)
0.01
UNITE
0.329 3)
0.374 ( 3)
0.331
3)
0.425 ( 3)
0.425
6)
0.425
3)
0,553
3)
0.553 3)
0.03
COMETKiwi
0.316 4)
0.359 4)
0.318
4)
0.408
4)
0.408
8)
0.408
4)
0.546 5)
0.548 4)
0.02
BLEURT-20
0.316 5)
0.359 (5)
0.318
5)
0.408
5)
0.408
9)
0.408
5)
0.546 4)
0.542
0.00
MS-COMET-22
0.309 6) 0.351 7)
0.311
6)
0.399
6)
0.399 (10)
0.399 6)
0.542 6)
0.539
0.18
UniTE-sre
0,301 ( 7) 0.342( 8)
0.303 7)
0.388
7)
0.388 (11)
0.388
7)
0.538 7)
0.536
0.01
COMET-QE
0.300 ( 8)
0.341 9)
0.302 8)
0.387
8)
0.387 (12)
0.387 8)
0.538 8)
0.536 8)
0.00
MS-COMET-QE-22
0,269 ( 9) 0.305
0.271 (10)
0,347 9)
0.347 (13)
0.347 9)
0.522 9)
0.519 9)
0.00
GEMBA-GPT-335
0.259 (10) 0.332
0.279 9)
0.125 (12)
0.422 ( 7)
0.334 (10)
0.488 (10)
0.484 (10)
0.00
GEMBA-GPT-4
0,245 (11) 0.358 ( 6)
0.262 (11)
-0.046 (14)
0.496 2)
0,316 (11)
0.483 (11)
0.483 (11)
2.00
MEE4
0,185 (12)
0.186 (12) (121
0.238 (10)
0.239 (14)
0.239 (12)
0,481 (12)
0.477 (12)
0.00
HWTSC- Teacher-Sim
0.126
(15)
0.127 (14)
0.163 (11)
0.163 (15)
0,163 (14)
0.451 (13)
0,448 (13)
0.00
REUSE
0.069 (16)
0.078 (16)
0.069 (16)
0,088 (13)
0.088 (16)
0.088 (16)
0.422 (14)
0,418 (14)
0.00
MaTESe
0.128 (13)
0.279
0.140 (13)
-0.523 (15)
0.529 1)
0.165 (13)
0.380 (15)
0.389 (15)
0.00
MaTESo-QE
0.093 (15)
0.229 (13)
0.103 (15)
-06636 (16)
0.493 3)
0.120 (15)
0.341 (16)
0.349 (16)
0.00
Constant-Metric
0.000
0.000 (17)
0.000 (17) -1.000 (17)
0.000 (17) 0.000 (17)
0.225 (17)
0.230 (17)
0.00
Table 15: The correlations (and metric ranks) for the no-grouping correlation on the WMT'22 en-ru dataset.
Metric
T
To
To
710
7133
T14
a0Ccg
acc
Metrie-X
0,239 (1 1)
0.329
(
2)
0.323 ( 1)
0.444 1)
0.445 ( 3)
0.445 1)
0.402 (10) 0.606 ( 1) 0.03
COMET-22
0,230 ( 2)
0.315
3)
0.309 ( 2)
0.420
2)
0.421 ( 4)
0.421 2)
0,396 (11) 0.577 ( () 0.07
UnITE
0,220 ( 3)
0.311
4)
0.297 ( 3)
0.399 3)
0.415 ( 5)
0.412 3)
0.445 8)
0.572 ( 3) 0.05
COMETKiwi
0,181 ( 5)
0.247 8)
0.242 ( 7)
0.331 5)
0.332 ( 9)
0,332 6)
0.372 (13)
0.565 ( 4) 0.05
UniTE-sre
0.180 ( 6)
0.267 7)
0.245 ( 6)
0.322 7)
0.347 ( 8)
0,343 5)
0.463 ( 6)
0.554 ( () 0.07
GEMBA-GPT4
0.155 ( 8)
0.356 1)
0.273 ( 4)
-0.203 (13)
0.519 1)
0,290 8)
0.548 1)
0.550 6) 4.00
MS-COMET-22
0.177 ( 7)
0,243 9)
0.238 ( 8)
0.327 6
0.328 (10)
0,328 7>
0,367 (14)
0.547 ( (7) 2.47
BLEURT-20
0.202 ( 4)
0.291 6)
0.269 ( 5)
0.347 4)
0.369 ( 7)
0.367 4)
0.474 5)
0.540 ( 8) 0.05
COMET-QE
0.148 (10)
0,200 (13)
0.197 (11)
0.262 9)
0.262 (13)
0,262 (10)
0,352 (15)
0.534 ( 9) 0.01
MS-COMETT
COSET-ET--
0.131 (11)
0,180 (14)
0.177 (12)
0.243 (10)
0.243 (14)
0,243 (11)
0.344 (16)
0.528 (10) 3.14
MaTESe
0.075 (14)
0.293 5)
0.224 ( 9)
-0.630 (15)
0.472 ( 2)
0.126 (14)
0.520 2)
0.520 (11)
0.00
MaTESe-QE
0.048 (15)
0,238 (10)
0.168 (13)
-0.728 (16)
0.379 ( 6)
0.085 (15)
0.499 ( 3)
0.499 (12)
0.00
GEMBA-GPT-3S
0.098 (13)
0,201 (12)
0.158 (14)
-0.237 (14)
0.308 (11)
0.196 (13)
0.475 ( 4)
0.494
5.00
WWWTSC-Teachhr-Sii
0.098 (12)
0.147 (15)
0.137 (15)
0.185 (11)
0.199 (15)
0.19% (12)
0,387 (12)
0.481 (14)
0.05
MFF4
0.149 ( 9)
0.222 (11)
0.201 (10)
0.267 8)
0.289 (12)
0.287 9)
0.448 ( 7)
0.473 (15)
0.03
REUSE
.0.76 (17)
-0.090 (17)
-0.097 (17)
-0.119 (12)
-0.099 (17)
-0.097 (17)
0,335 (17)
0.446 (16)
0.14
Constant-Metric
0.000 (16)
0.000 (16)
0.000 (16)
-1.0000 (17)
0.000 (16)
0.000 (16)
0.444 ( 9)
0.444 (17) 0.00
Table 16: The correlations (and metric ranks) for the group- group-by-iten by-item correlation on the WMT'22 en-ru dataset.

---

Metric
T18
714
BCCcy
accag
e²
Metrie-X
0.352 ( 1)
0.401 ( 1)
0.358 ( 1)
0.457 ( 1)
0.457
4)
0.457 1)
0.559
0.569
0.02
COMET-22
0.337 ( 2)
0.384 ( 2)
0.343 ( 2)
0,438 ( 2)
0.438
( 5)
0.438
2)
0.552
2)
0.548 2)
0.03
UniTE
0.316 3)
0.360 ( 3)
0.321 ( 3)
0.410 3)
0.410 6)
0.410
3)
0.541 ( 3)
0.543 3]
0.04
COMETKiwi
0,308 ( 4) 0,350 4)
0.313 4)
0.399 ( 4)
0.399 8)
0.399
4)
0.537
4)
0.538 4)
0.02
COMET-QE
0.299 6) 0,340 6)
0,304( 6)
0,387 ( 6)
0.387 (10)
0,387
6)
0.533
6)
0.529 5)
0.00
BLEURT-20
0.301 ( 5) (
5
0,306 5)
0.391 ( 5)
0.391 9)
0.391
5)
0,534
5)
0.527 6)
0.00
UniTE-sre
0.291 ( 8)
0.332 8)
0.296 8)
0.378 ( 8)
0.378 (12)
0.378
8)
0,529
8)
0.526
7)
0.02
MS-COMET-22
0.296 7) 0.337 7) 0.301 7)
0,384 ( 7)
0.384 (1I)
0.384
7)
0.531 7)
0.524 8)
1.00
MS-COMET-QE-22
0.261
0.297 (11)
0.266 (10)
0.337 9)
0.337 (13)
0.337 9)
0.514
0.505
0.00
GEMBA-GPT-3.5
0,250 (10) 0.321 (10)
0.272 9)
0.113 (12)
0.410 7)
0.324 (10)
0.482 (10)
0.475 (10)
0.00
GEMBA-GPT-4
0.224 (11)
0.327 9)
0.244 (11)
-0.089 (14)
0.460 3)
0.288 (11)
0.471 (11)
0.473 (11)
2.00
MEE4
0,171 (12)
(.195(11))
(12)
0,222 (10)
0.223 (14)
0.223 (12)
0.470 (12) 0.461 (12)
0.00
0.123 (13)
0.139 (15)
0.125 (14)
0.159 (11)
0.159 (15)
0.159 (13)
0.445 (13)
0.439 (13)
0.00
REUSE
0.084 (16)
0.095 (16)
0.085 (16)
0.108 (13)
0.108 (16)
0.108 (16)
0.425 (14)
0.422 (14)
0.00
MaTESc
0.120 (14)
0.258 (12)
0.139 (13)
-0,544 (15)
0.504 1)
0.153 (14)
0.381 (15)
0.387 (15)
0.00
MaTESe-QE
0.089 (15)
0.211 (13)
0.105 (15)
-0.650 (16)
0.466 ( 2)
0.113 (15)
0.345 (16)
0.353 (16)
0.00
Constant-Metric
0.000 (17)
-1.000 (17)
0.000 (17) 0.000 (17)
0.233 (17)
0.242 (17)
0.00
Table 17: The correlations (and metric ranks) for the group-by- correlation on the 22 en-ru dataset.
Metric
T
7b
Fr
T10
F15
T14
B0Ccy
acc
Metric-X
0,362 ( 1)
0.421 ( 1)
0.364 1)
0.489 ( 1)
0.489 ( 1)
0.489
1)
0.551 1)
0.565 1)
0.06
COMET-22
0,361 ( 2)
0.420 ( 2)
0.363 ( 2)
0.488
2)
0.488 ( 2)
0,488
2)
0,551
2)
0.555
2)
0.14
MaTESe
0.295 ( 7) 0.382 ( 3)
0.307 ( 4)
0.244
(12)
0.471 ( 5)
0.399
7)
0,541 ( 3)
0,536
3)
0.00
COMET-QE
0.306 ( 3) 0.356 ( 6)
0.308 3)
0.414 3)
0.414 6)
0,414 3)
0,523
4)
0,520 4)
0.00
COMETKiwi
0.303 6) 0.352
9)
0.304
7)
0.409
6)
0.409 (10)
0.409 6)
0.522 7)
0.520 5)
0.03
BLEURT-20
0.303 5) 0.352 8)
0.305 ( 6)
0.410 5)
0.410 9)
0.410 5)
0.522 6)
0.517
0.00
UNITE
0.305 ( (4) 0.354 7)
0.306 5)
0,412 4)
0.412 7)
0.412 4)
0.523 5)
0.516
0.05
GEMBA-GPT4
0,268 (11) 0.370 4)
0.284 (10)
0.108 (16)
0.486 4)
0.362 (11)
0.513 (11)
MS-COMET-22
0.288 ( 8) 0.335 (10)
0.289 ( 8)
0.389 7)
0.389 (11)
0.389 8)
0.514 8)
0.510 9)
0.02
UniTE-sre
0.286 ( 9) 0.332
0.287 9)
0.386 ( 8)
0.386 (12)
0.386 9)
0.513 (10)
0,508 (10)
0.00
SEScore
0.279 (10)
0.324 (13)
0.280 (11)
0.377 ( 9)
0.377 (13)
0.377 (10)
0121
0.506 (11)
0.00
McTESc-QE
0,251 (13)
0.328 (12)
0.261 (13)
0.047 0116
0.411 8)
0,339 (13)
0,513 9)
0.506 (12)
0.00
GEMBA-GPT-3.5
0.254 (12)
0.360 ( 5)
0.273 (12)
0 ( 3)
0,343 (12)
0.499 (13)
0.499 (13)
0.00
MS-COMET-QE-22
0.238 (14)
0.277 (14)
0.239 (14)
0.322 (10)
0.322 (14)
0,322 (14)
0.489 (14)
0,486 (14)
0.00
NWWTSC-Teacher-Si
0.227 (15)
0.264 (15)
0.228 (15)
0.307 (11)
0.307 (15)
0.307 (15)
0.484 (15)
0.477 (15)
0.00
MEE4
0.163 (16)
0.189 (16)
0.164 (16)
0.220
0.220 (16)
0.220 (16)
0.452 (16)
0.449 (16)
0.00
REUSE
0.100
0.116 (17)
0.101 (17)
0.135 (15)
0.135 (17)
0.135 (17)
0.420 (17)
0.417 (17)
0.00
Constant-Metric
0.000 (18)
0.000 (18)
0.000 (18) -1.000 (18)
0.000 (18) 0.000 (18)
0.260 (18)
0.267 (18)
0.00
Table 18: The correlations (and metric ranks) for the no-grouping correlation on the WM zh-en dataset.
Metric
T
Tc
710
71.3
F14
a0Ccg
acc
Metrie-X
0.191 ( 3)
0.255
5)
0.245 ( 4)
0.343
2)
0.344 ( 5)
0.344 3)
0,389 (10)
0.544 ( 1) 0.06
COMET-22
0.198 ( 1)
0,266
2)
0.255 ( 1)
0.361
1)
0.361 ( 3)
0,361 1)
0.392 9)
0.536 ( 2) 0.15
GEMBA-GPT-I
0.175 ( 4)
0.321
1)
0.252 ( 2)
-0.071
(13)
0.450 ( 1)
0,314 4)
0,518 1)
0.527 ( 3) 4.00
UniTE
0.191 ( 2)
0,261 ( 3)
0.246 ( 3)
0.335 3)
0.350 ( 4)
0,347 2)
0.420 5)
0.516 4) 0.29
MaTESe
0.127 (10)
0.225 7)
0.180 (10)
-0.108 (14)
0.289 ( 8)
0,220 (11)
0.498 2)
0.512 5) 1.00
COMETKiwi
0.159 6)
0,213 9)
0.204 ( 6)
0.288 6)
0.289 ( 9)
0.289 7)
0.372
0.509 ( 6) 0.16
UniTE-SKE
0.164 ( 5)
0.226 6)
0.211 5)
0.293 5)
0.306 ( 6)
0,304 5)
0.407 ( 7)
0.508 ( 7) 0.24
GEMBA-GPT33
0.123 (12)
0.256 4)
0.199 ( 8)
-0.271 (16)
0.377 ( 2)
0.225 (10)
0.494 3)
0.495 ( 8) 5.00
MaTESe-QE
0.097 (14)
0 (12)
0.141 (14)
-0.195 (15)
0.226 (12)
0.169 (14)
0.484 ( 4)
0.494 9) 1.00
COMET-QE
(11)
0.165 (13)
0.159 (12)
0.219 9)
0.219 (13)
0.219 (12)
0.355 (15)
0.483 (10)
0.01
MS- COMET-22
0.144 ( 8)
0.196 (10)
0.187 ( 9)
0.270 7)
0.270 (10)
0,270 8)
0.365 (13)
0.483 (11)
3.49
MS- CB-EEE-T-QU-22
0.117 (13)
0.158 (14)
(13)
0.214 (10)
0.214 (14)
0.214 (13)
0,351 (16)
0.479 (12)
1.99
SEScore
0.158 ( 7)
0,214 8)
0.203 ( 7)
0.295 4)
0.295 ( 7)
0.295 6)
0.371 (12)
0.472 (13)
1.13
HWTSC-Teachhee Teacher-Sim
0.086 (15)
0.116 (15)
(15)
0.148 (11)
0.156 (15)
0.155 (15)
0.357 (14)
0.440
0.14
MFF4
0.137 ( 9)
0.190 (11)
0.177 (11)
0.243 8)
0.256 (11)
0.254 9)
0.393 8)
0.437 (15)
0.07
REUSE
.0.25 (17)
-0.019 (17)
-0.076 (17)
-0.022 (12)
(0.0
-0,010 (17)
0,71
0.420 (16)
0.11
Constant-Metric
(166 (16)
0.000 (16)
0.000 (16)
-1.000 (17)
0.000 (16)
0.000 (16)
0.416 6)
0.416
0.00
Table 19: The correlations (and metric ranks) for the group-by by by-item correlation on the WMT'22 zh-en dataset.

---

Metric
To
110
F18
714
acCag
acc
Metrie-X
0,353 1)
0.411 1)
0.358
1)
0.478 1)
0.478 1) 0.478
0.544
0.557
0.05
COMET-22
0.350
0.408
0.355
2)
0.475
2)
0.475 2)
0.475 2)
0.543
0.546
0.08
MaTESc
0.288 (
0.373 3)
0.300
4)
0.231
(12)
0.462
4)
0.390 7)
0.536
0.530
0.00
COMET-QE
( 3)
0.355 4)
0.310
3)
0.414 3)
0.414 6)
0.414 3)
0.521 4)
0.518 4)
COMETKíwi
0.295 ( 4)
0.343 7)
0.300 5)
0.399 ( 4)
0.399 8)
0.399
4)
0.515 5)
0.516 5)
0.09
BLEURT-20
0.289 ( 6)
0.336 9)
0.293 7)
0.392 6)
0.392 (10)
0.392 6)
0.512 7)
0.507 6) 0.00
McTESe-QE
0,247 (12)
0,323
0.257 (13)
0.405 7) (145
0,333 (12)
0,510 8)
0,506 7) 0.00
UniTE
0,290 5)
0.338 8)
0.294 6)
0.393 ( 5)
0.393 9)
0,393 5)
0,513 6)
0.505 8) 0.01
GEMBA-GPT-4
0.347 5) (1I)
0.269 (11)
0.070 (16)
0.461 (5
0,338 (II)
0.502 (11)
0.504 9)
4,00
MS-COMET-22
0.277 ( 8)
0.322 (11)
0.281 8)
0.376 ( 7)
0.376 (11)
0.376 8)
0.506 9) 0.502 (10)
0.00
SEScore
0.268 (10)
0.311
0.272 (10)
0,362 9)
0.362 (13)
0.362 (10)
0.502 (12) 0.499 (11)
0.01
UniTE-sre
0.276 9)
0.321 (12)
0.280 9)
0.374 8)
0.374 (12)
0.374 9)
0.506 (10) 0.498 (12)
0.00
GEMBA-GPT-335
0.241 (13)
0.345
0.264 (12)
0,022 (17)
0.470 3)
0.326 (13)
0.492 (13)
0,491
0.00
CS-COMETT---22
0.268 (14) 0,231 (14)
(14)
0,312 (10)
(14)
0.312 (14)
0.483 (14)
0.479 (14)
SWWISC-TeacherSSi
0,228 (15)
0,265 (15)
(15)
0,308 (11)
0.308 (15)
(15)
(15)
0.475 (15)
MEE4
0,150 (16)
(16)
0.152 (16)
0,202 (13)
0.202 (16)
0,202 (16)
0.443 (16)
0.441 (16)
REUSE
0.108 (17)
0.125 (17)
0.109 (17)
0.146 (15)
0.146 (17)
0.146 (17)
0.422 (17)
0.419 (17) 0.00
Constant-Mette Mettri
0.000 (18) 0.000 (18) 0.000 (18) -1.000 (18)
0.000 (18) 0.000 (18) 0.265 (18)
0,270 (18)
0.00
Table 20: The correlations (and metric ranks) for the group-by-system correlation on the WMT zh-en dataset.
Metric
%
Tc
710
718
714
a0Cag
a00g
Metric-X
0.239 1)
0.312 2)
0.295 ( 1)
0,402 1)
0.403 ( 3)
0.403
0.439 8)
0.594 ( 1)
UniTE
0,224 3)
0.299
3)
0.275 3)
0.366
3)
0,380 ( 5)
0.378 3)
0.483 4)
0.569 ()
COMET-22
0,230
0.298
4)
0,281 2)
0,380 2)
0.381 ( 4)
0.381 2)
0.433 (10)
0.564 ( 3)
COMETKiwi
0.187 5)
0.242 7)
0.229 ( 6)
0.310 4)
0,311 ( 8)
0.311 5)
0.411 (11)
0,561 ( 4) 0.03
UniTE-sre
0.187 4)
0.262 6)
0,231 5)
0.302 5)
0.324 ( 7)
0.320 4)
0.504 2)
0.547 5)
MS CO-EM2--222
0.180 6)
0.234 8)
0.221 8)
0.300 6)
0.301 ( 9)
0.301 6)
0.405 (12)
0,540 6)
1.14
GEMBA-GPT-4
0.156 7)
0.339 (1)
0.267 ( 4)
-0.237 (12)
0.480 ( I)
0.262 8)
0.524 1)
0.525 ( 7)
4.00
COMET-QE
0.151 ( 9)
0.192 (11)
0.184 (10)
0.241 ( 8)
0.241 (12)
0.241 9)
0.390 (13)
0.513 8)
0.00
MS-COMET-QE-22
0.133 (10)
0.173 (13)
0.163 (12)
0.223 9)
0.224 (13)
0.223 (10)
0.382 (14)
0.512 9)
1.42
MEF4
0.154 8)
0.216 (10)
0.189 ( 9)
0.249 7)
0.269 (10)
0.267 7) 0.487 3)
0.487 (10)
0.00
WWWTSC-Teache-SSS
(11)
0.165 (14)
0.149 (13)
0.196 (10)
0.208 (14)
0.208 (11)
0.435 9)
0.480 (11)
0.02
MaTESE
0.075 (13)
0.278 5)
0.225 ( 7)
-0.646 (14)
0.433 ( 2)
0.114((3))
0.469 ( 5)
0.469 (12)
0.00
GEMBA-GPT-3.5
0.093 (12)
0.179 (12)
0.146
-0.264 (13)
0,261 (11)
0.171 (12)
0.456 6)
0,460 (13)
MaTESe-QE
0.047 (14)
0.223 9)
0.165 (11)
.0.741 (15)
0.354 ( 6)
0.075 (14)
0.441 7)
0.441 (14)
0.00
REUSE
...64 (16)
-0.069 (16)
-0.074 (16)
-0.085 (11)
.0.066 (16)
-0.064 (16)
0.378 (15)
0.401 (15)
0.02
Constant-Metric
0.000 (15)
0.000 (15)
0.000 (15) -1.000 (16)
0.000 (15)
0.000 (15) 0.371 (16)
0.371 (16)
Table 21: The group-by-item segment-level correlations on 2M en-ru using the Unbabel MQM score normal- ization largely follow the results in the main body of this work, which use the Google normalization.


Reviews:
[{"review_id": "c7qvFVtDM6", "reviewer": "Reviewer_7hqG", "summary": "", "questions": "In settings where there are no ties, or at least not as many as the WMT22 MQM languages, how does your proposed metric perform. \nWe are interested to know, would your metric reduce into one of the already used methods in Table 1 or becomes a new metric that needs further justification.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review of our paper.\n\nRegarding your question about when there are no ties: Thanks for this great question. Assuming you mean there are no ties in the human scores, then our proposal is equivalent to tau_10. Importantly, however, when tau_10 was used by WMT, they intentionally removed pairs that were tied according to the human, which we argue is an incorrect thing to do.\n\nWe believe that the tau_eq and acc_eq formulas are already justified in the absence of ties in the human and/or metric scores. They still compute an agreement between the metric and human scores when rewarding correct tie predictions. If there are no human ties, tie calibration will not introduce any ties because any ties will be incorrect and existing ties will correctly be penalized because the human did not believe the pair to be tied, which we believe is justified.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper addresses an important problem, especially now when there are more ties than ever due to higher quality generation models such as LLMs. The paper is well written, easy to follow (thanks to the toy examples), with sound evidence. The proposed meta-evaluation method is easy to incorporate in shared tasks (such as WMT metrics), thus this could lead to a real change in MT meta evaluation community.\n", "reasons_to_reject": "We donot see any manor reasons. If we want to try very hard -- the empirical results are based on 3 language pairs (en-de, zh-en, en-ru on WMT22). Perhaps it'd be interested to see how this works with other languages and annotation methods that are not MQM.\n", "questions_to_authors": "In settings where there are no ties, or at least not as many as the WMT22 MQM languages, how does your proposed metric perform. \nWe are interested to know, would your metric reduce into one of the already used methods in Table 1 or becomes a new metric that needs further justification.", "paper_topic_and_main_contributions": " This paper addresses a blind spot in meta evaluation of machine translation metrics.\nCurrently, meta evaluations (e.g., in WMT Metrics Shared Task) are based on correlation statistics that have a flaw in handling ties assessment scores. This issue is increasingly problematic as generation quality is approaching near human, and some metrics based on LLMs (such as chatgpt) have appear to top the charts.\n \nMain contributions:\n #1) Exposes a blind spot: a) shows how ties can affect correlations and lead to gaming of meta-evaluation. \n #2) Fixes the blind spot: Proposes a meta evaluation metric that is intuitive and fixes the ties issue of #1. Currently, correctly predicting ties lead to no reward in meta-evaluation, however, as described in this paper, instead the correct prediction of ties should be rewarded. \n #3) Sometimes correctly predicting the ties are hard in practice for some auto evaluation metrics. Eg. many recent metrics are designed to output real/float numbers which are not often unequal in the purest sense. This paper introduces a calibration procedure that treats small differences in float numbers as tie, which could lead to increased performance \n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jGuCsu46Tt", "reviewer": "Reviewer_HC2C", "summary": "", "questions": "Question A: Is there a downside to using Kendall’s Tau (given the weaknesses you mention in the paper) as the rank-based correlation statistic for tie calibration? ", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review of our work.\n\nRegarding your suggestion about including BLEURT: We think that is a good suggestion and we will incorporate it into an updated version. We did not include BLEURT in Table 6 because we only considered the primary submissions to the WMT’22 metrics shared task, which does not include BLEURT. We had to use BLEURT in our analysis in 8.2 because we evaluate the metric on WMT’21 and WMT’22, and some of the shared task submissions may have used the WMT’21 data for training or development, which could be a confounding factor in our analysis.\n\nRegarding your comment about using tau for tie calibration: Do you mean a tau that does not correctly handle ties or our proposal, tau_eq, that does?\n\nIf your question is about a tau that does not correctly handle ties, then yes, there is a problem with using that tau with tie calibration. For example, tau_13 does not include ties in its formula. Tie calibration could introduce a tie in order to convert from a discordant pair to a tie, thereby removing it from the formula, resulting in a higher tau. This is clearly not right because the new tie is wrong according to the human scores, but the tau value increases.\n\nIf your question is about tau_eq, then no, there is no disadvantage. The same epsilon value will be found if you run tie calibration with tau_eq and acc_eq. The only difference in the interpretation of the agreement statistic (accuracy between 0 and 1 versus a tau between -1 and 1).\n\nWe will include a discussion of this in the new version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper points out a major flaw with a nearly ubiquitous meta-evaluation metric. The authors included straightforward, and therefore very convincing, motivating examples that make it the use of Kendall’s Tau almost seem silly now. They propose a simple alternative meta-evaluation metric (pairwise accuracy) and include a way to apply this metric to MT metrics that rarely assign ties (tie calibration), making it easy for the MT community to adopt this meta-evaluation metric almost immediately. Additionally, the epsilon “tie calibration” value makes strides towards the interpretability of MT metrics. ", "reasons_to_reject": "This is hardly a reason to reject, but if I HAVE to say something: I think BLEURT should be included in Table 6 since it’s mentioned in Section 7 and you did analysis using BLEURT in 8.2. ", "questions_to_authors": "Question A: Is there a downside to using Kendall’s Tau (given the weaknesses you mention in the paper) as the rank-based correlation statistic for tie calibration? ", "paper_topic_and_main_contributions": "In this paper, the authors address how existing metrics for the meta-evaluation of machine translation (MT) metrics neglect to handle or ineffectively process ties in pairwise evaluations. Using a motivating example, they show that  Kendall’s Tau, the widely used rank-based correlation statistic for MT meta-evaluation, and its variants unfairly penalize metrics for predicting ties. They also show that Kendall’s Tau can be artificially inflated by introducing ties for challenging groups, resulting in the challenging groups being excluded from meta-evaluation. Finally, the authors propose a new ranking-based meta-evaluation metric—pairwise accuracy with ties—and a method (tie calibration) that allows pairwise accuracy to be fairly calculated for metrics that rarely predict ties. ", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 4 could be moved earlier in the paper since it is motivation for the work and not technical. "}, "extra_text": {}}, {"review_id": "PepfUW5TJ1", "reviewer": "Reviewer_FkvL", "summary": "", "questions": "Q1: Can the authors introduce some real examples of the tie problem  which is the core if this work? Indeed, for people who are not very familiar with the tie problem, it is difficult to grasp the idea of the paper.\n\nQ2: As stated by the authors in the Limitation Section,  the tie calibration algorithm uses  absolute differences in metric scores\nI was wondering why not testing on relative scores as well? Does it require more computation? I guess it's a \"simple\" normalization process, so could the authors provide these results as well?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback on our work.\n\nRegarding the evidence on a “fair” comparison: It is true that we cannot prove this comparison is more fair (and we mention this in the Limitations). Our argument for making this claim is that our proposed method addresses bias related to ties, which we do believe results in a more fair comparison.\n\nRegarding the importance of ties and Q1: In Section 5.1 (Figure 2), we include a toy motivating example in which the presence of ties results in an incorrect meta-evaluation of two different metrics with previously proposed taus. In short, incorrectly handling ties can result in erroneous conclusions about which metric is better than another.\n\nTies are a very important problem for metric comparison because incorrectly handling them could lead to unfair comparisons between metrics. The toy example does happen in practice with new metrics like GEMBA that output a large number of ties (see Table 4). Not handling ties will lead to wrong conclusions about these LLM-based metrics. Further, not handling ties opens the door for a metric to cheat the evaluation. In Section 5.2, we introduce the NaN problem, which is a direct result of incorrectly handling ties. We show how a metric could artificially inject ties in order to introduce NaN correlations and increase its performance overall. A metric could have used the NaN problem in the Workshop on Machine Translation’s Metrics Shared Task to improve its overall performance, effectively cheating the evaluation. Correctly handling ties is necessary for making decisions about which metrics agree more frequently with humans than others.\n\nRegarding Q2: The computational cost of using absolute or relative differences for tie calibration is the same. The main difference between them is whether you assume absolute or relative differences are comparable across the full range of scores that can be predicted by a metric. Ideally, metric developers would specify which is true for their metric, and tie calibration would use the corresponding difference. Since the type of difference used is not critical to our work, we will include results about relative differences in the appendix in the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall, the paper is clear and well written\n- Interesting problem which includes fairness in metrics comparison\n- The paper shows improvements over the baselines", "reasons_to_reject": "- It is difficult to grasp evidence on \"fair\" comparison between the different metrics in the paper.\n- The importance of the tie problem may be questionable and it is not clear to me whether this has a significant impact on metrics comparison.", "questions_to_authors": "Q1: Can the authors introduce some real examples of the tie problem  which is the core if this work? Indeed, for people who are not very familiar with the tie problem, it is difficult to grasp the idea of the paper.\n\nQ2: As stated by the authors in the Limitation Section,  the tie calibration algorithm uses  absolute differences in metric scores\nI was wondering why not testing on relative scores as well? Does it require more computation? I guess it's a \"simple\" normalization process, so could the authors provide these results as well?", "paper_topic_and_main_contributions": "This paper addresses MT meta-evaluation by examining the problems caused by ties in Kendall’s τ.\nThe authors propose a  version of pairwise accuracy that grants metrics for correctly predicting ties. \nThe authors also propose a tie calibration procedure that automatically introduces ties into the metric scores. This calibration enables a fair comparison between metrics that do and do not predict ties.\nThe experiments were conducted on the Multidimensional Quality Metrics (MQM) ratings data set of the WMT22 metrics shared task for three language pairs (English-German, Chinese-English and English-Russian).\nThe experimental results show evidence that the proposed method with tie calibration  lead to fairer ranking-based assessments of metric\nperformance.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "For readers who are not very familiar with the tie problem, and for a better understanding of the task, the tie problem should be introduced earlier in the paper via some real examples.\n\nTypo:\n- Line 591 \"also\" is doubled"}}]

Author rebuttal:

Meta-review: The paper shows the failures of Kendall’s tau in presence of ties when evaluation MT evaluation metrics scores and proposes a remedy. The proposed metric introduces predictions of ties and rewards them. With the increase of quality, ties become more frequent. Also ties in WMT have become more reliable, and frequent with the MQM HE scores (#of errors). Some variants of Tau discard ties loosing important information. The analysis of the calibration mechanism for regression-based metrics is very informative, the calibration method does not generalize across dissimilar data sets. 
Reasons to accept: paper  discovers and proposes a fix to relevant problem in MT meta-evaluation. The paper is well written and could have a major impact in the community leading to fairer comparisons across metrics. 
Reasons to reject: Limited language pairs are analyzed but they are very diverse. Evaluation limited to MQM HE framework, which is understandable.
Decision: Accept-Main