Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies (2024)

Abstract: Ten years ago, a single metric, BLEU, governed progress in machine translation research. For better or worse, there is no such consensus today, and consequently it is difficult for researchers to develop and retain intuitions about metric deltas that drove earlier research and deployment decisions. This paper investigates the “dynamic range” of a number of modern metrics in an effort to provide a collective understanding of the meaning of differences in scores both within and among metrics; in other words, we ask what point difference x in metric y is required between two systems for humans to notice? We conduct our evaluation on a new large dataset, ToShip23, using it to discover deltas at which metrics achieve system-level differences that are meaningful to humans, which we measure by pairwise system accuracy. We additionally show that this method of establishing delta-accuracy is more stable than the standard use of statistical p-values in regards to testset size. Where data size permits, we also explore the effect of metric deltas and accuracy across finer-grained features such as translation direction, domain, and system closeness.

heading: 1 Introduction
text: A decade ago, the BLEU metric served as the default metric for machine translation evaluation. It was not without its criticisms (Hovy and Ravichandran, 2003; Callison-Burch et al., 2006; Belz and Reiter, 2006) or compelling alternatives (Banerjee and Lavie, 2005; Popović, 2015), but a combination of adequate performance, robustness to new languages, simplicity, understandability—and also inertia—helped it retain this position. This is no longer the case. BLEU’s deficiencies quickly became apparent as deep learning approaches to machine translation replaced the earlier symbolic
0Paper code: github.com/kocmitom/MT-Thresholds Online tool: kocmitom.github.io/MT-Thresholds
paradigms (Mathur et al., 2020a). Today, a number of metrics—themselves deep-learning based— compete in an ecosystem where there is no longer any dominant, default metric.
This situation creates a problem for researchers working to keep abreast of developments in the field. Different metrics, including different models within the same metric family, have different dynamic ranges, i.e., the range of scores one can expect to see. Furthermore, the metric delta, i.e., the score difference signifying a meaningful change in performance between two systems, also varies across metrics. It is perhaps understandable that some practitioners therefore continue to use BLEU, as well, if only to ground their understanding.
This paper attempts to introduce some order and clarity into this situation. We make use of a large, new human evaluation dataset, ToShip23, to compare the score ranges of metrics on a large number of systems against pairwise system-level accuracy. Importantly, we break down these accuracy scores into bins based on metric deltas, which allows us to determine accuracies for each metric as a func-
1999
tion of the score differences between two systems. This provides a measure of confidence in the output that is stable across testset size, in contrast to standard statistical significant testing, which becomes more stable as testset size grows. We release a tool that allows a user to easily compare accuracies at different threshold across metrics.0
In this work we: §3.2 Empirically investigate the estimated accu-
racy for multiple metrics, human ability to perceive quality difference;
§3.3 Provide thresholds for popular metrics to help reviewers and practitioners interpret results; §4.1 Validate our estimated accuracies on WMT testsets and §4.2 investigate the effect of different language groups; §4.3 Show that string-based metrics, such as BLEU, should never be used to evaluate unrelated systems; §4.4 Show that statistical significance testing is insufficient to determine model improvement especially as it is affected by the testset size, but is important for small deltas; §5.1 Assess quality of automatic metrics over 6530 system pairs; §5.2 Summarize recommendations for machine translation evaluation.

heading: 2 Experimental Setup
text: Data. We perform experiments related to evaluation of MT outputs based on a proprietary dataset ToShip23 which is of a magnitude larger than any publicly available data and enables more fine grained glimpse into the metrics behaviour. The dataset is an extended version of ToShip21 dataset (Kocmi et al., 2021) with details described in Appendix B. We also use data from the annual WMT evaluation campaigns to validate our results, specifically the metrics shared task (Freitag et al., 2022b, 2023), to make results replicable. We only use MQM (Freitag et al., 2021a) and DA+SQM (Kocmi et al., 2022) subset of human evaluated systems because reference-based DA (Bojar et al., 2016) is suboptimal for the evaluation of modern MT systems (Freitag et al., 2022b). See Table 1 for an overview of dataset sizes.
Investigated Metrics. We evaluate the most frequently used metrics in machine translation: BLEU (Papineni et al., 2002), ChrF (Popović, 2015), spBLEU (Goyal et al., 2022), BLEURT (Sellam et al., 2020), COMET (Rei et al., 2020). BLEU and ChrF
are n-gram matching heuristics while the rest uses a parametric model to produce a segment-level score of a translation. CometQE21 and CometKiwi QE 22 are special cases which do not require a reference. We do not include any LLM-based metrics (Fernandes et al., 2023; Kocmi and Federmann, 2023) which are not replicable because of non-publicly available models. Find the specific models, implementation details, and our selection rationale in Appendix A.
Metric Delta. We focus solely on the pairwise system ranking: deciding which system is better based on a system-level score (usually average of all segment-level scores) difference between two systems. We refer to this as metric delta (∆).
Pairwise Accuracy. To test the correlations between automatic metrics and human judgement, we use pairwise accuracy (Kocmi et al., 2021): how many system pairs does the metric rank the same way as humans over the total number of system pairs in the dataset. Formally:
Acc = |sign(metric∆) = sign(human∆)|
|all system pairs| .

heading: 3 Unifying Metric Ranges
text: We first look at the “dynamic ranges” exhibited by different metrics across our datasets. We ground these deltas in human scores by comparing pairwise system-level accuracy at different thresholds of delta. With this, we are able to establish a table of average metric deltas for different accuracy levels, and build a simple model that maps any metric into the unified space of estimated accuracies.

heading: 3.1 Various Ranges for Metric Deltas
text: Figure 1 depicts the distribution of system-level score deltas for various metrics. Some metrics have similar ranges, such as ChrF and BLEU, while others use a much larger score range (Comet20 has ∼5× higher deltas to BLEU) or lower score range (CometQE21 has ∼1/5 range of BLEU).
In addition to the wide ranges of scores, we also observe that metrics do not always have the same direction or agreement with human judgment, which
results in their different performance as measured via accuracy (see Appendix C for more details).
It may be tempting to attempt to bring together these score ranges onto a single scale, say by linear interpolation, perhaps towards BLEU scale. But reconciling metrics by projection is not possible, due to an obvious point: metrics differ not just in the range of their scores, but in their accuracies. To better understand the problem, we look next into what are the implications of different levels of metric deltas. Specifically, we investigate how different delta correspond to humans being able to differentiate systems.

heading: 3.2 Accuracy of Metric Deltas
text: Many factors affect metric behavior: • Each metric weights various phenomena differ-
ently, especially fluency and adequacy (Amrhein et al., 2022; Karpinska et al., 2022). • The reliability of metrics differs when compared to humans (Mathur et al., 2020b; Freitag et al., 2021b, 2022b, 2023; Kocmi et al., 2021). • Reference-based metrics are affected by the quality of human references (Freitag et al., 2023; Zouhar and Bojar, 2024). The pairwise accuracy as usually reported (Kocmi et al., 2021; Freitag et al., 2023) represents a value over the full dataset for all system-pair metric deltas. It does not take into consideration the size of the delta between systems, which heavily affects
the accuracy; that is, whether the metric gap between two systems was large or small. However, this information is important in establishing equivalency of deltas across metrics.
To investigate this, we use a binning approach on the ToShip23 testset. Pairwise system deltas are sorted, and for each delta level, we group the closest 300 pairs into a same bin. For each bin, we plot the mean delta for that bin against the systemlevel pairwise accuracy.1
Figure 2 depicts this information for both BLEU and CometKiwiQE22 . The red line shows that we need around 1.3 BLEU delta to reach 70% pairwise accuracy and 3.5 BLEU to reach 80% accuracy against the human judgments. Because BLEU is not a reliable metric, it never reaches 90% accuracy with humans, even for deltas as high as 6 BLEU points. In contrast, CometKiwiQE22 reaches 90% accuracy already at around 0.9 points and gets close to 100% accuracy past 2 CometKiwiQE22 points.
Our use of fixed-size bins introduces a caveat into the evaluation. Because our data points do not have a uniform delta distribution, the “width” of each bin (defined as the difference between the smallest and largest delta) grows as we move towards larger deltas, where data points are sparser. This width is depicted by the blue line in Figure 2. As we increase the delta, there are fewer and fewer systems with as large delta and thus we need to take system pairs that are farther from the investigated delta. For example, for calculating the pairwise accuracy of 1 BLEU point, we take system pairs with a delta of 1 ±0.1 (half of 0.2), while for 3 BLEU the width of a bin is 3 ±0.25 points. The bin width mainly affects the tail of the evaluation.
As our evaluation is empirical, it is heavily affected by the underlying systems and the lines fluctuate. In the next section, we try to fit a smooth line to abstract the results, followed by discussion which phenomena affect the pairwise accuracy.

heading: 3.3 Aligning Metrics on Accuracy
text: Practitioners might be interested in getting an intuition behind a particular metric delta, e.g., +0.10 of Comet22 and how such delta relates to other metrics that they are familiar with. Clearly, the higher the delta, the more likely that human raters would also notice the quality difference between systems. It remains unclear what delta is enough to warrant
1Appendix D investigates other sizes of bins than selected 300 system pairs.
Estimated Accuracy
Coin toss 50% 55% 60% 65% 70% 75% 80% 85% 90% 95%
acceptance. To this end, we use the estimated accuracy results introduced in previous subsection. As the estimated accuracy line is noisy, we fit a curve through the data and use it to derive thresholds for comparing various metric deltas.
We use a parametrized sigmoid to fit a curve through the data. The choice of the sigmoid function is arbitrary and based on visual similarity and the feature that it converges towards fixed point and thus is bounded. This is a desired feature representing that each metric has a different overall reliability. We parameterize it using two variables φ and fit it with damped least square algorithm (Levenberg, 1944). The function is defined as:
f(x) = φ1
1 + exp(−φ2 · x) .
The resulting fit is visualized in Figure 3. Although not perfect, it offers insight into the metric delta behaviour, specifically comparing different different deltas’ estimated accuracy. We use the sigmoid functions to calculate estimated accuracy for various levels of delta in Table 2. This is the core result of our work and helps in understanding how different metrics compare to each other.
For example, an improvement of 1.06 BLEU has the same estimated accuracy (65%) as the 0.24 CometKiwiQE22 , while 3.35 BLEU has the same estimated accuracy as 0.67 CometKiwiQE22 . And +1 improvement on CometKiwiQE22 signals that in >90% scenarios, human annotators would agree with the ranking of CometKiwiQE22 , while BLEU never reaches this level of agreement. Note that estimated accuracies are empirical from a given ToShip23 dataset. Therefore, we do not claim that +0.56 Comet22 yields 80% accuracy for all scenarios but rather that it is as accurate as +2.34 BLEU.
As these thresholds are combined for all scenarios, we dive in the next section into validating out results on public WMT dataset, followed with investigation of what affects the metric delta and how reliable the comparison is in different settings.

heading: 4 Factors Affecting Metric Deltas
text: We have empirically derived the estimated accuracy for various metrics. In this section, we investigate factors that affect metric delta and show how reliable the thresholds remain under these factors. These include the testset size, dataset and domain selection, and translation direction.
Additional factors could influence the metric delta, but we lack the data to evaluate these aspects. A key consideration is whether the metric delta is
contingent on the underlying absolute values. In other words, we need to determine if a +1 BLEU delta varies in reliability based on these absolute values. For instance, does the impact of moving from 20 to 21 BLEU differ significantly from a shift from 60 to 61 BLEU in different system pairs?

heading: 4.1 Different Domains and Datasets
text: We derived the thresholds from ToShip23. Now, we validate them on WMT data to show how well they transfer. To address the relatively small size of WMT, we first combine the WMT 2022 and 2023 datasets, yielding 1414 system pairs. This dataset contains different set of segment sources and domains, and was evaluated with mix of MQM and DA+SQM human evaluation protocols. In order to test the thresholds, we take scores for all WMT system pairs and convert them into estimated accuracies via devised thresholds. For each estimated accuracy level, we take the closest 300 system pairs and calculate the real accuracy on WMT data. If the mapping would be perfect and we had enough samples, the estimated accuracy would match the real accuracy for each investigated level.
We show the results in Figure 4. In the ideal case, we would expect the real accuracies and estimated accuracies to match; however, the noise from empirical data affects the results. Some metrics are consistently underestimated, such as Comet22, which has higher real accuracies on WMT dataset that the estimated accuracies. On the other hand, CometQE21 has much lower accuracies on WMT data and our thresholds overestimate it.
Overall, the trend is clear and the thresholds normalize all metrics into a shared space of estimated accuracies. Therefore, we advise reporting accuracy when presenting results, together with significance testing and metric delta.

heading: 4.2 Language Pair
text: Notoriously, a large gap in absolute BLEU scores exists between languages (Denoual and Lepage, 2005; Post, 2018). This reflects properties like data sizes, attention progress in different languages, and target-side morphological complexity.
Unfortunately, there is not enough data to examine each language pair individually. Instead, we bin languages into two groups, into-English (XE) and out-of-English (EX) language pairs, which does leave us with enough data in the ToShip23 dataset. In addition, we separate system pairs containing Chinese, Japanese, or Korean (CJK) together.
Figure 5 show the accuracy with a subset of system pairs depending on a languages. There is some fluctuation between XE and EX, but the behaviour is comparable. This is interesting, since most of the underlying testsets have authentic source (e.g., not using testset in reverse direction, Toral et al.,
2018). The CJK group does also perform similarly for CometKiwiQE22 , but not for ChrF. This shows the thresholds are invalid for all metrics and scenarios and are affected by whether metrics evaluate all language similarly or not.

heading: 4.3 Iterated versus Unrelated Systems
text: Another main difference that affects the evaluation is if the systems are closely related. Key point of distinction is between iterated systems (a baseline system against specific improvements, produced by the same research group) or unrelated system (for example, WMT yearly evaluation which comes from different teams and systems produce vastly different translations). It has long been known that surface metrics like BLEU work best when evaluating closely-related iterated systems (CallisonBurch et al., 2006). It may be easier for both metrics and humans to distinguish an iterated system over its baseline, because comparing unrelated systems adds a difficulty of weighting different styles and errors.
To investigate this, we use the system labels of ToShip23 dataset, where some system pairs are baseline model and it’s improved iterated model, while other system pairs are completely unrelated and developed by different teams, similarly to WMT evaluation. Figure 6 confirms the assumption
that unrelated systems are much harder to evaluate and that the metric behaves differently. Therefore, automatic metrics are better to rank iterated systems than unrelated systems. While pretrained metrics, such as CometKiwiQE22 , seems to be robust enough for comparing both types of system pairs, other metrics such as BLEU have much harder time to distinguish unrelated systems. This effect should be investigated to larger detail in future work.
For example, +2 BLEU on iterated models has an accuracy with humans of about 90%, the same +2 BLEU on unrelated systems are barely better than toss of a coin (≈55%). This shows, that some metrics (specifically BLEU, ChrF, spBLEU) should not be used to evaluate unrelated systems. This findings was also suggested by Berg-Kirkpatrick et al. (2012), who showed that you need to get about one third larger BLEU improvements for unrelated systems to reach the same p-value.
Therefore, string-based metrics, such as BLEU, ChrF, or spBLEU, should never be used to compare unrelated systems.

heading: 4.4 Testset Size
text: Another phenomena that may affect the system delta is the number of sentences in the parallel testset used to evaluate pair of systems. Common wisdom says that the testset should be as large as possible. We ask if increasing testset size affects the system delta and its statistical significance.
To examine how testset size affects the metric delta, we take a system pair and sample testsets with increasing number of sentences. For each sample, we calculate CometKiwiQE22 delta and pvalue using paired Student’s t-test (Mathur et al., 2020a). We sample with repetition various testset sizes. For each testset size, we plot the average metric delta (or p-value respectively) over 50 runs together with the confidence interval.
From Figure 7, the metric delta fluctuates but keeps being mostly constant. The variance of the metric delta is higher for small testset sizes (under 500 segments). On the other hand, the p-value associated to the comparison hypothesis goes down simply by having a larger testset, phenomena shown for MT by Berg-Kirkpatrick et al. (2012).
This is a natural phenomenon of statistical significance testing (Greenland et al., 2016). P-values decrease with an increasing sample size, assuming the null hypothesis does not hold. This is due to the increase in statistical power—the probability that the test correctly rejects the null hypothesis when it
is false. Should the null hypothesis hold perfectly, which is rarely the case, increasing the sample size would not systematically affect the p-values. Therefore, it is possible to claim a statistically significant improvement over a baseline model even with a small metric delta, which might not be noticeable by humans, just by using a large-enough testset. This conclusion is not an argument against the use of statistical significance testing, which remains important, especially when observing smaller deltas.
Overall, this shows that metric delta is stable under different testset sizes, while statistical significance testing is affected by it. We assumed to be adding sentences from the same distribution. The metric delta can be manipulated by adding segments that are more difficult than the rest.

heading: 5 Discussion

heading: 5.1 Best-performing Metrics
text: With the ToShip23 dataset, we can also calculate total pairwise accuracy over all system pairs to devise which metrics perform the best on the (to date) largest dataset of MT human evaluation. We follow the same evaluation as in Table 2 from Kocmi et al. (2021). Twice as large dataset than ToShip21, extended by state-of-the-art systems from 2022 and 2023, we can see how metrics perform on system-level rankings. Table 3 shows that the best performing metric over the ToShip23
dataset is CometKiwiQE22 by a small margin over xCOMETXXL. CometKiwi QE 22 is a quality estimation metric, which has an additional bonus of not being affected by reference bias.
Additionally, we notice the overall accuracy dropped for all metrics in the last two years. This does not necessarily signify a drop in metric performance, but may have several other explanations: • Different systems: Newer architectures or sys-
tems are closer to each other in performance, thus harder to evaluate by humans • New testsets: While the 2019-2021 contains only two domains, the newer data have been evaluated
on a much larger set of domains, where some domains may be challenging for metrics • Human bias: The evaluation protocol changed, which may have shifted annotator’s scoring patterns.
However, the absolute pairwise accuracy is less important than the ranking of metrics, as it is heavily affected by the system pairs. We compare to MQM subset of Freitag et al. (2023), which ranks metrics in similar order supporting our findings. There are some notable differences, such as CometQE21 ranking as the worst metric in WMT, while BLEURTdefault is the worst in ToShip23. Since many aspects of the evaluation are different, we do not dive into a comparison, but rather highlight the overall picture. ToShip23 corroborates that QE metrics have reached the quality of reference-based metrics, as well as the (already well-established) fact that lexical-based metrics are not useful for evaluating high-resource MT models these days.

heading: 5.2 Recommendations for MT Evaluation
text: We conclude with a list of recommendations for automatic MT evaluation: • Use CometKiwiQE22 as the main metric. In addition to its better performance, as a quality estimation metric, it is not affected by references. • Use at least one additional metric of a different type; e. g. BLEURT20, which is reference-based and uses a different architecture from Comet. • For each metric delta, report estimated accuracy to help align reliability of used metrics. • Do not use BLEU, ChrF, or spBLEU to evaluate unrelated systems.
In addition, employ caution when using the same metric for evaluation that was used during training, as this practice may lead to artificially inflated results. For instance, it is advisable not to evaluate with the same metric used for Minimum Bayes Risk Decoding (Freitag et al., 2022a), QE metric used for corpus filtering (Peter et al., 2023), or avoid using metrics built on the same model as the translation system because LLMs tend to favor outputs generated by themselves (Liu et al., 2023).

heading: 6 Related Work
text: The closest work to ours is Lo et al. (2023), who investigate the relationship between metric deltas and the p-value of human ranking, concluding that not even 2 BLEU points reliably correspondent to human judgement. This aligns with our work that
two BLEU points reaches an estimated accuracy of only 77.2%. Their work also does not consider the directionality of the delta, and consequently they do not penalize situations where humans and metric disagree on which system is better.
Mathur et al. (2020a) found that even statistical significant deltas of up to three BLEU points do not reliably correspond to human judgement. In a broad survey, Marie et al. (2021) notes that various community “rules of thumb” about sufficient BLEU deltas might be the result of an evolved consensus that has no basis in scientific evidence. Similarly, Kocmi et al. (2021) demonstrated that among system pairs deemed statistically significant by humans and where BLEU disagree with humans, the median delta is 1.3 BLEU. Marie (2022) reinvestigated the WMT 2020 and 2021 results and showed that deltas lower than 2 BLEU needs to be tested for statistical significance.
Automated metrics in NLP and MT have been under scrutiny for a long time. Hovy and Ravichandran (2003) raised early doubts about BLEU. Callison-Burch et al. (2006) pointed to failure modes of BLEU and suggested it be used in more narrow situations. Post (2018) identified a problem with conflicting implementations of BLEU and offered a unified solution. The broader field of computer science has been concerned with what is a meaningful acceptance threshold of a metric (Mori et al., 2018). The acceptance thresholds are usually established to trade off risks in types of errors (Shatnawi et al., 2010). Kelley and Preacher (2012), studying effect sizes in psychology, summarize that effect sizes should be scaled appropriately. Alike, Plonsky and Oswald (2014) ask what effect size suffices and note its dependence on the variance and that all acceptance thresholds are arbitrary.

heading: 7 Conclusion
text: In this work, we investigated the interpretation of deltas from automatic machine translation metrics. Although metrics have different ranges of scores, what ultimately matters to the practitioner is how score deltas are grounded in human ability to perceive those differences, which we judge by pairwise system-level accuracy on a large collection of human judgments. We empirically determined thresholds for popular metrics to align them on accuracy and provide a tool0 that relates metrics to each other. Finally, we showed the importance of using metric-delta accuracy over p-values: the
former is stable across testset sizes. We undertook some investigations into subfactors of the data, showing that the results were robust to, for example, translation direction, and also that they generalized to different testsets. These investigations were limited by the data size. For future work, it would be useful to explore deltaaccuracy for different subsets and combinations of features, presuming that enough data were available for the task.
Limitations
While this work provides more informed guidelines on interpreting metric delta, they remain crude and do not fix the inadequacy of automated metrics. In order to guarantee improvements, human evaluations need to be carried out.
We use humans as a gold standard, however, they are noisy and also unreliable especially for systems that are close in performance.
Almost all MT systems used in this metaevaluation are not based on LLMs. Therefore, we may observe different behaviour of automatic metrics when evaluating LLM-based models.
Our estimated accuracy should not be used as the reason to reject a result, similarly as low significance p-value.
Ethics Statement
The human annotators have been compensated considerably higher than the minimum wage standards in their respective countries. This commitment reflects our dedication to fair labor practices and the well-being of those contributing to our work.

heading: Acknowledgements
text: We would like to thank Arul Menezes, Roman Grundkiewicz, Martin N. Danka, Benjamin Marie and to the Microsoft Translator research team for their valuable feedback.

heading: A Metric Implementation Details
text: There are many automatic metrics that has been developed. In our study, our selection of metrics has focus on either the most used metrics or the best performing ones. Here is the description, reason for their selection and details of implementations used. For metric quality, please, refer to Section 5.1 or Kocmi et al. (2021); Freitag et al. (2023). However, we plan to extend the list of automatic metric even after paper publishing. For other metrics and models, see https: //github.com/kocmitom/MT-Thresholds/.
We are aware that the list is heavily affected by COMET variant models. However, when investigating best performing metrics from Freitag et al. (2023), we can see that most are either based on COMET framework, they are not publicly available, or build on propritetary LLMs.
Out of the lexical-based metrics, we select three of them, which are the most used. However, we emphasize that these metrics should no longer be used for MT evaluation (Freitag et al., 2022b). We use SacreBLEU (Post, 2018) in version 2.3.1 with default setting:
• BLEU (Papineni et al., 2002): the most popular and currently one of the worst performing MT metrics (we used a specific tokenizer for Japanese and Chinese as recommended) • ChrF (Popović, 2015): second most popular lexical-based metric with better performance • spBLEU200 (Goyal et al., 2022): metric popular when evaluating on Flores testset
Two BLEURT models (commit cebe7e6): • BLEURTdefault (Sellam et al., 2020): the default
model when using BLEURT framework called BLEURT-Tiny. It is important to note, that its performance is worse than BLEU (Section 5.1) and should not be used as authors suggest. • BLEURT20 (Pu et al., 2021): the best performing Bleurt model
We evaluate five Comet models (v2.1.0), the most popular metric framework aside BLEU: • Comet20: most frequently used model and the
default reference based model until the end of year 2023. The model name wmt20-comet-da. • Comet22: currently the default referencebased model (wmt22-comet-da), outperforming Comet20. • CometQE21 : we picked wmt21-comet-qe-mqm for its unusual behaviour of using very small delta while reaching high pairwise accuracy. • CometKiwiQE22 : wmt22-cometkiwi-da is the best quality estimation model. • xCOMETXXL: the best performing publicly available metric as evaluated by Freitag et al. (2023).

heading: B ToShip23 Dataset Details
text: For this work, we introduce and analyze an extended version of a non-public ToShip23 dataset. The main changes of a dataset are almost twice as many system pairs as in ToShip21 (Kocmi et al., 2021); more than ten new domains and new parallel testsets; improved human evaluation protocol; and evaluating the latest state-of-the-art MT models.
The parallel testsets for evaluating MT models that we use in the extended part are mostly a collection of non-published human translated sentences. We focus on using testsets in authentic direction, from original source into human translated reference (avoiding reverse testsets whenever possible, Toral et al., 2018). In contrast to ToShip21, which uses mainly two domains (news and speech), we extended the domains by more than ten.
We reduced the total number of languages in the ToShip23 from 101 to 94. The removed languages
∼1 BLEU delta. Each column is one system-pair. Dark background represent metric disagreeing with humans on system ranking. This highlights that normalizing metrics towards BLEU range is not feasible.
are those which are not supported by either BERT (Devlin et al., 2019) or XLM-RoBERTa (Conneau et al., 2020) – language models used in the most popular metrics – therefore, we could not include those languages in our analysis.
The MT systems being part of the dataset are coming from the same distribution as in ToShip21, but evaluating the most recent state-of-the-art models including a limited number of LLM based translations. Lastly, we improved the human evaluation protocol, moved from source-based DA towards DA+SQM (Kocmi et al., 2022).

heading: C Metrics Disagreement on Ranking
text: Automatic metrics often disagree on a ranking which system is better even for large enough deltas. We illustrate this phenomena in this section.
We use the mostly unwritten (and long-debunked (Mathur et al., 2020a)) operating assumption that +1–2 BLEU points denotes a significant finding as an anchor point to illustrate the range of metric deltas on a subset of systems in Figure 8. This figure reports metric deltas for six randomly-selected system pairs from WMT23 data, whose delta was roughly 1 BLEU.
As we can see in Figure 8, while for first two system pairs, all metrics and humans agree on the system ranking, it is not the case for later four system pairs. For example, even Comet20 score of 3.4 (fifth system pair) may result in disagreement with humans.

heading: D Number of System Pairs in a Bin
text: In our work, we fixed the number of systems in a bin for given metric delta to 300 system pairs. We now show how this decision affected our evaluation. To this end, we show various bin sizes in Figure 9. The bin width works as a smoothing parameter. With bin size of 100 system pairs, the curve fluctuates, especially as one system pair transfer into 1% change on the accuracy scale.
We set the parameter to 300 system pairs because that is already a smoother curve, while not too wide so that the epsilon around the investigated delta is also not too high. However, this parameter should be re-investigated in the future works.

heading: Future Work
text: However, this parameter should be re-investigated in the future works