You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation
Authors: Gergely Flamich
David Vilar
Jan-Thorsten Peter
Markus Freitag
Venue: COLM 2025 (2025)
Keywords: translation
accuracy
naturalness
tradeoff
distortion
perception
no-reference metric
TL;DR: We prove mathematically and demonstrate empirically that optimizing a single metric for machine translation *cannot* lead to a system that is both accurate and fluent. We also establish a connection between no-reference metrics and our theory.

Abstract: The goal of translation, be it by human or by machine, is, given some text in a source language, to produce text in a target language that simultaneously 1) preserves the meaning of the source text and 2) achieves natural expression in the target language. However, researchers in the machine translation community usually assess translations using a single score intended to capture semantic accuracy and the naturalness of the output simultaneously. In this paper, we build on recent advances in information theory to mathematically prove and empirically demonstrate that such single-score summaries *do not and cannot* give the complete picture of a system's true performance. Concretely, we prove that a tradeoff exists between accuracy and naturalness and demonstrate it by evaluating the submissions to the WMT24 shared task. Our findings help explain well-known empirical phenomena, such as the observation that optimizing translation systems for a specific accuracy metric (like BLEU) initially improves the system's naturalness, while “overfitting" the system to the metric can significantly degrade its naturalness. Thus, we advocate for a change in how translations are evaluated: rather than comparing systems using a single number, they should be compared on an *accuracy-naturalness plane*.

Published as a conference paper at COLM 2025
You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation
Gergely Flamich"
David Vilar
Jan-Thorsten Peter
Markus Freitag
Imperial College London g. celmmiia@@ieerial..... ac. uk
Google
(vilar, jtp, freitag)@google.com
Abstract
The goal of translation, be it by human or by machine, is, given some text source language, to produce text in target language that simultane- ously 1) preserves the meaning of the source text and 2) achieves natural expression in the target language. However, researchers in the machine translation community usually assess translations using a single score in- tended to capture semantic accuracy and the naturalness of the output simultaneously. In this paper, we build on recent advances in information theory to mathematically prove and empirically demonstrate that such single-score summaries do not and cannot give the complete picture of sys- tem's true performance. Concretely, we prove that a tradeoff exists between accuracy and naturalness and demonstrate it by evaluating the submissions to the WMT24 shared task. Our findings help explain well-known em empirical phenomena, such as the observation that optimizing translation systems for a specific accuracy metric (like BLEU) initially improves the system's 's naturalness, while "overfitting" the system to the metric can significantly degrade its naturalness. Thus, we advocate for a change in how translations are evaluated: rather than comparing systems using a single number, they should be compared on an accuracy-naturalle plane.
1 Introduction
Machine translation (MT) is one of the undeniable success stories of computer science and machine learning research. Its history is long and varied, dating back to at least the 1960s and having progressed through symbolic, statistical and, most recently, neural approaches. By now, the performance of state-of-the-ale systems is close to or matches that of human translators. But how do we assess the quality of translation systems? Indeed, how should we assess them? Already in the early days of MT, researchers recognized the need to evaluate translations along two axes: the ALPAC project considered the " "intelligibilitt"" and "fidelity" of the translations (Pierce & Carroll, 1966), while DARPA evaluations used "adequacy" and "fluency" (White & O'Connell, 1993). Although these concepts are intuitively appealing, they partly fell out of favour for two main reasons. First, they lack solid, widely-accepted formal definitions, which made it 1) difficult to define guidelines for human evaluations, and 2) impossible to establish their theoretical properties and their interaction. Second, in recent years, optimising solely some notion of accuracy where we compare an MT hypothesis with given reference, has already lead fluent translations in practice. Hence, even the relevance of such a distinction has become unclear. Nonetheless, using single metric for automatic translation evaluation has always had issues. For example, for few years now, it has been clear that classical "overlap metrics," " such as BL.UU and chrF are no longer good enough to assess translation quality, as they do not correlate well with human ratings, which prompted researchers to move towards neural metrics instead (Freitag et al., 2022a).
fell
'Work done while a Student Researcher at Google as doctoral student at the University of Cambridge. Now at Imperial College London. Note: the paper title is wordplay on "You cannot feed two birds with one scone," more graceful and less lethal variant of popular English saying.
1

---

Published as a conference paper at COLM 2025
However, in the latest WMT general task, similar phenomenon occurred: systems with the best automatic scores (based on neural metrics) did not achieve the best score among human raters (Kocmi et al., 2024). This and related phenomena motivated us to reexamine translation evaluation practices.
The accuracy-naturalness tradeoff. As note above, at present there is implicitly held belief (or wishful thought) in the community that there should exist some metric, which can be optimized on to yield a translation system that is simultaneously accurate and natural natural-sounding. To examine the plausibility of this belief, we propose formal definitions of accuracy and naturalness by extending the mature, and widely-accepted information- theoretic framework of Blau & Michaeli (2018). Then, we show that under these definitions, the above goal is hopeless: there no metric, optimizing which will yield a system that is simultaneously accurate and perfectly natural-sounding. Specifically, our contributions are:
We extend Blau & Michaeli (2018)'s distortion-perceptio theory to translation, and mathematically prove and empirically demonstrate that there is a tradeoff between the accuracy of translations and their naturalness.
We introduce target-only naturalness assessment, and establish theoretical link between averaging these scores and approximating statistical distances.
We approximate accuracy-naturalnes curves using L.M-ggenrated samples.
Using our theory, we explain the above-mentioned phenomena when optimizing for accuracy metrics: while accuracy and naturalness correlate far from their Pareto- frontier, they begin to anti-correlate close to the frontier.
2 Background and Notation
Assessing translation accuracy. The primary measure of translator's ability / quality is their accuracy, that is, how well their translations capture the meaning of the source text. Therefore, quantifying accuracy necessarily involves comparison between the translator's 's output and a reference. In practice, the reference could be the source text, some translation produced by a professional translator that is taken be ground truth, or both. In this paper, we refer to any metric that performs such a comparison as reference metric. Classic examples include BLEU (Papineni et al., 2002) and chrF (Popović, 2015), which compare candidates with reference translations. They are convenient and simple, because they assess translations on lexical level with the hope that proximity to the reference translation will ensure that the translations are correct at the semantic level as well. In recent years, the community has moved toward neural reference metrics, such as MetricX (Juraska et al., 2024) and COMET (Rei et al., 2020), which aim to assess translations directly at the semantic level instead. These metrics typically aim to directly predict human opinion scores from large language model (LLM) features. They still rely reference translations, but are more robust and hence better suited for evaluating modern MT systems (Freitag et al., 2022a). As a further development, quality estimation (QE) metrics drop the need for reference translation, but as they take the source text as input, we consider them reference metrics too.
In our paper, we identify accuracy with reference metrics and use the terms reference metric and accuracy metric interchangeably. Arguably, neural metrics measure a combination of adequacy and fluency, but in our framework they are nonetheless accuracy metrics because they still rely on a reference translation (or the source sentence in case of QE metrics).
Assessing translation naturalness. Producing fluent translations has always been an ob- jective of translation systems. In the case of MT, early systems struggled to even produce grammatically correct sentences. As the quality of automatic systems improved, this be- came less of an issue, but the community encountered another problem, long known in the (human) translation community: translated text often does not sound completely natural, it contains translationese (Gellerstam, 1986). To study and eliminate this phenomenon, re- searchers have proposed several candidate metrics for naturalness, such as the language model marginal (log- ppobbabbbyyyyytyy (Freitag et al., 2022b; Lim et al., 2024) well linguis- tically motivated metrics as proposed by Vanmassenhove et al. (2021). These scores are also called no-reference metrics as they do not rely on additional data. Note that no-reference
2

---

Published as a conference paper at COLM 2025
metrics are not to be confused with QE metrics, which consider to be reference metrics as we explained above. However, although they are intuitive, no-reference metrics lack more formal mathematical justification. In this paper, we develop abstract notion of naturalness based on information theory, and show how no-reference metrics conform to our definitions, which provides unifying framework.
Terminology and notation. We denote some randomly selected source text by the random variable x E X' and some random target text by y E E with A' ())) the set of all possible source (target) texts. We denote probability distributions by the capital roman letters P, Q and R, and denote the expectation of a function f with respect to distribution P as Pf (x)]. Source-target pairs are drawn from the joint distribution Pay. This joint distribution also immediately defines the marginal over the source sentences Px and marginal over the target sentences Py. For the purposes of this paper, we think of Px as a "true" monolingual distribution over the source language and py/x as the distribution of human translations into
the target language given the source sentence x. This philosophy reflects current practices in machine translation: the source language is typically derived from monolingual corpus, which professional translators then translate into the target language. Indeed, there has translations
been recent explicit effort in the MT community to avoid back-translatee in both directions (Toral et al., 2018; Läubli et al., 2020).
Since we obtain Py by marginalising over the input, it is the distribution over "human translations into the target language." Thus, it will be useful at times to consider a different distribution Ry, which we can think of as a "true" monolingual reference over the target language. Finally, we model translation system as a conditional distribution Qyie- Then,
can also immediately consider the "translator's marginal" Qy(-) Ex-P, (y)xx())))
3 The Accuracy-Naturalness Tradeoff in Translation
One of the key contributions of this paper is extending and generalising the work of Blau & Michaeli (2018), and applying it to translation. This section sets the scenery and then extends their theoretical results, so that they cover the scenarios we encounter in machine translation today. The key theoretical difference between their work and ours is that they develop their theory for data reconstruction. Our setting is fundamentally different: translation is not reconstruction task, the input and output spaces are different.
Reference metrics (accuracy). We start by considering a direct transfer of the distortion metrics defined in Blau & Michaeli (2018). These are functions A : y x ) -> IR that take as input a reference translation y' and a candidate translation y and output a real number that measures how close the candidate translation is to the reference translation. We require that 1) A be bounded from below and 2) A is minimised when yf equals y". Lexical metrics like (negative) BLEU and (negative) ChrF clearly fulfill these conditions, but we need to expand this definition to accommodate modern MT evaluation metrics. First we note that some neural metrics (including QE metrics) take the source sentence as an additional "reference" We thus expand our functional form to A : X x y x y + R. The interpretation of the output value should also be generalized to the degree in which the candidate y' captures the meaning with respect to x and y', and thus it is minimized when y is semantically equivalent to the given reference. As such, a reference metric corresponds to a notion of inaccuracy. Current neural metrics are known or at least assumed to violate this requirement. For example, they might be vulnerable to "universal translations, " which yield low distortion regardless of the reference (Yan et al., 2023). However, as we will see, our theory only requires the requirement to hold "on average," " which is indeed the case for neural metrics. In most cases, the reference metrics the community uses are " "accuracy-like" (higher is better), which can be considered just as a negated distortion, and thus we define the accuracy (or reference-score) of a translation system Qy x as:
A(yyxx) -Exy'~ Ey Oy |A(x, y']l
(1)
Virtually all commonly used translation metrics are designed to be reference metrics, they differ mainly on what arguments they utilize. For example, BLEU and ChrF ignore the
3

---

Published as a conference paper at COLM 2025
source, QE metrics ignore the target reference, and MetricX 'rr COMET use all three. We note that in the community there is an unwritten feeling that neural metrics place a heavier weight on how well particular translated sentence sounds than on the accuracy of the translation. However, so long as there is an element of comparison with a reference, and assuming that our assumptions are satisfied, neural metrics are still reference metrics.
No-reference metrics (naturalness). Following Blau & Michaeli (2018), we argue that the naturalness of translated text should be intrinsic to it, and should not depend on the source text. The basis of our thesis is the following desideratum: a set of translations ought t to be perfectly natural if no observer critic can reliably distinguish them from a monolingual corpus in the target language. More precisely, let Qylx denote a (possibly and usually stochastic) translator. Then, we consider the "translator' 's marginal" Qy(-) Ex-Pa (Q)xx()), which is the output distribution we obtain when we randomly select input Px and translate it: y Qylx. Then, we say that translation system Qy'x is perfectly natural with respect to monolingual reference distribution Ry in the target language, if the system's 's marginal matches the reference: Qy = Ry. Crucially, it is up to the practitioner to define Ry. Of course, one could set Ry Py to the marginal defined by our parallel dataset; however, this is not necessary and indeed there are many good reasons not to! For example, the reference translations might contain "human translationese" (Gellerstam, 1986) and it is usually better to assess the quality of the system against text that was conceived in the target language. Furthermore, translation systems for different domains all require different notions of "naturalness" due, e.g,, to the specific terminology that they use. We might also wish to select Ry such that its support is filtered for unwanted content, such as slurs/curse words.
In practice, requiring perfectly natural translations is too stringent and hence we relax it such that Qy R Ry instead, which we formalise using statistical distances. Concretely, we pick some divergence D(Q, R), that is, D(Q, > 0 for any two distributions Q, R and D)Q,R) = O if and only if Q = R. However, similarly to distortion, a statistical distance measures the unnaturalness or disfluency of system Qylx- Hence, we define the naturalness with respect to a statistical distance D and reference distribution Ry as the negated distance
N(Qy x) = Ry),
(2)
What distance D should we choose? Natural choices of D include integral probability metrics (IPM) such as the total variation and Wasserstein distances, and f-divergences, such as the Bullback-Leibler or Rényi divergences, as these families all have natural connections to distinguishability (Sriperumbudur et al., 2009); see Appendix A for a discussion.
Finally, we emphasize that a key feature of our theory to recognize naturalness as "corpus-/ dattsse--eetss-eeee property. As such, the theory has no notion of "sentence-level"" naturalness, In other words, our theory concerns itself with the naturalness of a translation system Qylx, but not with the naturalness of individual text segments.
3.1 The Tradeoff
Having defined accuracy and naturalness, we now study their interaction. First, observe that the two objectives are distinct: we can construct system that is perfectly natural, but highly inaccurate by setting Qylx < Ry. Such "translation system" ignores its input and outputs some randomly chosen text from the reference distribution Ry. It would be a system with perfectly natural outputs, but utterly useless.
However, if we have a system that is optimally accurate, can we ever expect it to be perfectly natural? To answer this, consider neural metrics, such as MetrieX or COMET, which are trained so that they implicitly jointly assess adequacy and fluency (in the MT sense). Concretely, let A be neural metric we obtain by optimizing some appropriate objective. Now, consider a translation system Qy that minimizes A ie., AQQ) ming, A (Qy)x) By definition, 0%/x is optimally accurate, but when is it also perfectly natural? Letting (+(-) Ex-P, 10 (-)], we see that by our definition, Q is perfectly natural when
In their latest version.
4

---

Published as a conference paper at COLM 2025
D(Q,, 0, which occurs if and only if Q3 Ryy Due to the flexibility of our framework, this allows us to construct perfectly natural system from perfectly accurate one: we just need to set the monolingual reference distribution Ry + Qy! However, this choice makes Ry, and hence our notion of naturalness depend on Pxy, which we argue is problematic for assessing translation naturalness. Our philosophy is that the notion of naturalness in a given language Y/ should not depend implicitly or explicitly on another language X. Otherwise, we arrive at the absurd conclusion that the naturalness in language y is different when translating from language X2 into ) and when translating from language X into Y However, Qÿ breaks our principle in two ways: 1) it depends on Qyjx which in turn depends on Pxy and 2) it depends the marginalisation. On the other hand, setting the reference distribution Ry to anything other than Q3 derived from a minimizer of A means that 1 will not be perfectly natural. This argument illuminates that if we follow
the philosophy that the monolingual reference Ry should not depend on Pxy, then the degenerate case of perfect accuracy implying perfect naturalness is exceedingly unlikely in practice. However, we emphasize that this conclusion relies on explictly disallowing Ry to depend on Pay, the motivation for which comes from the theory's 's application to translation. In some other scenarios allowing such dependence might be sensible: for example, in Appendix B, we generalise Theorem 1 of Blau & Michaeli (2018), which does allow Py-dependence and admits a similar
result.
The above argument guarantees that there is almost always tradeoff between accuracy and naturalness, but gives no estimate of its severity. Indeed, we might expect the tradeoff to be mild for language pairs with almost one-to-one correspondence, such Spanish-Catalee or Serbian-Croatin On the other hand, the tradeoff should be more pronounced for more Hungarian-Swahitt We quantify these tradeoffs in Section 5, specifically in Figure 3. Hence, to study the tradeoff, we define the accurac--naturalnnss (AN) function2
distant language pairs such German-Japanese
(NN) = max ((Q subject to - D(Qy, Ry) N.
(3)
Qy/x
A(N) represents the best accuracy we can ever expect for a translation system, subject to achieving naturalness of at least N. We have the following result, analogous to Theorem 2 of Blau & Michaeli (2018); see Appendix for the proof.
Theorem 3.1. Let ANN) be the accuracy-naturalln distance D, parallel distribution Pxy and monolingual reference Ry. Then:
function defined by distortion A, distributional
1. A (NN is non-increasing in N.
2. if D is convex in its first slot, then A(N) is concave in N.
That A(N) is non-increasing in N, taken together with our argument above that tradeoff exists for virtually all practically relevant cases implies that close to the curve, accuracy and naturalness must anti-correlate. That is, an increase in the naturalness of a system beyond certain point must come at the sacrifice of some accuracy, and vice versa. The concavity of A(N) in N, on the other hand, means that we can expect larger and larger degradation in naturalness for a small gain in accuracy and vice versa, especially at the extremes.
Example. To make the inevitability of the tradeoff more concrete, consider translating the German sentence "Meine Lehrerin ist nett" to English. A sensible translation would be "My teacher is nice," which sounds natural, but loses the information from the German sentence that the teacher is female. An alternative translation could be "My female teacher is nice," " which is now fully accurate, but sounds quite unnatural in English. Another, naturally occurring example of this phenomenon is the translation of Japanese honorifics. A completely accurate translation into English would need to introduce diverse paraphrases denoting the social relation between the speakers, which would result in unnatural text. This type of strict semantic differentiation is currently not explicitly assessed by any metric that we are aware of, but might be relevant in certain scenarios.
2The accuracy function is related to the better-known distortion-percephie function ((P) ming,, (A(Q s.t. D(Qy, Ry) & P (Blau& & Michaeli, 2018) via 6(P) -A(PP).
5

---

Published as a conference paper at COLM 2025
4 Translation evaluation in practice
Before moving to the empirical analysis, we lay out the basis for a practical evaluation pipeline. Thus, let X = [x,, X2,..,,,.,, x and y = y2... yN be a parallel test set, with source-reference pairs drawn iid: y; Pxy. Furthermore, let Ry be a monolingual ref- erence distribution in the target language, which we could obtain by estimating it from some monolingual prrpu..Finally. we consider set of M translation systems Q = (y Oxk) to be evaluated. Following standard practice, we wish to compare models based on their outputs. To this end, we require that each system Q y/x Q translates the source texts in the test set and produces translation set Y = {yn Q
y/kr Xn E x). Our goal then is to estimate the accuracy and naturalness of each system using X, Yref, Ry and the Y
4.1 No-reference metrics statistical distances
So far, we followed a bottom-up approach, and developed our theory of accuracy and naturalness from the ground up. In this section, we switch top-down view, and ask whether current evaluation practices for assessing naturalness conform to our theory. The prevailing practical approach for data quality assessment are no-reference metrics. These can be either human-generated, such as the fluency scores derived from MQM assessments for text² (Freitag et al., 2021), automated, such as the perplexity of a language model for text. We formalise no-reference metrics by assuming we have a set of critics C = {f1, f2...-fk (usually with K 1), each of which assigns score to each set of translations Y"! Then, for each fx C, we compute the score st N Eycr" f and average the result get the
final score sM --- E s... Within the context of MQM assessment, the critics could be the set of human raters evaluating the output of a translation system. Then, ft (y) could be the kth rater's 's fluency score for the hypothesis y in Y" derived from their MQM evaluation according to the conversion scheme of Freitag et al. (2021). Does this evaluation procedure fit our definition on naturalness? If so, how? As an answer to this question we propose the following statistical distance between two distributions Q and R:
(2 R P) = Ey-p [[x))]f(X) Ey=R[f(Y)]]
(4)
where P is probability distribution over critics (technically, it is a stochastic process). Note the similarity of Equation (4) to integral probability metrics (Müller, 1997): instead of maximising, we average over critics instead; see Appendix A for a detailed discussion. Formally defining D, is somewhat technical, hence we relegate the details of the definition and the analysis of basic properties to Appendix E. Here we argue that averaging no- reference scores is, in fact, rough Monte Carlo estimate of an appropriately chosen D1. Indeed, assuming that the critic scores are all non-negative and are such that the reference distribution Ry always achieves perfect score for any f ~ P, that is, EY [f() 0,
then for a test set Y" we have (i Ry | PP) R (KN)- I Eyer fy(y) with fa ~ p, which is precisely the score we obtain from averaging no-reference metrics.
idd
Experiments
In this section, we empirically verify our theory. We based all our experiments on publicly available data: the submissions and human MQM evaluations of the WMT24 general task
5.1 Approximating the Acturacy-Naturalness curve using LLMs
First, we aim to get a basic feel for the accuracy-naturalnes curve. Unfortunately, per- forming the optimisiation in Equation (3) is hopeless. Hence, we approximate it in the
SStrictly speaking, MQM evaluation of fluency is not completely reference-free, as the judges have access to the source sentence. But fluency-related scores are judged mainly on a monolingual level. 4 Available at https: //gi thub. cwm/ www.nfeeeenee/mt2ee news- news-systems systems.

---

Published as a conference paper at COLM 2025
Gemma 2B log: perplexitty
-3.6
Gemma 2B log perplexity
-3.6
-3.8
-3.8
Seit meiner Kindheit
träume ich vom Fliegen.
-4
-..
Ich wollte schon als Kind fliegen.
-44
-44
60
62
64
0.78
0.82
0.84
0.86
chrF
Comet
Figure 1: Approximation of the accuracy naturalness curve on the WMT24 en -> de test set. Left: chrF vs LM log-perplexity. Right: COMET vs LM pog-perplexity. To further illustrate the tradeoff, we add selected translations for the test sentence "I've wanted to fly since was child." The translation with higher COMET score is fully accurate translation of the sentence, while the other translation is less accurate but more fluent according to our metric.
following way: using Gemini 1.5 Flash (Gemini Team et al., 2024), we generated K 1024 candidate translations { for each source sentence x; in the WMT24 en - de test set; see Appendix F for the precise details. Next, we chose to two accuracy metrics A to test for: chrF (Popović, 2015) and COMET (Rei et al., 2020). For the naturalness metric, we used the pretrained Gemma2 2B model log-perplexities (Gemma Team et al., 2024). Concretely, let T(y) denote the probability a sequence of tokens as predicted by the Gemma2 model, let lyl denote the number of tokens in and hence let LPP Q) Ey [- log ({))/ denote the log log-perplexity of a distribution Q with respect to r. Then, our concrete instantiation of D1 is D(Q, R) [LPP(() LPP(R) [-] As we show in Appendix E.1.2, this quantity can also be viewed as more general version of the D1 distance. We estimated the monolingual target reference distribution Ry, using 20,000 entries of the 2024 NewsCrawl DEU Monolin- gual dataset". Of course this choice for Ry represents only a specific type of language (that of news text), and does not represent "colloquial German" for example. Depending on the application, Ry could be adapted by using a different dataset; in our case it merely serves an illustrative purposes In our experiments, we found that LPP(Ry, was always smaller than LPP Q") ), for each system we examined, hence D is equivalent up to an additive constant to the average model log-perplexity on the test dataset D(Qy, Ry) LPP(Qy) + O1)). To optimise Equation (3), we first construct the objective's "one-shot Lagrange dual" with tradeoff parameter B:
B V
C(x,y,y' = Axx,y,)) +
log [ty).
55)
Then, for each entry x in the WMT test set, we computed the optimal (aka oracle) translation for a large range of betas B E [10-4, : y; = arg maX [(x, yi,y ß). Finally, we report the average of the metrics over the best translations selected:
A
N Σ yi),f,
D" N
Σ
y? logT
(6)
We plot the results in Figure 1 and see that it verifies our theory: 1) there indeed tradeoff curve (see also Theorem B.1) and 2) the curve is non-increasing and concave, as Theorem 3.1 predicts. Finally, note that curves produced this way overestimate the true AN function and hence are overly optimistic, see Appendix F for the details.
5.2 Evaluating existing translation systems using the accuracy-naturalness framework
The plots of the previous section depict a simulation of the tradeoff curve through a con- trolled experiment, but how do real systems behave with respect to the ideal curve? In
5 Available https://datt /tdatt..ttatt.......m// wttp://wwwwwwww/////eeea//eseeee
7

---

Published as a conference paper at COLM 2025
-3.5
-3.5
Gemma 28 log perplexity
Gemma 2B log perplexity
-4
-4.5
-4.5
56
58
60
62
64
66
68
70
0.78
0.8
0.82
0.84
0.86
chrF
Comet
MOM Fluency
MOM Fluency
-1.5
-1.5
56
37
58
59
60
61
62
0.8
0.81
0.82
0.83
0.84
0.85
chrF
Comet
Claude-3.5
CommandR-plee
Gemini-L.5-Pro JOL, Research
Llama3-708
Mistral-Lange Unbubel-Tower7788 Oracle A-N curve
Dubformer
IKUN-C
TranssionMT
ONLINE-B
ONLINE-W
Human Reference
Figure 2: System performances the WMT24 -> de test set. marks represent LLMs, marks represent systems specifically trained for MT, A marks represent "online" systems and the represents the performance of an alternative human reference. For further explanation of these categories, please see Kocmi et al. (2024). Top row: "Fully automatic" accuracy-naturalness comparisons using chrF and Comet as accuracy metrics and Gemma2B log-perplexity (see section 5.1) I) as the naturalness metric. We also include the approximate AN curves from Figure 1, though note that it is optimistic estimate of the real curve. Bottom row: comparisons of automatic accuracy metrics (chrF and Comet) against human MQM fluency ratings, as described in section 5.2.
the top row Figure 2, we include points for the systems that participated in the shared task in addition to the previously computed curve. All systems lie below the curve, as expected. Furthermore, the best performing systems in the evaluation (according to human judgement) are the ones that are closer to the curve. In order to investigate this assertion in more detail, the bottom row of Figure and in Figure we compare human fluency scores against automated accuracy metrics and human adequacy scores for those languages for which MQM evaluations are available. Note that in this case we cannot compute (or simulate) the tradeoff curve for human judgements.
We different aspects that confirm our theory. A first observation is that for "low- performing" systems (the ones farther away from the (0,0)-origin, which would represent optimal performance), accuracy/ recuracy/adequacyy adequacy and fluency do correlate, but as the quality of the systems improve ("as we come closer to the curve") there is an anticorrelation. From comparing the plots in the bottom row of Figure 2, we see that Comet, a neural metric, correlates with fluency "for much longer," that is, its accuracy-naturalnes tradeoffs should be much milder. Nonetheless, our theory shows that the tradeoff exists here too, and therefore at some point these will begin to anti-correlate with fluency well. The underlying phenomenon is the same one that explains why BLEU (a pure accuracy metric)
6See Appendix D for details on the computation of fluency and adequacy scores from MQM evaluations.
8

---

Published as a conference paper at COLM 2025
en de
00
2b
MQM Fluency
-05
Dubfermer Upioly Tower20B
MQM Fluency
Dubformer
GPT-
MQM Fluency
DEUT-CTCOM
-0.1
-1
Unbabell Tower70B
Chagee 3.5
-033
-02
-06
MQM Adequacy
MQM Adequacy
MQM Adequacy
Figure 3: Plot of the accuracy-naturalness tradeoff derived from human MQM adequacy and fluency ratings, as described in Section 5.2. We highlight the top-performing systems at the Pareto frontier.
is no longer an adequate metric for system performance. We also observe a similar effect for different language pairs in Figure 3: the en- and ja-zzh systems exhibit a more pronounced accuracy-naturalness trade-off, while it is milder for en-tes systems.
Secondly, we focus on interpreting the results for the performance of single systems. Unbabel (Rei et al., 2024) had the dominant system in the automatic metrics, but in human evaluations other systems were judged better. Unbabel's system was clearly optimized for COMET, an accuracy measure. Looking at the plots in Figure 3 we can see that the Unbabel system has quite strong adequacy score, but due to the tradeoff the fluency of the system suffers. Other systems that human raters judged to be better lie closer to the tradeoff curve.
Related Work
Already in the early days of machine translation the need of evaluating across two dimen- sions was recognized (Pierce & Carroll, 1966; White & O'Connell, 1993). These were also the preferred metrics for many of the NIST MT evaluation in the early 2000s until there was shift to single HTER scores (Snover et al., 2009; Habash et al., 2011). A similar trend can be found in the WMT evaluations. Whereas in the first editions adequacy and fluency was used (Koehn Monz, 2006), this approach was quickly dropped in favor of ranking approaches (Callison-Burch et al., 2008) and later single direct assessment score was given to each segment (Bojar et al., 2016). The use of MQM scores (Kocmi et al., 2022) allows for a distinction into adequacy and fluency evaluation, as we have done in this paper, but these type of evaluation is not available for all conditions and systems. All usual automatic metrics for machine translation evaluation are accuracy metrics as defined in Section 3. To the best of our knowledge there is not much work around intrinsic automatic metrics for the evaluation of the fluency of translations. Vanmassenhove al. (2021) propose series of statistical measures and show that they show different characteristics for machine translated and human produced texts, but these do not constitute statistical distances we require here. There has been more attention into classifying translations into naturally occuring texts or translationese, which could be considered a measure of naturalness e.g. (Kurokawa et al., 2009; Lembersky et al., 2012; Freitag et al., 2022b).
Similar to our work, Lim et al. (2024) also recently identify tradeoff between adequacy and fluency, in their case identifying adequacy with the conditional and fluency with the prior probabilities in a source-channel formulation of the translation problem. Our approach is more general, as we show that the tradeoff exists for every possible accuracy metric considered. They also argue for reintroducing explicit adequacy and fluency guided do. Orthogonally, Elangovan et al. (2024) recently established a basis for claiming whether a particular automatic metric represents human judgements well.
evaluations,
As we have noted on several occasions, work is heavily inspired by the work of Blau & Michaeli (2018). Before its publication, the neural image restoration and compression community was grappling with the analogous issue and widely held the same miscon-

---

Published as a conference paper at COLM 2025
ception the MT community holds now: despite developing ever-more sophisticated, so called "perceptual distortion metrics" that were inspired by/ by/based on the human visual system, and which correlated well with human judgments, systems optimized using these metrics inevitably showed undesirable visual artefacts. The community's response to this was that "we just haven't found the right metric yet," and spent considerable effort solving this issue. Blau and Michaeli's work showed that this effort is hopeless, and since then evaluating the performance of image reconstruction and compression algorithms along plane has become standard practice. We hope our work will help
the ceatoottion-prcepttine bring about a similar change in the MT community.
The original idea has since been significantly extended and analysed in the context of data compression, see Blau & Michaeli (2019); Theis & Wagner (2021); Chen et al. (2022). We extend the framework by considering the translation setting and allowing to choose the reference distribution Ry- One important aspect we left mostly unexplored, but which is an important next step once the tradeoff is recognised, is optimising it directly. In the image compression community, this has been recognised for a while, with many state- of-the-art compression models trained using a mixture of accuracy and naturalness (via adversarial losses), see e.g., HiFiC (Mentzer et al., 2020). However, adversarial approaches have recently been far less prominent in the text generation space, due to the difficulty in pack-propagating through the generated text. Indeed, finding even the right framework for optimising no-reference metrics is an open problem (Theis, 2024). In the field of machine translation and text generation there has been some work on optimizing several objectives concurrently, e.g. (Duh et al., 2012; Kumar et al., 2021). Similar frameworks can potentially be used to optimize for a specific tradeoff of adequacy and fluency.
7 Discussion
In this paper, we showed that there is an inherent tradeoff in the accuracy and the natural- ness of translations in virtually any practical scenario. We achieved this by extending and generalising Blau & Michaeli (2018)'s distortion-perreption tradeoff to translation. Further- more, we established connection between no-reference metrics and statistical distances, which not only justifies our approach, but also the use of metrics such as those used on Blau & Michaeli (2018) and popular no-reference metrics used in image reconstruction such as LPIPS (Zhang et al., 2018). We empirically demonstrated the tradeoff by approximat- ing the true AN curve using LLM-generated candidates and computing oracle scores on them. Finally, we showed how the accuracy naturalness tradeoff can bring more light into the interpretation of the results of evaluation campaigns, by plotting the submissions to plane with respect to both
the WMT24 general shared task on the recuracy-naturally naturalness metrics.
automatic and human MQM-based accuracy
Modern neural MT metrics pose an interesting question with respect to our theory. As they use either reference translation or the source sentence for comparison when judging candidate translations, they are instances of accuracy metric as we have defined them in this paper. But the clear consensus in the community is that they also have a strong fluency component. Our work can help to provide a better interpretation tion as what these metrics are judging, and what effect optimizing for them has on the performance of MT systems.
Although this paper is mainly theoretical in nature, we posit that it can have important consequences in practice. Our first and foremost goal is to make the community aware of this tradeoff. We suggest to expand future evaluations to explicit consider again a distinction between accuracy and fluency (or similar dimensions) when evaluating MT systems.
Similar as in the evolution seen in the image reconstruction community, translation systems can be optimized and evaluated with respect to this tradeoff. Different applications may have different optimal points along the curve: while factual texts (e.g. scientific or legal documents) naturally favor adequacy, literary translations may benefit from more natural flow of the produced text. Having the possibility of steering the behaviour of the systems can improve their flexibility and improve the final quality of translation systems.
10

---

Published as a conference paper at COLM 2025
Reproducibility
The experimental section of this paper is mostly based on publicly available data produced in the WMT24 evaluation campaign. The biggest source of variability is sampling of Gemini, which by its own nature is a stochastic process, and has an additional dependence on the actual version of the model. However, we are studying the general behaviour of the curve, rather than the actual values, the conclusions will still be valid for different samples or model updates.
Acknowledgements
We thank Eleftheria Briakou for her detailed comments that helped improve earlier version of our manuscript. We also thank Colin Cherry, Géza Kovács, Dan Deutsch, Arthur Gretton and Deniz Gündüz for fruitful discussions that helped us formulate our arguments more clearly and effectively.
References
Yochai Blau and Tomer Michaeli. The perception-distortion tradeoff. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6228-6237, 2018.
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The rate-distortion- perception tradeoff. In International Conference on Machine Learning, pp. 675-685. PMLR, 2019.
Ondłej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Christian Bojar, Ondfej andT Buck, Rajen Chatterjee, Christian Federmann, Liane Guillou, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Aurélie Névéol, Mariana Neves, Pavel Pecina, Martin Popel, Philipp Koehn, Christof Monz, Matteo Negri, Matt Post, Lucia Specia, Karin Verspoor, Jörg Tiedemann, and Marco Turchi (eds.), Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653 (30653/v1/ http//////ww///// URL https://aclanthoolgg.oorg
2301/.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. International Conference on Learning Representations, 2020. URL htttps://operreeee www.ee.wee/eee www.:/eeeeffommiiiim (ttt///rmmmmddgggaaVeeee.e
Chris Callison-Burch. Cameron Fordyce, Philiipp Koehn, Christof Monz, and Josh Schroeder. Further meta-evaluation of machine translation. In Chris Callison- Burch, Philipp Koehn, Christof Monz, Josh Schroeder, and Cameron Shaw Fordyce (eds.), Proceedings of the Third Workshop on Statistical Machine Translation, pp. 70-106, Columbus, Ohio, June 2008 wrrg/00-0009/.
Association for Computational Linguistics. URL https: http://///lclaahoooooooo
Jun Chen, Lei Yu, Jia Wang, Wuxian Shi, Yiqun Ge, and Wen Tong. On the rate-distortion- perception function. IEEE Journal Selected Areas in Information Theory, 3(4):6644677 2022.
Thomas M Cover and Joy A Thomas. Elements of Information Theory. John Wiley & Sons, 2012.
Kevin Duh, Katsuhito Sudoh, Xianchao Wu, Hajime Tsukada, and Masaaki Nagata. Learning to translate with multiple objectives. In Haizhou Li, Chin- Lin, Miles Osborne, Gary Geunbae Lee, and Jong C. Park (eds.), Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1-10, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https //wpaaaoaaaann....../ org/P12-1001/
11

---

Published as a conference paper at COLM 2025
Aparna Elangovan, Jongwoo Ko, Lei Xu, Mahsa Elyasi, Ling Liu, Sravan Bodapati, and Dan Roth. Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and llm-as-a-judge. arXiv preprint arXio:2410.03775, 2024.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, Lingusttss 9:1460- 1474, 2021. doi: 10.1162/tach aacclaa00040500 a_003 URL https: http:::///claanoooggy..ooo www.:tthhooooooooooo/.ogo 87/.
1tt tacl-1.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. Results of WMT22 metrics shared task: Stop using BLEU neural metrics are better and more robust. In Philipp Koehn, Loic Barrault, Onditj Bojar, Fethi Bougares, Rajen Chatter- jee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), Pp. 46-68, Abu Dhabi, United Arab Emirates (Hybrid), December 2022a. Association for Computational Linguistics. URL https://aclanthol.ogg org/2022 wmt-1.2/. 1.2/.
Markus Freitag, David Vilar, David Grangier, Colin Cherry, and George Foster. A natu- ral diet: Towards improving naturalness of machine translation output. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 3340-3353, Dublin, Ireland, May 2022b. Asso- ciation for Computational Linguistics. doi: 10.18653 / /1122222.22222/// w1t2002.findingssaall. findings-acl.265 URL https://aclanthol.ee wwwp:/o2...o..o..o..o.o fwndingsnnnnnn acl. 263/.
Martin Gellerstam. Translationese in swedish novels translated from english. Translation studies in Scandinavia, 1:88-95 1986.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love, Paul Voigtlaender, Rohan Jain, Gabriela Surita, Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Komnaphop Kawintiranon, Orhan Firat, Yiming Gu, Yujing Zhang, Matthew Rahtz, Manaal Faruqui, Natalie Clay, Justin Gilmer, JD Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki Morioka, Kevin Hui, Krishna Haridasan, Victor Campos, Mahdis Mahdieh, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de Liedekerke, Siddharth Goyal, Paul Barham, DJ Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lepikhin, Michela Paganini, Xavier Garcia, Fan Yang, Dasha Valter, Maja Trebacz, Kiran Vodrahalli, Chu- layuth Asawaroengchai, Roman Ring, Norbert Kalb, Livio Baldini Soares, Siddhartha Brahma, David Steiner, Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael Lopez Kaufman, Laurent El Shafey, Junhyuk Oh, Tom Hennigan, George van den Driessche, Seth Odoom, Mario Lucic, Becca Roelofs, Sid Lall, Amit Marathe, Betty Chan, Santiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Lenc, Chih-Kuan Yeh, Aakanksha Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, Anastasia Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin, Mario Pinto, Geng Yan, Adria Puig- domenech Badia, Piyush Patil, Steven Hansen, Dave Orr, Sebastien M. R. Arnold, Jordan Grimstad, Andrew Dai, Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi Chen, Elena Gribovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kris Cao, Da-Woon Chung, Matthew Lamm, Gabe Barth-Maron, Thais Kagohara, Kate Olszewska, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider,
12

---

Published as a conference paper at COLM 2025
Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang, Bat-Orgil Batsaikhan, Mateo Wirth, James Qin, Ivo Danihelka, Tulsee Doshi, Martin Chadwick, Jilin Chen, Sanil Jain, Quoc Le, Arjun Kar, Madhu Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lampros Lamprou, Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan Wang, Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali Ahmed, Dominik Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan Pope, Steven Hand, Vittorio Selo, Vihan Jain, Nikhil Sethi, Megha Goel, Takaki Makino, Rhys May, Zhen Yang, Johan Schalkwyk, Christina Butterfield, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver Woodman, Marvin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim Blyth, Ian Mackinnon, Machel Reid, Obaid Sarvana, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Attaluri, Gregory Thornton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung, Evgenii Eltyshev, Xiance Si, Timothy Lillicrap, Demetra Brady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu, Ross Mcllroy, Kartikeya Badola, Paramjit Sandhu, Erica Moreira, Wojciech Stokowiec, Ross Hemsley, Dong Li, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghi, Salem Haykal, Pablo Sprechmann, Xiang Zhou, Diana Mincu, Yujia Li, Ravi Addanki, Kalpesh Krishna, Xiao Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye Zhang, Emanuel Taropa, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, HyunJeong Choe, Alex Tomala, Chalence Safranek-Shrader, Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaleldin Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wei Chen, Peter Humphreys, Kate Baumli, Connie Tao, Rajkumar Samuel, Cicero Nogueira dos Santos, Anders Andreassen, Nemanja Rakićević, Dominik Grewe, Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia, Hannah Sheahan, lain Barr, Yingie Miao, Paul Natsev, Jacob Devlin, Feryal Behbahani, Flavien Prost, Yanhua Sun, Artiom Myaskovsky, Thanumalayan Sankaranarayana Pillai, Dan Hurt, Angeliki Lazaridou, Xi Xiong, Ce Zheng, Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, Alejandro Lince, Mingqiu Wang, Basil Mustafa, Albert Webson, Hyo Lee, Rohan Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras, Elahe Dabir, Shyam Upadhyay, Fry, Josip Djolonga, Yi Su, Jake Walker, Jane Labanowski, Ronny Huang, Vedant Misra, Jeremy Chen, RJ Skerry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu, Alex Castro-Ros, Beer Changpinyo, Romina Datta, Sumit Bagri, Arnar Mar Hrafnkelsson, Marcello Mag- gioni, Daniel Zheng, Yury Sulsky, Shaobo Hou, Tom Le Paine, Antoine Yang, Jason Riesa, Dominika Rogozinska, Dror Marcus, Dalia El Badawy, Qiao Zhang, Luyu Wang, Helen Miller, Jeremy Greer, Lars Lowe Sjos, Azade Nova, Heiga Zen, Rahma Chaabouni, Mi- haela Rosca, Jiepu Jiang, Charlie Chen, Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov, Jean-Baptiste Lespiau, Josh Newlan, Zeyncep Cankara, Soo Kwak, Yunhan Xu, Phil Chen, Andy Coenen, Clemens Meyer, Katerina Tsihlas, Ada Ma, Juraj Gottweis, Jinwei Xing, Chenjie Gu, Jin Miao, Christian Frank, Zeynep Cankara, Sanjay Ganapathy, Ishita Dasgupta, Steph Hughes-Fitt, Heng Chen, David Reid, Keran Rong, Hongmin Joost van Amersfoort, Vincent Zhuang, Aaron Cohen, Shixiang Shane Gu, Anhad Mohananey, Anastasija Ilic, Taylor Tobin, John Wieting, Anna Bortsova, Phoebe Thacker, Emma Wang, Emily Caveness, Justin Chiu, Eren Sezener, Alex Kaskasoli, Steven Baker, Katie Millican, Mohamed Elhawaty, Kostas Aisopos, Carl Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia, Matthew Wiethoff, Elnaz. Davoodi, Albert Weston, Lakshman Yagati, Arun Ahuja, Isabel Gao, Golan Pundak, Susan Zhang, Michael Azzam, Khe Chai Sim, Sergi James Keeling, Abhanshu Sharma, Andy Swing, YaGuang Li, Chenxi Liu, Carrie Grimes Bostock, Yamini Bansal, Zachary Nado, Ankesh Anand, Josh Lipschultz, Abhijit Karmarkar, Lev Proleev, Abe Ittycheriah, Soheil Hassas Yeganeh, George Polovets, Aleksandra Faust, Jiao Sun, Alban Rrustemi, Pen Li, Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh Baddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn Bloxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Renshen Wang, Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Duc Dung Nguyen, James Svensson, Le Hou, Sarah York, Kieran Milan, Sophie Bridgers, Wiktor Gworek, Marco Tagliasacchi, James Lee-Thorp, Michael Chang, Alexey Guseynov, Ale Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sheleem Kashem, Elizabeth Cole, Antoine Miech, Richard Tanburn, Mary Phuong, Filip Pavetic, Sebastien Cevey, Ramona
Boral, Lisa Anne Hendricks, Corey
13

---

Published as a conference paper at COLM 2025
Comanescu, Richard Ives, Sherry Yang, Cosmo Du, Bo Li, Zizhao Zhang, Mariko linuma, Clara Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu, Danilo Martins, Rachel Sa- putro, Aníta Gergely, Steven Zheng, Dawei Jia, Ioannis Antonoglou, Adam Sadovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina Samangooei, Mina Khan, Tomas Kocisky, Angelos Filos, Chintu Kumar, Colton Bishop, Adams Yu, Sarah Hodkinson, Sid Mittal, Premal Shah, Alexandre Moufarek, Yong Cheng, Adam Bloniarz, Jaehoon Lee, Pedram Pejman, P'al Michel, Stephen Spencer, Vladimir Feinberg, Xuehan Xiong, Nikolay Savi- nov, Charlotte Smith, Siamak Shakeri, Dustin Tran, Mary Chesus, Bernd Bohnet, George Tucker, Tamara von Glehn, Carrie Muir, Yiran Mao, Hideto Kazawa, Ambrose Slone, Kedar Soparkar, Disha Shrivastava, James Corrn-Kerr, Michael Sharman, Jay Pavagadhi, Carlos Araya, Karolis Misiunas, Nimesh Ghelani, Michael Laskin, David Barker, Qiujia Li, Anton Briukhov, Neil Houlsby, Mia Glaese, Balaji Lakshminarayanan, Nathan Schucher, Yunhao Tang, Eli Collins, Hyeontaek Lim, Fangxiaoyu Feng, Feng, Adria Recasens, Guangda Lai, Alberto Magni, Nicola De Cao, Aditya Siddhant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghani, Jenny Brennan, Yifan He, Kelvin Xu, Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb Arnold, Solomon Chang, Julian Schrittwieser, Elena Buchatskaya, Soroush Radpour, Martin Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent Hel- lendoorn, Thibault Sottiaux, Sarah Cogan, Aliaksei Severyn, Mohammad Saleh, Shantanu Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo yiin Chang, Craig Swanson, Biao Zhang, Benjamin Lee, Paul Kishan Rubenstein, Gan Song, Tom Kwiatkowski, Anna Koop, Ajay Kannan, David Kao, Parker Schuh, Axel Stjerngren, Golnaz Ghiasi, Gena Gibson, Luke Vilnis, Ye Yuan, Felipe Tiengo Ferreira, Aishwarya Kamath, Ted Klimenko, Ken Franko, Kefan Xiao, Indro Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel, Vivek Sharma, Peter Choy, Sayed Hadi Hashemi, Jessica Landon, Mara Finkelstein, Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger, Dennis Daun, Khuslen Baatarsukh, Matthew Tung, Wael Farhan, Henryk Michalewski, Fabio Viola, Felix de Chaumont Quitry, Charline Le Lan, Tom Hudson, Qingze Wang, Felix Fischer, Ivy Zheng, Elspeth White, Anca Dragan, Jean baptiste Alayrac, Eric Ni, Alexander Pritzel, Adam Twanicki, Michael Isard, Anna Bulanova, Lukas Zilka, Ethan Dyer, Devendra Sachan, Srivatsan Srinivasan, Hannah Muckenhirn, Honglong Cai, Amol Mandhane, Mukarram Tariq, Jack W. Rae, Gary Wang, Kareem Ayoub, Nicholas FitzGerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan Garrette, Kashyap Krishnakumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn, Josip Matak, Inaki Iturrate, Michael B. Chang, Jackie Xiang, Yuan Cao, Nishant Ranka, Geoff Brown, Adrian Hutter, Vahab Mirrokni, Nanxin Chen, Kaisheng Yao, Zoltan Egyed, Francois Galilee Tyler Liechty, Praveen Kallakuri, Evan Palmer, Sanjay Ghemawat, Jasmine Liu, David Tao, Chloe Thornton, Tim Green, Mimi Jasarevic, Sharon Lin, Victor Cotruta, Yi-Xuan Tan, Noah Fiedel, Hongkun Yu, Ed Chi, Alexander Neitz, Jens Heitkaemper, Anu Sinha, Denny Zhou, Yi Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria Georgaki, Sneha Kudugunta, Clement Fara- bet, Izhak Shafran, Daniel Vlasic, Anton Tsitsulin, Rajagopal Ananthanarayanan, Alen Carin, Guolong Su, Pei Sun, Shashank V, Gabriel Carvajal, Josef Broder, Iulia Comsa, Alena Repina, William Wong, Warren Weilun Chen, Peter Hawkins, Egor Filonov, Lu- cia Loher, Christoph Hirnschall, Weiyi Wang, Jingchen Ye, Andrea Burns, Hardie Cate, Diana Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, Ionel Gog, Yana Kulizhskaya, Ashwin Sreevatsa, Shuang Song, Luis C. Cobo, Anand Iyer, Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huaixiu Steven Zheng, Hui Li, Ananth Agarwal, Christel Ngani, Kati Goshvadi, Rebeca Santamaria-Fernande Wojciech Fica, Xinyun Chen, Chris Gorgolewski, Sean Sun, Roopal Garg, Xinyu Ye, M. Ali Eslami, Nan Hua, Jon Simon, Pratik Joshi, Yelin Kim, Ian Tenney, Sahitya Potluri, Lam Nguyen Thiet, Quan Yuan, Florian Luisier, Alexandra Chronopoulou, Salvatore Scellato, Praveen Srinivasan, Minmin Chen, Vinod Koverkathu, Valentin Dalibard, Yaming Xu, Brennan Saeta, Keith Anderson, Thibault Sellam, Nick Fernando, Fantine Huot, Junehyuk Jung, Mani Varadarajan, Michael Quinn, Amit Raul, Maigo Le, Ruslan Habalov, Jon Clark, Komal Jalan, Kalesha Bullard, Achintya Singhal, Thang Luong, Boyu Wang, Sujeevan Rajayogam, Julian Eisenschlos, Johnson Jia, Daniel Finchelstein, Alex Yakubovich, Daniel Balle, Michael Fink, Sameer Agarwal, Jing Li, Dj Dvijotham, Shalini Pal, Kai Kang, Jaclyn Konzelmann, Jennifer Beattie, Olivier Dousse, Diane Wu, Remi Crocker, Chen Elkind, Siddhartha Reddy Jonnalagadda, Jong Lee, Dan Holtmann-Rice, Krystal Kallarackal, Rosanne Liu, Denis Vnukov, Neera Vats, Luca Invernizzi, Mohsen Jafari, Huanjie Zhou,
14

---

Published as a conference paper at COLM 2025
Lilly Taylor, Jennifer Prendki, Marcus Wu, Tom Eccles, Tianqi Liu, Kavya Kopparapu, Francoise Beaufays, Christof Angermueller, Andreea Marzoca, Shourya Sarcar, Hilal Dib, Jeff Stanway, Frank Perbet, Nejc Trdin, Rachel Sterneck, Andrey Khorlin, Dinghua Li, Xihui Wu, Sonam Goenka, David Madras, Sasha Goldshtein, Willi Gierke, Tong Zhou, Yaxin Liu, Yannie Liang, Anais White, Yunjie Li, Shreya Singh, Sanaz Bahargam, Mark Epstein, Sujoy Basu, Li Lao, Adnan Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung, Neeraj Gaur, Alanna Walton, Lucas Dixon, Ming Zhang, Amir Globerson, Grant Uy, Andrew Bolt, Olivia Wiles, Milad Nasr, Ilia Shumailov, Marco Selvi, Francesco Pic- cinno, Ricardo Aguilar, Sara McCarthy, Misha Khalman, Mrinal Shukla, Vlado Galic, John Carpenter, Kevin Villela, Haibin Zhang, Harry Richardson, James Martens, Matko Bosnjak, Shreyas Rammohan Belle, Jeff Seibert, Mahmoud Alnahlawi, Brian McWilliams, Sankalp Singh, Annie Louis, Wen Ding, Dan Popovici, Lenin Simicich, Laura Knight, Pulkit Mehta, Nishesh Gupta, Chongyang Shi, Saaber Fatehi, Jovana Mitrovic, Alex Grills, Joseph Pagadora, Isendsuren Munkhdalai, Dessie Petrova, Danielle Eisenbud, Zhishuai Zhang, Damion Yates, Bhavishya Mittal, Nilesh Tripuraneni, Yannis Assael, Thomas Brovelli, Prateek Jain, Mihajlo Velimirovic, Canfer Akbulut, Jiaqi Mu, Wolfgang Macherey, Ravin Kumar, Jun Xu, Haroon Qureshi, Gheorghe Comanici, Jeremy Wiesner, Zhitao Gong, Anton Ruddock, Matthias Bauer, Nick Felt, Anirudh GP, Anurag Arnab, Dustin Zelle, Jonas Rothfuss, Bill Rosgen, Ashish Shenoy, Bryan Seybold, Xinjian Li, Jayaram Mudigonda, Goker Erdogan, Jiawei Jiri Simsa, Andrea Michi, Yi Yao, Christopher Yew, Steven Kan, Isaac Caswell, Carey Radebaugh, Andre Elisseeff, Pedro Valenzuela, Kay McKinney, Kim Paterson, Albert Cui, Eri Latorre-Chimoto, Solomon Kim, William Zeng, Ken Durden, Priya Ponnapalli, Tiberiu Sosea, Christopher A. Choquette-Choo, James Manyika, Brona Robenek, Harsha Vashisht, Sebastien Pereira, Hoi Lam, Marko Velic, Denese Owusu-Afriyie Katherine Lee, Tolga Bolukbasi, Alicia Parrish, Shawn Lu, Jane Park, Balaji Venkatraman, Alice Talbert, Lambert Rosique, Yuchung Cheng, Andrei Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin, Lu Li, Khalid Salama, Bartek Perz, Wooyeol Kim, Nandita Dukkipati, Anthony Baryshnikov, Christos Kaplanis, Xiang- Hai Sheng, Yuri Chervonyi, Caglar Unlu, Diego de Las Casas, Harry Askham, Kathryn Tunyasuvunakool, Felix Gimeno, Siim Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, Aaron Parisi, Dangyi Liu, Tomy Tsai, Toby Shevlane, Christina Kouridi, Drew Garmon, Adrian Goedeckemeyer, Adam R. Brown, Anitha Vijayakumar, Ali Elqursh, Sadegh Jazayeri, Jin Huang, Sara Mc Carthy, Jay Hoover, Lucy Kim, Sandeep Kumar, Wei Chen, Courtney Biles, Garrett Bingham, Evan Rosen, Lisa Wang, Qijun Tan, David Engel, Francesco Pongetti, Dario de Cesare, Dongseong Hwang. Lily Yu, Jennifer Pullman, Srini Narayanan, Kyle Levin, Siddharth Gopal, Megan Li, Asaf Aharoni, Trieu Trinh, Jessica Lo, Norman Casagrande, Roopali Vij, Loic Matthey, Bramandia Ramadhana, Austin Matthews, CJ Carey, Matthew Johnson, Kremena Goranova, Rohin Shah, Shereen Ashraf, Kingshuk Dasgupta, Rasmus Larsen, Yicheng Wang, Manish Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki Osawa, Celine Smith, Ramya Sree Boppana, Taylan Bilal, Yuma Koizumi, Ying Xu, Yasemin Altun, Nir Shabat, Ben Bariach, Alex Korchemniy, Kiam Choo, Olaf Ronneberger, Chimezie lwuanyanwu, Shubin Zhao, David Soergel, Cho-Jui Hsieh, Irene Cai, Shariq Iqbal, Martin Sundermeyer, Zhe Chen, Elie Bursztein, Chaitanya Malaviya, Fadi Biadsy, Prakash Shroff, Inderjit Dhillon, Tejasi Latkar, Chris Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer Greene, Marin Georgiev, Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang, Praseem Banzal, Doug Fritz, Vikram Rao, Xuezhi Wang, Jiageng Zhang, Viorica Patraucean, Dayou Du, Igor Mordatch, Ivan Jurin, Lewis Liu, Ayush Dubey, Abhi Mohan, Janek Nowakowski, Vlad-Doru Ion, Nan Wei, Reiko Tojo, Maria Abi Raad, Drew A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez, Zhichun Wu, Hoang Nguyen, Ji Liu, Madhavi Sewak, Bryce Petrini, HongHyun Choi, Ivan Philips, Ziyue Wang, Ioana Bica, Ankush Garg, Jarek Wilkiewicz, Priyanka Agrawal, Xiaowei Li, Danhao Guo, Emily Xue, Naseer Shaik, Andrew Leach, Sadh MNN Khan, Julia Wiesinger, Sammy Jerome, Abhishek Chakladar, Alek Wenjiao Wang, Tina Ornduff, Folake Abu, Alireza Ghaffarkhah, Marcus Wainwright, Mario Cortes, Freder- ick Liu, Joshua Maynez, Andreas Terzis, Pouya Samangouei, Riham Mansour, Tomasz Kepa, Fravçois-Xavier Aubet, Anton Algymr, Dan Banica, Agoston Weisz, Andras Orban, Alexandre Senges, Ewa Andrejczuk, Mark Geller, Niccolo Dal Santo, Valentin Anklin, Majd AI Merey, Martin Baeuml, Trevor Strohman, Junwen Bai, Slav Petrov, Yonghui WU, Demis Hassabis, Koray Kavukcuoglu, Jeff Dean, and Oriol Vinyals. Gemini 1.5:
R Taylan
15

---

Published as a conference paper at COLM 2025
Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/// wwg./bs/////../.b./ httbb//2443005555 05530..
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Pi- otr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean- Bastien Grill, Behnnm Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christo- pher A. Choquette-Choo Danila Sinopalníkov, David Weinberger, Dimple Vijayuumar Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucinsske. Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Lu- ciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Im- proving open language models at practical size. arxiv preprint ar 240:2408.001118, 2024. URL / www.xii..or//bb.... atrgg/bb//// 2408. 00118.
Arthur Gretton, Karsten M Borgwardt, Malte] J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 3(1):723-7773 2012.
Nizar Habash, Joseph Olive, Caitlin Christianson, and John McCary, Machine Translation from Text, pp. 133-397. Springer New York, New York, NY, 2011. ISBN 978-1-4499-7713-7. doi: 2. URL https: /ttp:://///oooo////////
Leigh Halliwell. The lognormal random multivariate. In Casualty Actuarial Society E-Forum, Spring, volume 5, 2015.
Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 492-504, Miami, Florida, USA, November 2024. Association for Computa- tional Linguistics. URL https://acclanthollogy.. hww.hhooolgloooogg////ee org/ 1mt
Tom Kocmi, Rachel Bawden, Ondiej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca
16

---

Published as a conference paper at COLM 2025
Knowles, Philiipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshi- aki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. Findings of the 2022 conference on machine translation (WMT22). In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussá, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 1-45, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL htttt:://aallathhoooog..oo org. wwwgg 1/.
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Chris- tian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, et al. Findings of the wmt24 general machine translation shared task: the Ilm era is here but mt is not solved yet. In Proceedings of the Ninth Conference on Machine Translation, pp. 1-46, 2024.
Philipp Koehn and Christof Monz. Manual and automatic evaluation of machine translation between European languages. In Philipp Koehn and Christof Monz (eds.), Proceedings on the Workshop on Statistical Machine Translation, pp. 102-121, New York City, June 2006. Association for Computational Linguistics. URL httt:::://cclaahholoogoooo org/W06-3114/.
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Isvetkov. Controlled text generation as continuous optimization with multiple constraints, 2021. URL https://arxiv..
0tth/2108.018550
David Kurokawa, Cyril Goutte, and Pierre Isabelle. Automatic detection of translated text and its impact on machine translation. In Proceedings of MT-Summit XII, pp. 81-88, 2009.
Samuel Läubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. A set of recommendations for assessing human-machine parity in language translation. Journal of artificial intelligence research, 67:653-672, 2020.
Gennadi Lembersky, Noam Ordan, and Shuly Wintner. Adapting translation models to translationese improves SMT. In Walter Daelemans (ed.), Proceedings of the 13th Con- ference of the European Chapter of the Association for Compulational Linguistics, pp. 255- 265, Avignon, France, April 2012. Association for Computational Linguistics. URL https: https::/aaclanthoooy...ggg////
Zheng Wei Lim, Ekaterina Vylomova, Trevor Cohn, and Charles Kemp. Simpson's 's paradox and the accuracy-fluencd tradeoff in translation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 2: Short Papers), pp. 92-103, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 / URL https://aclanthologgy w acl shortt.h/ 9/
Fabian Mentzer, George D Toderici, Michael Tschannen, and Eirikur Agustsson. High- fidelity generative image compression. Advances in neural information processing systems, 33:11913-11924, 2020.
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29(2):429-43,, 1997.
Donald Bruce Owen. A table of normal integrals: A table. Communications in Statistics- Simulation and Computation, 9(4)339-411) 1980.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-ling Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 01111//177008.1000 org/P02-1044 1040/.
URL https: / aclanthology.
17

---

Published as a conference paper at COLM 2025
John R. Pierce and John B. Carroll. Language and Machines: Computers in Translation and Linguistics. National Academy of Sciences/ National Research Council, USA, 1966. doi: 10.17226/952 /9447.
Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Ondreej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653 (/663/vv// /1/11530099 org/W15-3040 3049/.
URL https://aclaatthooogy
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2685-2702, Online, November 2020. Association for Computational Linguistics. doi: 10.18653 / v1 11120.emnll-main.. 213/-
URL https: //claathooll org/ wrg.2020..0.... emnlp-main.
Ricardo Rei, Jose Pombal, Nuno M. Guerreiro, João Alves, Pedro Henrique Martins, Patrick Fernandes, Helena Wu, Tania Vaz, Duarte Alves, Amin Farajian, Sweta Agrawal, Antonio Farinhas, José G. C. De Souza, and André Martins. Tower v2: Unbabel-IST 2024 submis- sion for the general MT' shared task. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 185-204, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653 /v1
URL https: hwtp///olaaaattogg...oooo /oootttp////llanttooooggy org/ 2024. wmt-1.124
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque, Input complexity and out-of-distribution detection with likelihood-based generative models. In International Conference on Learning Representations, 2020. URL https:// openreview wwwrreewww.eeeffoummmtttem nett/forum?ddsyyXXWPWWWW
Matthew Snover, Nitin Madnani, Bonnie Dorr, and Richard Schwartz. Fluency, adequacy, or hter? exploring different human judgments with a tunable mt metric. In Proceedings of the fourth workshop on statistical machine translation, pp. 259-268, 2009.
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckriet. On integral probability metrics, .phi-divergences and binary classifi- cation. Xiviv preprint ar arXic:0901.2690 2009.
Lucas Theis. What makes an image realistic? In Proceedings of the 41st International Conference on Machine Learning, 2024. URL https: http:s////aaiiioorr/as///////// 2tte//ab//2283.4499.... 04493.
Lucas Theis and Aaron Wagner. A coding theorem for the rate-distortion-prreep function. In Neural Compression: From Information Theory to applications-Worrkhhe ICLR 2021, 2021.
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. Attaining the unattainable? re- assessing claims of human parity in neural machine translation. In Proceedings of the Third Conference Machine Translation: Research Papers, pp. 113-123, Association for Computational Linguistics, 2018.
Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. Xrx preprint ar arXit:2102.000 002877 2021.
John S. White and Theresa A. O'Connell. Evaluation of machine translation. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993, 1993. URL https:: httpss:/accaathhologg. wrg./93- 1040/.
Andreas Winkelbauer. Moments and absolute moments of the normal distribution. arXio preprint ar Xix: 43404340, 2012.
18

---

Published as a conference paper at COLM 2025
Yiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Mingxuan Wang. Bleurt has universal translations: An analysis of automatic metrics by minimum risk training. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5428-5442 5443, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
19

---

Published as a conference paper at COLM 2025
Appendices
A Selecting distributional divergences
There are many mathematically natural candidates available, such as total variation, the Wasserstein distances, Kullback-Leibler or other f-divergences and so on. A particularly appealing class of metrics consider are the integral probability metrics (IPMs; Müller, 1997). For convenience, we introduce the shorthand P(f) = Ex~p (x))] Then, IPMs are defined
IPM = supp P(f)L
FEF
where the precise IPM is defined by the function class F we choose to optimise over. IPMs include the total variation distance, Wasserstein distance and maximum mean discrepancy (MMD; Gretton et al., 2012). An appealing aspect of IPMM is their direct connection to the ability of the optimal observer / critic (as a member of the function class F) to distinguish Q-samples from P-samples (Sriperumbudur et al., 2009). It is useful to consider that in many cases, we can consider an IPM as the separation achieved by the optimal critic: if we P(f)l, then (P P| IQ(*)= P(
consider f arg max, fa (exxx==1((])
This viewpoint reveals the main practical issue with IPMs, which was already noted for f f-divergences by Theis (2024): the optimal critic depends the distributions Q and P that we are trying to evaluate. One solution that is often employed in lossy data compression is therefore to train the critic alongside the the generative model, leading to adversarial training framework (Mentzer et al., 2020; Theis, 2024). The advantage of this approach is that besides obtaining critic for evaluating the model at test time, it allows us to optimise our models using the due to the discrete nature of text outputs, adversarial approaches of text generation have lagged behind autoregressive approaches (Caccia et al., 2020), hence more research is needed before translation models can be trained using the analogous accuracy-naturalness objective. Hence, we focus on metrics that do not require optimisation: no-reference metrics. Following Section 4, we will be interested quantities that can be estimated using the reference distribution Ry and the system outputs Y""
objective in Equation (5) directly. Unfortunately,
A.1 Using LLM model scores
Assume that we have "distilled" the true marginal into language model Ry = P.M. This simplifies the problem to assessing the distance between Q2 and P.-MM This should be easier: let f be the probability mass function of PLM Then, we can also compute the probability fy) of the output text y under the language model. Then, we might be tempted to compute the perceptual scores as
N Σ
C
log (y"").
(8)
However, this is problematic, because, as we can see by the law of large numbers:
c. E Ey-os I- (oglogf(())
(9)
H[O [PPM
(10)
= Dxal0" | + H(O"),
(11)
where P] is the cross-entropy of Q from P. The issue here, is that H[Q"]] could
differ wildly among the translation systems, and hence the C scores are incomparable! A different perspective on this problem is to note that the minimizer of the cross-entropy is in
20

---

Published as a conference paper at COLM 2025
general not P.M (unless PyM is deterministic, which is uninteresting):
Qj arg arg min H
(12)
Qy
arg mig ( H[Q)] Qy
(13)
Intuitively, Qÿ balances proximity to PLM (due to the KL term) and diversity (due to the entropy term); this issue was already noted by Blau Michaeli (2018) as well.
A.2 Using normalised LLM model scores
One potential solution to the problem we noted in the section above is to estimate the entropy term ]-
The ideal solution. The cleanest solution would also estimate H[Q"] directly, and subtract it from our estimate. However, this requires us have access the scores the translations, i.e. we would want the dataset to comprise of (y log grr (yn)) pairs, where (yn) is the probability mass function of distribution Q Then, we could compute the
perceptual score
1
N
9"H (yn)
D
log f(y))*
(14)
which would then converge to PKL[Qj P(M) and would be comparable. Unfortunately, without the foresight collect the model scores well, this approach is not practical.
Approximating the ideal solution. An alternative, more feasible solution is to take inspira- tion from the outlier detection literature, and estimate the scores using universal source coding algorithm such as Zip (Serrà al., 2020). Concretely, let g (y) denote the length of the Zip compressor's 's output in bits upon compressing the string y. If y is long enough, (yi should converge to log qtr (yn), which warrants the estimate
1
Σ (logf(y")) E ( (
(15)
zip N
This approach can be justified by appealing algorithmic information theory and in particular Kolmogorov complexity (Theis, 2024).
different interpretation. These metrics admit an alternative interpretation as an approxi- mation to statistical distance which we describe in Section 4; see Appendix E for technical details.
B A generalisation of Theorem 1 of Blau & Michaeli
In the main text, argued that for the purposes of translation the monolingual reference distribution Ry should not depend on Pxy, and derived the existence of the tradeoff from this principle.
In this section, we take a different approach and show that if we always set Ry < Py, our theory also admits direct generalisation of Theorem of Blau & Michaeli (2018), and hence a tradeoff exists in this case too. However, before we state the theorem, we make the following definitions.
Definition B.1. A distortion measure A X' y x y - R is distribution preserving Pxy with respect to Py if for the system Ox maximising Equation (1) we have Q2 -Py-
Technically, should take the minimum across several compressors for better approximation (Appendix in Serrà al. (2020)).
21

---

Published as a conference paper at COLM 2025
Now, assume that we have A that is distribution preserving at Pxyi how good is it in other scenarios? Could A be distribution preserving we change Pxy slightly? This motivates the generalisation of Definition 2 of Blau & Michaeli (2018):
Definition B.2. A distortion measure A is stably distribution preserving at Pxy if is distribution preseroing at all Pxy in TV e-ball around Px, for some > 0.
Now, we have the following generalisation of Theorem 1 of Blau & Michaeli (2018).
Theorem B.1. If the relationship from x y not deterministic in the sense that [y x] > then A is not stubly distribution preserving at Pxy with respect to Ry(Py (Py).
Our proof technique is a straight-forward extension of the Blau and Micali's proof. By way of contradiction, we assume that there is a A that stably distribution preserving. Then: 1) we show that the minimizer of Eq 3 is unique when A is stably distribution preserving and then 2) we show that the non-inveribbility between and y implies that the minimizer is non-unique. Taaiiiiue.Tkinngggg these two facts together, we obtain contradiction, hence the result holds.
Next, we define the following notion of non-uniqueness for the minimizer of distortion measure A:
Definition B.3. Let A be a distortion metric (see Section 2). We say that the minimizer of A is non-unique if there exist two distributions Qy1x' Og such that
2(0%))) = ((2))) = min A (Qyxx)
Qylx
and we have
Ex (D.,,//Tr(Oy)> Og 0.
The proof proceeds analogously to the proof of Theorem 1 in Blau & Michaeli (2018). We now proceed to prove Theorem B.1:
Proof. We first show, that our notion of non-determinism is equivalent Blau & Michaeli's notion of "non-invertibility"
Lemma B.1. Let x, ~ Pay random variables over X x y. Then, following are equivalent:
1. dot-determ:n H(y x > 0.
2. Non-inoertibility:: There exists Sx S. X and Sy Y with Px(Sx) > O and |Sy] > such that Pyrx is either 1) supported on an uncountable set or 2) admits probability mass function ply x) and it holds that Vx,y E Sx Sy we have pyy x) > 0.
Proof. The conditional entropy H x 0 when and only when Pyjx admits probability mass function ply x) and there exists function g such that Px-almost surely p(y xx 1ly g(x)] (Exercise 2.5; Cover & Thomas, 2012). Then, both directions of the lemma follow directly from the contraposition of the above equivalence.
We believe our definition clearer than the equivalent definition of Blau Michaeli (2018), as it makes intuitively more sense: the relationship > y is non-deterministic, because knowing x still leaves some uncertainty in
We proceed with the following uniqueness result.
Lemma B.2, A is stably distribution preserving at Px, with respect to (-).), then the minimizer O = arg ming,, A(Qy)x) is uniquely defined by Pylx-
22

---

Published as a conference paper at COLM 2025
Proof. The proof of the lemma relies on the following two facts: Fact The minimizer A does not depend on the input distribution. To see this, note that by definition,
A(Qyxx) Exyr-Pay Qy [A(x,', y')l]
= Ex-P (yr-pyur- [A ( y"]]
ExuB [x(x
where we defined
f(x, Qylx) = (xx, y")]
Now, we note the following "minimean" result (somewhat analogously to a minimax theorem):
min (Q2
min Ex [f(x, Qyix)] Ex Gylx
min Qyix) Cyjx
(16)
Gy/x
To see this, note that for all Qylx it holds that
Ex~Ra [(xx Qylx)] 2 Ex
min f(x, Qyn))
Gylx
However, setting Qyx==x aig Qylxex f(x, Qylx=x) for each x, that
If 0yx)0 = Ex min f(x 2yyw)
Gylx
Now, since by Equation (16) the expectation with respect to Px and the minimization exchangeable, the minimizer Qjx does not depend on Px-
Fact 2: Convex perturbations in the c-TV ball of Pay. Next, note that if A is stably dis- tribution preserving at Pxy within an e > 0 radius TV-ball, then it is distribution preserving at any joint distribution Pxy Py pyxxe P 8 where
= a (1=
where a > 1 - e and Qx is any distribution over X'. This follows, since Pay is in the c-TV ball around Pxy:
Drv (Pxy Pxy) Drv +1
< aDr (xxy PyxPx) + DrV (Pxy PyxQxx
(convexity of Drv)
< (1 aa)
(first term vanishes and Drv - 1.)
(17)
Let Py denote the marginal distribution defined by the joint Ps,, given by
(-) = Ex-p, (o-.A [Py)x-)]
a.E. P2 [Pyxx()] + (1 Ex ( (-)x(())
.Py (1 a) Ex (.yyPPyy())))
(18)
Forr convenicence, let Pyxxs denote the semi-direct product of Pylx with Px that is for S E X y have PS) Fx P [1 e S]].
23

---

Published as a conference paper at COLM 2025
Furthermore, by Fact 1, since minimizer of A does not depend the distribution of x, have that O Oyix' Letting Px aPx + (1 a)Qa be the x-marginal of Pxy. This now allows to compute Qj:
27(-) = ExwR, (y)x((1)
alExxp Rypn + -1-1111110 a (1- = =a + -1) (1= Q, (2)
aPy + ( ())
(19)
Now, since A is distribution preserving at Pay by Fact 2, we must have Q; Py, where Oj is the marginal distribution defined by the minimizer θyjx of with respect to Pay. This means that Equations (18) and (19) must be equal, hence must also have
Ex-O, (x)xxx) Ex (Pyxe())))))
Since Qx, was arbitrary, this equation holds if and only if Qyyx = Py|x as required.
We now move onto our second result:
Lemma B.3. Assume the relation from -> is non-deterministic. Furthermore, assuoee that the minimizer of A is 0yix = Pxly- Then, 2/yx is non-unique.
Proof. Follows from Lemma B.1 combined with Lemma from Blau & Michaeli (2018). L
To finish the proof, assume by way of contradiction that the Pxy defines a non-deterministic relationship from x - y and that the distortion is stably distribution preserving at Pay- Then, by Lemma B.2, the minimizer of A is Qý = Pyix and this estimator is unique. However, by Lemma B.3, 0% is non-unique, which leads contradiction.
C Proof of Theorem 3.1
Theorem 3.1. Let A(N) be the acuracy--ntturannsss function defined by distortion A, distributional distance D, parallel distribution Pxy and monolingual reference Ry. Then:
1. (NN is non-increasing in N.
2. if D is convex in its first slot, then ANN) is concave in N.
Proof. (1) Non-increasing. We study the set admissible translation systems. Thus, let
CN = (Qyx D(y,, Ry) 2 N}.
(20)
Now note that since D is bounded from above, for N N' we have DD Ry) N > N', hence CN C Cyy. Therefore, A(N) is the supremum of the same objective but over bigger constraint set, hence we must have A(N) < A(N').
(2) Concave when D is convex in the first slot. We wish to prove for all Np, N₂ in the range of and À E [0,1] that
A(ANo (1 A)N)) 2 AA(No) + (1 (1- (1-A))(NN).
(21)
To begin, let
09/x arg max - -(0yk) s.t. DD(y,, Ry) 2 No
(22)
Qyix
Q 0y/x arg max - ((0xx)
s.t. - D(2y, Ry) 2 No.
(23)
2ylx
24

---

Published as a conference paper at COLM 2025
In other words, Equation (21) as
Words,-A(Qgi, -A(0g)) A(No) and -A(0gx) A(N)). This allows us to rewrite
A(AN + +1-1)NN) 2 -AA(O (1- 1))(())
(24)
Next, we define
Oxx = 1C + (1-1)
Now, let ()(-) 10y) (-), and define Qf and O analogously. Furthermore, let
N₂ DDQ,, Ry). Then, since D convex in its first argument by assumption, we have that
N = DDOY,, Ry)
2 AD(O Ry) -
-1 (1 A),yy, Ry)
2 ANo + (1 (1- (1-ANN
(by the constraints in eqs. (22) and (23))
Combining this with the fact that ((N) non-increasing in N, we find that
A(ANo + (1 - A)N1) 2 A(Nx).
Hence, to show Equation (24), it remains to show that
A(Nx) 2 - -A(CO -1 (1-1)A(Q,))
To see this, observe that
A(N,) max ((Qylxx)
s.t. -(y,, N₁
Oy/s
2 xx()))))
where the inequality follows since D(O,, (.Q,RRy) Ry) N₂ by definition, hence D(2,, Ry) 2 N₂ in the admissible set. Finally, note that
x(0%) Exy Pay By - ( yA(x
(25)
-Bxx AME AB y, y (1-
A)B [A(xy,,)]]] (definition)
=AAA(O))) (1 (1- ()A(O))))
(definition)
which finishes the proof.
D MQM details
For the en de and ja + zh language pairs, the WMT24 evaluation exactly followed the evaluation scheme proposed in Freitag et al. (2021). The error categories are shown in Table (limited to the actual errors found in the data), together with our classification into accuracy or fluency errors. This classification is straightforward, as most categories are already labelled as such. is worth noting that we ignored the "Source issue" and "Other" categories, we found the marked errors to be uninformative in most cases. In principle it would be possible to classify most of the individual errors in this category to either adequacy fluency, but it would involve huge amount of additional manual work. The en + es translation used another error schema. Its classification into adequacy and fluency errors is shown in Table 2.
Once we have the classification into these categories we can compute adequacy and fluency scores using the same weighting schema for errors as given in Table in Freitag al. (2021).
25

---

Published as a conference paper at COLM 2025
Accuracy Errors
Fluency Errors
Other
Fluency Grammmar
Fluency / Incconssstennccyy
Fluency/Punctuun /Punctuation
Fluency/Regisse /Reiister
Accuracy Addition
Fluency Fluency/Spelling /Spelling
Accuracy/ Creative Reinterpretation
Fluency / TText-reeaiing
Accuracy Gender Mismatch
Locale convention Address format
Other
Accuracy BeaiaacyMMistaansaatee Mistranslaaion
Locale convention /Currency format T'mme format Style/ Archaic obscure word choice Style/Bad
Source issue
Accuracy Carracyy/mmii /Omission
Locale convention /
Accuracy/Sore 'Souree language fragment
Non-translation!
Style/ Baad sentence structure Style/ Unseturnntt or awkward Terminology /nappropriaae for context Terminology Inconsistent
Table 1: Our classification of MQM errors into adequacy and fluency errors for the en de and ja + zh translation directions.
Accuracy Errors
Fluency Errors
Other
Capitalization
Date-time format
Inconsistency
Addition
Lacks creativity Grammar
Agreement
Do not translate Mistranslation MT hallucination Omission
Measurement format
Number format
Other Source issue
Punctuation
Register
Untranslated Wrong named entity Wrong term
Spelling
Unnatural flow
Whitespace
Word order
Wrong language variety
Table 2: Our classification of MQM errors into adequacy and fluency errors for the en -> es translation direction.
26

---

Published as a conference paper at COLM 2025
E Technical details for the divergence in Section 4.1
In this section, we define a generalised variant of the divergence from Section 4.1. Our starting point is to notice that IPMs could be viewed as a certain infinity-norm, since we taking supremum as part of their definition. This suggests that should be able define analogous p-norms for E [1, oo) over appropriate spaces of distributions too.
We base the following discussion on Müller (1997) to make the above idea precise. Let $ be measurable space and b ;:3 S + [1, oo) a measurable function. Let B, be the set of measurable functions f S such that
[((s] sCS b(s)
If sup
Co.
(26)
For example, taking b 1 we get that Bj is the set of bounded functions from S to R. Next, let M& be the set of all signed measures such that [μ](b) μ (b) + H (b) < co, and let IP denote the set of all probability metrics over S. Then, by Lemma 2.1 of Müller (1997), IM, and B3 are in strict duality under the dual pairing
(+)) MD x BB R (1, f) = m(f).
Next, define M M2 as the set of all signed measures μ with μ(S) = and 27 the quotient space of By by the equivalence relation f 8 - f g is constant. Then, Lemma 2.2 of Müller (1997) shows that M and 2 are also in strict duality under the same bilinear map as above. Now, for function class F C BB, we can define the following seminorm on M
||l|| (upl((f)) fcF
(27)
Importantly, this seminorm induces the integral probability metric d over IP:
dr(Q,P) = - -|,-
(28)
Note that ||=1|= is the co-seminorm on MM This motivates us to define a larger family of seminorms as follows. Let P be probability measure over Bg. Now, define the seminorm
|| p.P P(
(29)
Then, similarly to 1|,,|,, this induces semimetric on PP: =
(30)
Dp(O.PI = ||Q - Plpp, = P(
(31)
Now that we defined our divergence, we show that it admits a similar connection to classification as IPMs, analogously to Theorem 17 of Sriperumbudur et al. (2009):
Theorem E.1 (Connection to expected classification risk). Let F, Q Q, for p [0, oo) be as above. Then: For a loss function L: -1,1 x R + IR, define the expected binary classification risk of P as
R³ = Bx (Ly = -Ej-pleEx-
(32)
-1 - (x-
(33)
where e = P = 1). P P Y = 1] and Q = P Y = - -1.. Then, for L (a) = - a/e and
L 1_)(2)) a/ (1 E), we have
R = -Do Q P) (P.Q P) < R$
(34)
27

---

Published as a conference paper at COLM 2025
Proof. Define +x) max and k(f) = [[f(X) Y -1] -
- -E((( Y =1.. Then,
with our choice for the loss function, we have
P = E [X) =
1, -(f( 1Y=
(def)
= E [K(F)
(
= -E, [[((F]] +2 B, 1(K(f)))+
(x = -|x| + 2(x)+) (def)
2 -DI(Q.PPP))
-E, [(K
2 -E, (ICKEIP
(Jensen for p 2 1)
PD P)
(def)
Do P PP)
p 0
R'O
(Theorem 17 of Sriperumbudur et al. (2009))
E.1 Examples of D, when P of Gaussian process type
Based on its definition, Dp can be easily approximated via a Monte Carlo estimate for both samples from the process P and samples from Q and P. However, in this section, we show that for certain broad class of processes, we can evaluate the expectation over samples of P. More precisely, we compute Dp when P is of Gaussian process type, i.e., for f ~ P we have
(A f) ~ GP (m,,k)
for link function À and Gaussian process GP (m,k) k) with mean function In and covariance function and where indicates function composition. It turns out, that in these cases, Dp is intimately related to the maximum mean discrepancy MMD, (Gretton et al., 2012), where the MMD is with respect to a reproducing kernel Hilbert space with the kernel taken to be the covariance function of the Gaussian process above. Indeed, we show in Appendix E.1.1 that in certain cases, Dp and MMD; are even identical up to a multiplicative constant. However, as are averaging instead of maximising in the case of Dp, in such cases we might call it mean- mean discrepancy (MeMeD).
Now, we have the following result for the MeMeD when p p= 2:
Theorem E.2. Let P and Q be probability measures over X. Let P be of Gaussian process type with mean function m, covariance function and link function à, i.e. for f P we have (A o f) GP (m, k). Then, setting
C(x,y) -[)-)((())(()]] E sP ((( (g(x)) (g()))
(35)
we get that
D2(Q,P ((Q.PIP)2= P) (. ((x,
(36)
M2 -Ex,y-ou(CC(x,Y) 2E 2Ex,y-xxPICCX [C(X, Y) + Pxy [C(X, (37)
Furthermore, for a dataset of N samples [Xn) of Q and M samples (Ym JM of P, then D2
admits the following unbiased estimator:
1
1 Σ C(Y,
P) N(N 1)
Σ C(X, M(M 1)
ISLSEN
ISLJEM
2
Σ C(X..))
(38)
NM
ISKSN
13
28

---

Published as a conference paper at COLM 2025
Proof. By definition, we have
D2(O,P PP) =
(39)
m))m,
(40)
where the first set of angle brackets denotes the dual pairing as above, while the second set of angle brackets denote the inner product over Mg. Furthermore, Ty is the integral operator defined by
π E M2 (T) / ((x)((y)dT(())
(41)
Then, by the linearity of expectation, we get
D2(Q.P P(Q.PIP) P) (Q- P.F(y-llT(((-P)))
(42)
To compute Eywp(Ty), note that by the law of the unconscious statistician, we have for any π E M
(E/-p(T/)T(()) / [=-pl(())
(43)
/ C(x,y) dr(y).
(44)
Substituting this back, proves the first part of our claim. Now, the fact Equation (38) is an unbiased estimator follows from the theory of U-statistics, similarly to the argument of Lemma 6 of Gretton et al. (2012).
To demonstrate the power of Theorem E.2, we now consider three concrete cases when P is of Gaussian process type.
E.1.1 When P is Gaussian process
The simplest case of being of Gaussian process type is when the link function simply the identity. Indeed, in this case we can compute the entire family of Dp metrics for all
E [0, ou) without invoking Theorem E.2:
Theorem E.3 (D based using Gaussian processes). Let P and Q be probability measures over X. Let P GP(m,, k) be Gaussian process with mean function m and kernel k over the sample space. Let [(z) - fo e-- dt denote the Euler gamma function and M(a, b, denote Kummer's confluent hypergeomettric function. Then,
1+p
2 MMD, )
(1 (P(m) (om)-QQA Qm))mmmm)))))))))))
P) MMD 2P/2
M
V π
2' 2'
(45)
In particular, we have
D2(P,Q 1 (P, (P(m) Q(m))
(P(P V (P, exp
(P(m)) Q())mmmmm)) )2)
2 MMD; Q)
( (P(m) Q(mm)
[MMD QP, ))
+ (P(m) Q(m)) 1- Φ
(46)
Furthermore, when m = O, we have
V p+1 Dp(OPIP)- MM. ->
e
(47)
Finally, the above results imply that Dp metrizes weak convergence if and only if MMD, does.
29

---

Published as a conference paper at COLM 2025
Proof. By the consistency property of Gaussian processes, it holds that dual pairings of GPs with linear functionals are Gaussian distributed. Thus, for f ~ GP (m,k) we have
P(f) Q(f) = (f. - Q) P P Q)x),
(48)
where
- P - (P -P-00-000///tttt0--000002/321212
-PMD,(P,Q),
(49)
Now, set = -m, P Q) and o= = MMD;(P,Q). Then letting e ~ (0, 1), we have
Dp(P,Q P) E[[oe μ||"]
(50)
Then, applying equation 17 of Winkelbauer (2012) the right-hand side of the above, we get
M (-
Dp Py op
2
(51)
π
2'
from which the identities for the distances hold. For the limit result, note that when = 0, we have
Dp(P,Q|P)= = 0. 2.. 1 1/p
2 (12
(52)
The result then follows by applying Stirling's approximation.
E.1.2 When P is a log-Ganssian process
An example that is perhaps closer to practice is requiring that the scores be non-negative. One way of achieving this is to set P to be a log-Gaussian process, that is, for f ~ P, we have that log ~ GP(m, k) for some mean function m and covariance function k.
Theorem E.4. Let P and Q be probability measures over X. Let be a log-Gaussian process with mean function m and covariance function k, i.e. for f f~ P we have log GP(G, Then, D2 can be computed according Theorem F.2 by setting:
μ(x) E (e = exp [(()+ k(x, x)
(53)
C(x,y) ((x)))(y)exxp(aa
(54)
Proof. By applying Theorem E.2 with À log, we simply need to compute
C(x,y) Fy~p(f(x))( (xx)( (exxx(((x) + s(y))]
(55)
-gg-GP(mxk)
(56)
Now, gxx) and gyy) are correlated Gaussian random variables, hence evaluating the expec- tation pointwise using the formulae from Halliwell (2015), we get
E (ao.p((ms)lxx(((/1)) g()))] ((x)x(y) exp(k(x,)))
(57)
One interesting case of this distance when we set m(x) = [|] log (x) the negative
log probability of a language model and set k(x,y) 1[x y] (i.e. the scores constitute pure noise process). Then, we see that D2 becomes
D2 P P) =e Ex-Q (X) M/1] (xxx]
Ex-p (XM 1/1x|]
(58)
30

---

Published as a conference paper at COLM 2025
E.1.3 When P is Probit process
An example that is even closer to practice is requiring that the scores be bounded. One way of achieving this is to set d to be a Probit process. That is, letting O be the CDF of the standard Gaussian, for f ~ P we have that Φ (f) ~ GP (m, k) for some mean function mr and covariance function Then, we have the following result:
Theorem E.5. Let P and Q be probability measures over X. Let P be Probit process with mean
function and covariance function k, i.e. for f ~ P we have ab I GP(m,)). Then, D2 can be computed according to Theorem E.2 by letting
where Σ [ P
BN b p) PXY-NOXXX Say a
(59)
denote the CDF of the standard bivariate normal with correlation evaluated at (a, b), and setting
m(x)
m(y
k(x,y)
C(x,y) = BNN
(60)
V1+
Proof. Similarly in the previous section, can apply Theorem E.2 with A @. Then, we are interested in computing
Ex-rl((x)(()]] = (5==
(61)
(62)
where = [mn(x), m(y) and E xk(x,x) k(x,y)
(63)
k(x,y) k(y.y)
Note, that V = + 21, where e ~ N I), from which we get U = del + #1 and V = be, + CC2 + 1/2, where = VEni Iz1/a and = VEn - b2 follow from the Cholesky decomposition of Hence, we find
Eu,y~N(n,) (VU))((V) -Ee-N(o,)) Ø(ac1 + (b) + ce2 + #2)
Eq- ( + M1 11) (22- ( + c22 + +222 [i]]
bei + 12
(gr(((0)))
+ + 11
vI+ 2²
(identity 10, ...010.8 of Owen (1980)
BNN
(
111
N22
Σz1
VI+ VI
2.2
En
(identity 20, 20,010.3 of Owen (1980) & simplifying)
which finishes the proof.
Caveat with the LLM-based approximation of the AN curve
Note, that the approach we use to approximate the accuracy-naturalness curve in Section 5.1 will always produce an overly optimistic estimate for the curve, since compute our estimate as
L = [[max L),
where L is the one-shot Lagrange dual of the AN function from Equation (5), and we maximise over systems/ /LLL -gnerrree candidates for each translation first, and then we average over the different entries of the test set second. However, in the true Lagrange-dual of the AN function, the order of the expectation and the maximisation is swapped: max E[C). Finally, have the standard result that
E max L) > max E[2],
hence our procedure always overestimates the AN curve, ie,, it makes the achievable region look larger than it actually is.
31


Reviews:
[{"review_id": "Ql9KwNx6iJ", "reviewer": "Reviewer_LQ9z", "summary": "The paper extends the work on distortion-perception trade-off to the field of (machine) translation. The paper tries to provide a rigorous mathematical justification for the trade-off between translation accuracy and naturalness, proposing that these criteria are fundamentally at odds. The work contains a number of diverse contributions:\n\n* A theoretical extension of the distortion-perception trade-off to translation.\n* Empirical validation of the hypothesis.\n* Extensive supplemental material containing rigorous mathematical derivations and examples.\n\nThe work is ambitious and in my opinion exceeds the scope of a conference paper.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Scope of the work; the authors combine mathematical work and extensive empirical evaluation at a scale that is rarely seen in conference papers.\n* Thought-provoking results. The distinction between adequacy/accuracy (\"Translation model\") and naturalness/fluency (\"Language model\") is not new, however the statement that as translation quality increases, they become fundamentally at odds with one another, seems like a very interesting result which can spark debate in the community.\n* I would highlight the synthetic result which shows convincingly the trade-off at the Pareto frontier (illustrated in Figure 1).\n* Clear writing; the paper is easy to follow and explains all the concepts well.", "reasons_to_reject": "I have doubts about how well the empirical results (specifically the analysis of WMT system submissions) actually fit the hypothesis.\n* In Figure 2, there is no clear sign of a trade-off. In fact, better-performing systems appears to be closing in on the Pareto frontier in a fairly straight line, improving both accuracy and naturalness.\n* In Figure 3, each language pair paints a completely different picture. I don't observe any systematic \"correlation far away from\" and \"anti-correlation close to\" the Pareto frontier and I doubt it can be reliably inferred statistically, given the small sample sizes.\n\nIs it possible that while the trade-off exists _in theory_, in practice we are actually not close to the \"quality limit\" and we can still continue improving both aspects?\n\nAs a separate issue, I have some doubts about the assumptions formulated in sections 2 and 3.\n\nThe definitions state that sentence pairs are drawn from the joint distribution $P(x,y)$. If that's so, I don't understand how it's possible to assume that this leads to a \"natural\" $P(x)$ (after marginalization, i.e. $P(x) = \\sum_{y} P(x,y)$) but a \"translation-biased\" $P(y)$  (after marginalization, i.e. $P(y) = \\sum_{x} P(x,y)$ ). Concretely, the authors state that:\n\n> For the purposes of this paper, we think of Px as a “true” monolingual distribution over the source language and Py|x as the distribution of human translations into the target language given the source sentence x. Therefore, since we obtain Py by marginalising over the input, it is the distribution over “human translations into the target language.”\n\nI believe we can either assume some sort of sampling bias which skews $P_{x,y}$, but then both $P_x$ and $P_y$ are affected. Alternatively, we assume that we can get a correct estimate of $P(x,y)$ but then the estimates of _both_ marginal probability distributions should be correct as well? (As in, English-German parallel sentences are neither natural English, nor natural German. Or they are both natural. But I don't see why one side is natural and the other one isn't. If the assumption is indeed that all the sentences come from \"original language\" and their target-language version are always translations, isn't that flawed? Parallel corpora will typically contain a mix of both, and each has very different characteristics in practice.)\n\nThere is also the underlying related assumption that \"starting from a different source language leads to a different $P(y)$\". How is this mathematically justified? I can relate to the examples and would tend to agree that this occurs _in practice_, I just don't follow the chain of thought here.\n\nMaybe I'm misunderstanding Appendix B but if we assume either non-invertibility (like the original paper) or non-determinism, would we still need these assumptions?\n\nAgain, I don't necessarily doubt that this occurs in practice, I just think there's a flaw in the \"pure mathematical argument\" in the main text.\n\nFinally, going back to the empirical results, there are some relatively strong assumptions about MQM annotations which I would prefer to see validated first: (a) fluency score doesn't depend on the source, even though annotators have access to the source sentence (b) expected fluency of a natural sentence in the target language is perfect (0 errors).", "questions_to_authors": "* Why did you select a relatively narrow domain (NewsCrawl) for estimation of $R_y$? I realize this is the domain of the translated data, however, it doesn't necessarily represent the full spectrum of \"natural-sounding German\".", "extra_scores": {}}, {"review_id": "q64Y8PnObC", "reviewer": "Reviewer_Z5DR", "summary": "This paper explores the question of tradeoffs between accuracy and naturalness in machine translation. The paper formalizes a definition of naturalness based on target language text corpora and demonstrates that under that definition and their definition of accuracy, there will generally be a tradeoff between accuracy and naturalness, resulting in a pareto frontier that translation systems may approach. They show some well-known results (correlation between adequacy and fluency at lower quality levels) and also provide evidence related to recent questions about whether there may be less correlation between the two as translation quality improves. This work provides a formalization that may be useful in considering how machine translation systems are trained and evaluated, though in practice there are likely to be many questions and debates about how best to implement it (e.g., what constitutes an appropriate corpus for measuring naturalness, what metrics are appropriate for measuring adequacy).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The definition of naturalness based on a corpus rather than through an attempt to define a prescriptive definition of naturalness provides quite a bit of flexibility. This neatly sidesteps some thorny issues, though they may reappear when one attempts to implement this in practice.\n- The paper makes good and appropriate use of existing data for the evaluation, which makes it reproducible.\n- The paper provides a thorough definition, description, and formalization of its approaches.", "reasons_to_reject": "- Line 129 mentions that condition 2 (that the function $\\Delta$ is minimized when $y^c$ equals $y^r$) is difficult to prove for neural metrics. This seems like a potentially risky understatement, given that some neural metrics are known to have pathological errors such as “universal translations” that score high regardless of the reference (https://aclanthology.org/2023.acl-long.297.pdf).\n- More generally, there is a bit of handwaving about what exactly various neural metrics are measuring, which in some ways brings us back to the original underlying question of how possible it is to fully treat adequacy and fluency as two completely separate dimensions of quality.", "questions_to_authors": "- The choice of the term “reference metric” has the potential to be confusing, in part because it is defined to include QE metrics, which are sometimes referred to as reference-free or referenceless metrics.\n- Can you comment on the interplay between naturalness and topics or domains? There is some discussion in the paper of the idea that there may be certain domains where it may be preferred to lean towards accuracy (e.g., scientific papers) vs. naturalness (e.g., literature). What about situations where you are considering translations of things that are very specific to the source language (e.g., specific cultural or political topics, or even regional or geographic topics) that are very different from or very infrequently discussed in the target language? In such a situation, in what ways do you expect (or not expect) these to be captured in terms of naturalness in the ““true” monolingual reference over the target language”?\n- Lines 163-166 state that the “proposed theory has no notion of “sentence-level” naturalness”. This is the first time that sentence-level questions are brought up, and it is not clear where this is coming from, how it should be interpreted, or how it connects to the next sentences. This could be expanded or rephrased.\n- It would be appropriate to cite prior work on training and/or decoding machine translation systems with multiple objectives, e.g., https://aclanthology.org/P12-1001.pdf (While this focuses on earlier MT paradigms, it touches on many of the same issues raised here.) and more recently https://proceedings.neurips.cc/paper_files/paper/2021/file/79ec2a4246feb2126ecf43c4a4418002-Paper.pdf", "extra_scores": {}}, {"review_id": "KOLZKKOtdT", "reviewer": "Reviewer_AhdH", "summary": "This submission proposes a mathematical framework for modeling the trade-off between accuracy and naturalness in Machine Translation. Some empirical results are shown to illustrate how it relates to actual MT evaluation results, using WMT24 data.\n\nThe paper is well structured, well written and overall quite clear, even in the more theoretical sections. The described framework is inspired by Blau & Michaeli's work on distortion-preception trade-off in image processing. The extension to MT is smart and well down, and well-connected to the various types of metrics used in the field.\n\nIt is unclear how this impacts MT research and evaluation, however. Although the distinction between naturalness and e.g. fluency is interesting and clearly described, such trade-offs have been well-known in the field for a while, as clearly stated in several places in the manuscript. This work shines a new light on this known problem that had received little theoretical (at least mathematical) work. Two outstanding issues remain:\n1. Synthetic results (Sec. 5.1) yield outcomes that (reassuringly but unsurprisingly) match theory well. On the other hand, practical evaluation results (Fig. 2 and 3) do not appear to fit the theory particularly well. Authors argue that the trade-off curve only becomes relevant for better-performing models. Fair point, however in all examples, everything seems quite far from the curve, and it looks like essentially optimizing to the upper-right corner is the way to go. (This is indirectly acknowledged on lines 305-306.)\n2. The practical impact for the MT community is unclear. If this trade-off changed the way data compression is evaluated and done, there does not seem to be a clear way to achieve that in MT. Evaluating along multiple dimensions has been tried before (again as rightly pointed out here), and the community has mostly move to one-dimensional metrics that are believed (or hand-waved) to optimize both accuracy and naturalness. It is unclear how this paper would change the state of things.\n\nNote that I did not read the entirety of the massive appendix. There may be further discussions or results there that address my concerns.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting new view on a previously known trade-off, with a neat mathematical framework, and clear relevance in the age of NMT and LLM-based translation.", "reasons_to_reject": "The adequacy-fluency trade-off (and related) has been well-known in MT for a while, as acknowledged in various places in the paper. Beyond the neat mathematical framework, it is unclear what the practical impact of this work is, especially as the empirical results do not seem to agree well with theory. Some interesting lines of research are suggested but mainly left for future work.\nFinally, while interesting to Machine Translation researchers, it is not clear that this work will have widespread interest for the rest of the COLM crowd, even for other NLP applications.", "questions_to_authors": "* l. 167, 240: I realize this may be a personal preference: style guides tend to advise against having a single subsection within a section.\n* l. 178, \"By definition ... perfectly accurate\": do you mean \"optimally\" accurate, given Q*? There is no guarantee that the translation system is capable of perfect translation.\n* l. 212, \"convex in the first slot\": assuming this means D(Q,P) is convex in Q?\n* l. 216: : \"convexity of A(N):\" should arguably be concavity.\n* l. 255: I assume this is convex in Q as per assumptions -- can you clarify it?\n* l. 263: Why is the perfect score zero, is that an assumption or convenience choice with no loss of generality?\n* l. 355-356: Isn't this suggestion of optimizing non-reference metrics (perplexity) essentially how neural models (NMT & LLM) work anyway?", "extra_scores": {}}, {"review_id": "TrugqaVNft", "reviewer": "Reviewer_z3nj", "summary": "This is a nice paper. It argues convincingly that there is a tradeoff between naturalness and accuracy when systems/people translate text. For example, when translating the German sentence \"My teacher_FEMALE is nice\" into English, the translation \"My teacher is nice\" is natural but only partially accurate because it does not make explicit that my teacher is female, which in German is represented explicitly, while the translation\"My female teacher is nice\" is un-natural - monolingual English texts are not written in this fashion.\n\nThe authors extend Blau and Michaeli's (2018) distorsion-perception theory to translation and show mathematically and empirically that there is a tradeoff between accuracy and naturalness when translating texts. Their results can be used to develop better evaluation techniques and enable translation systems to be tuned for different operational points in the accuracy/naturalness plane.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a nice theoretical/practical framework for studying the translation process and evaluating translations produced by humans and/or machines in a richer two dimensional space defined by the accuracy and naturalness axes. \n\nThe authors develop an abstract notion of naturalness based on information theory and use it to create a unifying framework for studying translations across the two axes. From their theory, it mathematically follows that the more one wants to increase the naturalness of a translation system, its accuracy must drop. This is a nice theoretical insight and is consistent with one's intuitions.\n\nThe authors also explore the utility of their theory empirically by examining outputs produced by a variety of systems. This leads to insights that were not possible to derive using prior frameworks.", "reasons_to_reject": "I am most concerned with the heavy use of automatically generated translations in the empirical study (section 5.1). I would have preferred if the authors used\n      1) publicly available MT and human outputs produced in WMT evaluations across several years: MT outputs created by older technology tended to produce more accurate but less fluent translations, while newer technology tends to produce more fluent translations. Showing that the framework introduced in this paper yields accuracy and naturalness score that are consistent with the above observation would strengthen the submission;\n       2) a broader set of translation references, such as that introduced by Dreyer and Marcu (https://aclanthology.org/N12-1017/); comparing the accuracy and naturalness scores across millions of references translations would yield additional insights.\n\nHowever, I find the results sufficient and convincing for the submission to be accepted at COLM'25. Follow-up work can address my concerns. And the community would benefit from learning and debating these findings now and developing public evaluation software/scripts of higher utility.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper posits that in MT there is an accuracy-naturalness tradeoff and (calling back to existing work on the perception-distortion tradeoff, Blau & Michaeli) shows that in theory both cannot be perfectly satisfied at the same time, plus showing in practice how current models do on either objective, not contradicting the theory (though not a stellar endorsement).

Reviewer LQ9z praises the super-conference-level scope of the work and its ability to spark discussion on known concepts, appreciating writing and theory, but also noting a disconnect to the practical results as WMT systems seem very far from the theoretical limits so the proposed behavior near that limit cannot be empirically observed (authors rebut that the theoretical limits are overly optimistic due to computational constraints in their calculation). (They also take issue with the formulation of bitext joint distributions, but I would side with the authors here in that indeed the marginals of such an empirical joint distribution are not created equally.)

Reviewer Z5DR in addition praises the corpus-based naturalness definition, use of existing data, and thorough definitions. They note pathological cases in neural metrics like the ones used here, though the authors believably rebut that behavior in expectation is good enough for their theory.

Reviewer AhdH in addition wonders how this paper could have impact in MT given the community's preference for single metrics over an evaluation along multiple axes---and whether it fits well at COLM (which authors rebut, but I see the reviewer's point, but I really like the paper, so I would argue for it fitting in COLM anyway). They also bring up the question of how accuracy-naturalness can relate to the adequacy-fluency tradeoff that the MT community has chosen to internalize more, but the authors say this simply came down to not wanting to deal with heavy connotation baggage.

Reviewer z3nj also very much appreciates the paper, but adds criticism about the use of automatic translation in the evaluation over existing human and machine translations, especially given that older systems may illuminate different areas of the plane.

In summary, the paper is clearly of high quality and clarity, with sufficient originality and relevance to the community. I personally really like the way it is set up and agree that it should prove to be a very stimulating and interesting paper, even if mildly disconnected from the current reality of systems.
Decision: Accept