RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Tradeoff between p(\boldsymbol{x|y}) and p(\boldsymbol{y})
Because \text{accuracy}_{M} and \text{fluency}_{M} have formal definitions, we start with these dimensions.
Human and machine translation
We now show that human and machine translations show the same tradeoff between \text{accuracy}_{M} and \text{fluency}_{M} , which correspond to p(\boldsymbol{x|y}) and p(\boldsymbol{y}) estimated by an NMT model.

Data. We analyze 15 translation studies from CRITT TPR-DB (CRITT) that include 13 language pairs Carl et al. ( 2016b ) . We also use a subset of the Russian Learner Translator Corpus (RLTC) that has been aligned at the sentence level by Kunilovskaya ( 2023 ) . For machine translation, we use WMT test sets that are annotated with Multidimensional Quality Metrics labels (MTMQM) (Freitag et al., 2021a , b ; Zerva et al., 2022 ; Freitag et al., 2023 ) .
To reduce spurious correlations, we remove duplicate translations and source segments with fewer than four unique translations. Additional details are provided in the Appendix.

Models. We use NLLB-200’s 3.3B variant model (Costa-jussà et al., 2022 ) to estimate p(\boldsymbol{y|x}) and p(\boldsymbol{x|y}) . 4 4 4 NLLB model card For consistency, we also extract p(\boldsymbol{y}) based on the same model, skipping all inputs except for special tokens (e.g., <eos> tags). 5 5 5 To ensure reproducibility across models, we repeat our analysis in Appendix using M2M100 Fan et al. ( 2021 ) . All probabilities are log scaled.

Results. Figure 3 is analogous to the densities in Figure 2(b) , and shows distributions of tradeoff scores for source segments in CRITT, RLTC and MTMQM. In all three cases most source segments induce tradeoffs (i.e. produce negative correlations). To test for statistical significance we compared the actual distributions against randomly permuted data. The results of all paired-sample t-tests are significant ( p<.001 ), and are included in the figure. 6 6 6 Each permuted data set is created by randomly shuffling the pairings of p(\boldsymbol{x|y}) and p(\boldsymbol{y}) within the set of possible translations of each source segment. When samples are aggregated at the corpus level, p(\boldsymbol{y}) and p(\boldsymbol{x|y}) show significant positive correlations ( p<.001 ) for CRITT ( r=.625 ), RLTC ( r=.685 ) and MTMQM ( r=.675 ), showing that Simpson’s paradox applies in all three cases.
Tradeoff between accuracy and fluency
We now turn to human ratings of accuracy and fluency, and demonstrate that the two are again negatively correlated at the segment level.

Data. Only RLTC and MTMQM are rated by human annotators. The subset of RLTC released by Kunilovskaya ( 2023 ) includes accuracy and fluency scores derived from error annotations. For MTMQM, we aggregate accuracy scores based on “Accuracy” errors, and aggregate errors in “Fluency”, “Terminology”, “Style” and “Locale convention” for fluency rating. Targets that are labelled “Non-translation” receive scores of zero for both accuracy and fluency scores. 7 7 7 Further details of accuracy and fluency score derivation are available in Appendix.

Results. Figure 4 shows correlations at the level of individual source segments. The majority of correlations are negative, and paired-sample t-tests reveal that both distributions are significantly ( p<.001 ) different from distributions obtained from random permutations. The results therefore suggest that accuracy and fluency (as rated by humans) trade off at the level of individual segments. At the corpus level, accuracy and fluency are positively correlated for MTMQM ( r=.396,p<.001 ), but not RLTC ( r=-.085,p<.001 ), suggesting that Simpson’s paradox applies only to MTMQM.

Figure 4 is directly analogous to Figure 3 , and we expected that source segments which showed strong tradeoffs (i.e. extreme negative correlations) in Figure 3 would also show strong tradeoffs in Figure 4 . The two tradeoff measures, however, were uncorrelated, 8 8 8 The Pearson correlations between the two tradeoff measures for RLTC and MTMQM are r=.005,p=.885 and r=.024,p=.04 . which suggests that \text{accuracy}_{M} and \text{fluency}_{M} overlap only partially with human ratings of accuracy and fluency.

A similar conclusion is suggested by Figure 5 , which shows Pearson correlations of translation probability ( p(\boldsymbol{y|x}) ), \text{accuracy}_{M} ( p(\boldsymbol{x|y}) ) and \text{fluency}_{M} ( p(\boldsymbol{y}) ) with human ratings of accuracy and fluency for RLTC and MTMQM. 9 9 9 Values are ranked by percentile. As expected, p(\boldsymbol{x|y}) shows a higher correlation with accuracy than fluency, and p(\boldsymbol{y}) shows the opposite pattern. Figure 5 however, suggests that p(\boldsymbol{x|y}) is not superior to p(\boldsymbol{y|x}) as a predictor of accuracy, and that p(\boldsymbol{y}) is not superior to p(\boldsymbol{y|x}) as a predictor of fluency. One reason why our model estimates of accuracy and fluency depart from human ratings is that \text{accuracy}_{M} and \text{fluency}_{M} are sensitive to segment length. For example, a longer segment will have lower \text{fluency}_{M} than a shorter segment even if the two are both perfectly idiomatic.
Conclusion
We showed that accuracy and fluency and p(\boldsymbol{x|y}) and p(\boldsymbol{y}) both trade off when translating individual source segments.
This finding suggests that current protocols for assessing translation quality may need to be adjusted.
Human assessments for recent WMT General Task are performed with Direct Assessment and Scalar Quality Metrics (DA+SQM) Kocmi et al. ( 2022 , 2023 ) .
This approach conflates meaning preservation and grammar into a single score indicative of overall quality of a translation.
In contrast, MQM is much more costly, but produces highly detailed scores that use multiple sub-categories for both accuracy and fluency. Future approaches could therefore consider a middle ground that extends DA+SQM to include accuracy and fluency as independent aspects as in WMT16 Bojar et al. ( 2016 ) . This direction would allow automatic MT evaluation metrics such as BLEURT Sellam et al. ( 2020 ) and COMET Rei et al. ( 2022 ) (both fine-tuned to DA scores) to be adapted to provide independent scores for accuracy and fluency.

Our results also suggest the value of developing MT models that navigate the accuracy-fluency tradeoff in human-like ways. In some settings (e.g. translating legal texts) accuracy is more important Popović ( 2020 ); Martindale and Carpuat ( 2018 ); Vela and Tan ( 2015 ); Specia et al. ( 2011 ); Martindale et al. ( 2019 ) , but in others (e.g. translating informal conversation) fluency may take priority Poibeau ( 2022 ); Frankenberg-Garcia ( 2022 ) . One natural approach builds on noisy channel models Yu et al. ( 2016 ); Yee et al. ( 2019 ); Müller et al. ( 2020 ) , which incorporate both p(\boldsymbol{y}) and p(\boldsymbol{x|y}) along with tradeoff parameters that specify the relative weights of the two. Tuning these parameters for specific registers may allow a model to find the right balance between accuracy and fluency in each case.
Limitations
Although we provided evidence for both accuracy-fluency and \text{accuracy}_{M} - \text{fluency}_{M} tradeoffs in translation, we did not explore factors which predict which source segments are likely to produce the greatest tradeoffs.
Outside of our simulation we do not have access to ground-truth values of p(\boldsymbol{x|y}) and p(\boldsymbol{y}) , and are only able to approximate these values using specific NMT models. Our work is also limited by the fact that MTMQM only includes translations generated by certain kinds of NMT models, and it is possible that our results do not generalize to translations generated by other types of models, such as statistical or rule-based MT systems. Finally, both RLTC and MTMQM have accuracy and fluency ratings derived from error annotations that are very similar in range. This constraint makes quality assessment and comparison at the segment level challenging.
Ethics Statement
We do not foresee any potential risks and harmful use of our work. Our analyses are based on licensed data which are freely available for academic use.
A.1 Simulation
We assume \boldsymbol{x} and \boldsymbol{y} are drawn from a Gaussian joint distribution P(\boldsymbol{x},\boldsymbol{y}) where both \boldsymbol{x} and \boldsymbol{y} are centered at zero. We set an initial square matrix A with dimensionality equal to the total number of dimensions in \boldsymbol{x} and \boldsymbol{y} combined. Assuming all elements in \boldsymbol{x} and \boldsymbol{y} have \sigma^{2}=1 and pairwise positive covariance, all diagonal elements of A are set to 1 and other elements 0.7. To ensure the covariance matrix is positive semi-definite, we replace the initial matrix A with a final covariance matrix defined as A^{\top}A .

Based on this setup we first draw 100 random samples of \boldsymbol{x} from P(\boldsymbol{x}) . For each \boldsymbol{x} we then sample 100,000 values of \boldsymbol{y} from a distribution q(\boldsymbol{y})=\prod_{i}q(y_{i}) , where each element y_{i} of \boldsymbol{y} is sampled uniformly within two standard deviations of its mean. We then compute p(\boldsymbol{x|y}) , p(\boldsymbol{y}) and p(\boldsymbol{y|x}) for each ( \boldsymbol{x} , \boldsymbol{y} ) pair using the known joint P(\boldsymbol{x},\boldsymbol{y}) .
A.2.1 Corpora
CRITT Translation Process Research Database Carl et al. ( 2016b ) is a collection of translation behavioural data in the area of Translation Process Research. From the public CRITT database we obtain 15 studies across 13 pairs of languages: RUC17 (enzh, Carl and Báez, 2019 ), ENJA15 (enja, Carl et al., 2016a ), NJ12 (enhi, Carl et al., 2016b ), STC17 (enzh, Carl and Báez, 2019 ), SG12 (ende, Nitzke, 2019 ), ENDU20 (ennl, Vanroy, 2021 ), BML12 (enes, Mesa-Lao, 2014 ), ACS08 (daen, Sjørup, 2013 ), MS13 (ptzh, Schmaltz et al., 2016 ), JLG10 (pten, Alves and Gonçalves, 2013 ), BD13 (daen, Dragsted, 2010 ), LWB09 (daen, Jensen et al., 2009 ), DG01 (plfr, Płońska, 2016 ), BD08 (daen, Dragsted, 2010 ) and CREATIVE (enzh, Vieira et al., 2023 ). 10 10 10 https://sites.google.com/site/centretranslationinnovation/tpr-db/public-studies After deduplication and removing source segments with fewer than 4 unique translations, the total number of source segments included is 399, each with an average of 10.9 unique translations.

RLTC is a subset of the Russian Learner Translator Corpus that has been segment-aligned by Kunilovskaya ( 2023 ) . We include a total of 1079 source segments from 5 genres: ‘Essay’, ‘Informational’, ‘Speech’, ‘Interview’ and ‘Educational’. The average number of unique translations for each source segment is 10.5.

MTMQM is obtained from Freitag et al. ( 2021a ) , which contains translations of TED talks and news data from the test sets of WMT general task between 2020 and 2023. 11 11 11 https://github.com/google/wmt-mqm-human-evaluation The translations are annotated with MQM labels. After preprocessing we are left with 11219 source segments and an average of 9.9 unique translations per source segment.
A.2.2 Accuracy and fluency scores by MQM
For MTMQM, we aggregate accuracy and fluency penalty score by error annotations, and based on weights recommended by Freitag et al. ( 2021a ) . Major and minor errors receive penalties of 5 and 1 respectively. Fluency/Punctuation is assigned a penalty of 0.1. We calculate the final rating as s_{c}=\max(0,25-e_{c}) , where e_{c} denotes the total penalty in error category c . 12 12 12 The maximum score is set at 25 because a translation can only get a maximum of 25 penalty score. Because some systems submit the same translation but receive different ratings, we average these scores and remove the duplicate entries.
A.3 Alternative result with M2M100 translation model
In Figure 6 and 7 , we replicate our findings of \text{accuracy}_{M} and \text{fluency}_{M} in Section 2 and 3 with estimates of M2M100 (1.2B variant) Fan et al. ( 2021 ) . 13 13 13 https://huggingface.co/facebook/m2m100_1.2B
A.4 Tradeoff examples
Tables A.4 , A.4 and A.4 include the full set of translations plotted in Figure 1 . The tables specify accuracy, fluency, \text{accuracy}_{M} , \text{fluency}_{M} and translation probability p(\boldsymbol{y|x}) for each segment. All translations listed are submissions to WMT General Task between 2020 to 2022.
Figure 1: Simpson’s paradox. Each panel shows translations of three source segments indexed by color and marker shape. At the source segment level, accuracy and fluency (left) and p(\boldsymbol{x|y}) and p(\boldsymbol{y}) (right) both show negative correlations. At the corpus level, both pairs of dimensions show positive correlations ( p<.05 ; see panel labels). Source segments and translations are drawn from past WMT General Task submissions and data points have been jittered for clarity.; Simpson’s paradox. Each panel shows translations of three source segments indexed by color and marker shape. At the source segment level, accuracy and fluency (left) and p(\boldsymbol{x|y}) and p(\boldsymbol{y}) (right) both show negative correlations. At the corpus level, both pairs of dimensions show positive correlations ( p<.05 ; see panel labels). Source segments and translations are drawn from past WMT General Task submissions and data points have been jittered for clarity.
(a) Simulation with one-dimensional \boldsymbol{x} and \boldsymbol{y} . The three panels correspond to three different source “segments” \boldsymbol{x} of decreasing probability p(\boldsymbol{x}) , and the points in each panel are candidate translations \boldsymbol{y} . Brighter colors indicate translations with larger p(\boldsymbol{y|x}) . Pearson correlations across translations ranked in the top 10% based on p(\boldsymbol{y|x}) are shown at the top of each panel.; (b) Kernel density plots of tradeoffs across the top 10% (left) and across all translation choices (right). The tradeoff persists in higher dimensional space, and is stronger when selecting only \boldsymbol{y} with the highest values of p(\boldsymbol{y|x}) .; Tradeoffs between p(\boldsymbol{x|y}) and p(\boldsymbol{y}) in synthetic data.
Figure 3: Tradeoffs between estimated p(\boldsymbol{x|y}) and p(\boldsymbol{y}) across source segments from three corpora. Paired-sample t-tests against randomly permuted p(\boldsymbol{y}) and p(\boldsymbol{x|y}) are shown at the top of each panel.; Tradeoffs between estimated p(\boldsymbol{x|y}) and p(\boldsymbol{y}) across source segments from three corpora. Paired-sample t-tests against randomly permuted p(\boldsymbol{y}) and p(\boldsymbol{x|y}) are shown at the top of each panel.
Figure 4: Tradeoffs between human ratings of accuracy and fluency across segments from two corpora. Paired-sample t-tests against randomly permuted scores are shown at the top of each panel.; Tradeoffs between human ratings of accuracy and fluency across segments from two corpora. Paired-sample t-tests against randomly permuted scores are shown at the top of each panel.
Figure 5: \text{accuracy}_{M} and \text{fluency}_{M} predict human accuracy and fluency ratings for RLTC and WMT submissions to the general translation task in 2022 and 2023. zhen and ende refer to Chinese-English and English-German language pairs. All correlations reported are significant ( p<.001 ).; \text{accuracy}_{M} and \text{fluency}_{M} predict human accuracy and fluency ratings for RLTC and WMT submissions to the general translation task in 2022 and 2023. zhen and ende refer to Chinese-English and English-German language pairs. All correlations reported are significant ( p<.001 ).
Figure 6: Histogram of tradeoffs between estimated p(\boldsymbol{x|y}) and p(\boldsymbol{y}) estimated by M2M100, which is analogous to Figure 3 in the main text. When analyzed at the corpus level, the correlations ( p<.001 ) for CRITT, RLTC and MTMQM are .689 , .703 and .801 respectively.; Histogram of tradeoffs between estimated p(\boldsymbol{x|y}) and p(\boldsymbol{y}) estimated by M2M100, which is analogous to Figure 3 in the main text. When analyzed at the corpus level, the correlations ( p<.001 ) for CRITT, RLTC and MTMQM are .689 , .703 and .801 respectively.
Figure 7: \text{accuracy}_{M} and \text{fluency}_{M} estimated by M2M100 are also predictive of human accuracy and fluency ratings ( p<.05 ). The figure is analogous to Figure 5 .; \text{accuracy}_{M} and \text{fluency}_{M} estimated by M2M100 are also predictive of human accuracy and fluency ratings ( p<.05 ). The figure is analogous to Figure 5 .
Translations of “ I issued you a refund of the book. " from English to German, which correspond to three of the orange dots in Figure 1 . Option (i) is acceptable but gab (past tense of give) is less accurate than the conjugations of ausstellen (issue) used in (ii) and (iii). Option (iii) is the least natural because stellte … aus (Präteritum tense) is typically used only in formal writing.