Venue: NAACL 2025

# How Good Are LLMs for Literary Translation, Really?

Literary Translation Evaluation with Humans and LLMs

Ran Zhang\({}^{1,4}\) Wei Zhao\({}^{2}\) Steffen Eger\({}^{3,4}\)

\({}^{1}\)University of Mannheim, School of Business Informatics and Mathematics

\({}^{2}\)University of Aberdeen, Department of Computing Science

\({}^{3}\)University of Technology Nuremberg (UTN), Dep. Computer Science & AI

\({}^{4}\) Natural Language Learning and Generation (NLLG) Lab

###### Abstract

Recent research has focused on literary machine translation (MT) as a new challenge in MT. However, the evaluation of literary MT remains an open problem. We contribute to this ongoing discussion by introducing LitEval-Corpus, a paragraph-level parallel corpus containing verified human translations and outputs from 9 MT systems, which totals over 2k translations and 13k evaluated sentences across four language pairs, costing 4.5k. This corpus enables us to (i) examine the _consistency_ and _adequacy_ of human evaluation schemes with various degrees of complexity, (ii) compare evaluations by students and professionals, assess the effectiveness of (iii) LLM-based metrics and (iv) LLMs themselves. Our findings indicate that the adequacy of human evaluation is controlled by two factors: the complexity of the evaluation scheme (more complex is less adequate) and the expertise of evaluators (higher expertise yields more adequate evaluations). For instance, MQM (Multidimensional Quality Metrics), a complex scheme and the de facto standard for non-literary human MT evaluation, is largely inadequate for literary translation evaluation: with student evaluators, nearly 60% of human translations are misjudged as indistinguishable or inferior to machine translations. In contrast, BWS (Best-Worst Scaling), a much simpler scheme, identifies human translations at a rate of 80-100%. Automatic metrics far dramatically worse, with rates of at most 20%. Our overall evaluation indicates that published human translations consistently outperform LLM translations, where even the most recent LLMs tend to produce considerably more literal and less diverse translations compared to humans.

## 1 Introduction

With the advent of Large Language Models (LLMs), literary translation, once the exclusive domain of human translators [7, 1], now stands at the threshold of a technological transformation [12, 13]. LLMs have the potential to increase the volume of translated literary works, particularly for low-resource languages [16]. However, they pose a dual threat: potentially decreasing the salaries of professional translators or even displacing them entirely and diminishing the aesthetic value of translations if LLMs' output quality is persistently overestimated. As a consequence, the field has received much increased attention recently, both in the academic domain [23, 1, 15, 16, 17, 18] and in the translation community [19, 24, 25].

Evaluating literary translation poses unique challenges that extend beyond mere linguistic accuracy. It demands a delicate balance of cultural insight, authorial voice, creative interpretation, and aesthetics [26, 27, 28]. These nuances surpass the scope of traditional sentence-level MT

Figure 1: Adequacy comparison per scheme complexity and evaluator expertise. SQM represents Scalar Quality Metrics. Professional and student evaluators perform free and MQM error annotation, respectively. See Figure 4 and Section F.5 for more details.



evaluation methods typically used in non-literary domains, presenting substantial obstacles for existing assessment techniques Van Egdom et al. (2023).

We notice the following gaps in existing research regarding literary translation: (1) For **automatic evaluation**, previous metrics like d-BLEU, BLEURT and BLONDE Liu et al. (2020); Sellam et al. (2020); Jiang et al. (2022) fall short in discrimination between professional human translations and machine outputs in literary MT Thai et al. (2022). More powerful recent popular metrics such as XCOMET Guerreiro et al. (2024), GEMBA-MQM Kocmi and Federmann (2023), and Prometheus Kim et al. (2024) remain untested for literary texts. (2) The adequacy of **human evaluation** frameworks such as Multidimensional Quality Metrics (MOM) Freitag et al. (2021); Lommel et al. (2014), Scalar Quality Metric (SQM) Graham et al. (2013), and Best-Worst Scaling (BWS) Kiritchenko and Mohammad (2017) remains largely unexplored: to our knowledge, there is no comprehensive comparison of these schemes for evaluating literary translations. This critical gap is concerning, as current studies apply these approaches arbitrarily, resulting in findings that are not directly comparable and potentially leading to unreliable conclusions. (3) Existing **datasets** are small, cover few LLM models, and lack verification for human translations, including translator qualifications and experience Yan et al. (2024). These issues can lead to substantial misjudgments when comparing LLM outputs to human translations, potentially overestimating the abilities of LLMs to the detriment of human professional translators.

In this paper, we address these gaps with the following key contributions:

* We introduce LitEval-Corpus, a paragraph-level parallel corpus with verified high-quality human translations across four language pairs. This corpus also facilitates the comparison of 9 MT systems beyond the GPT series, including commercial models, popular LLMs, and previous state-of-the-art (SOTA) MT systems.1 Footnote 1: Our dataset and code are publicly accessible on GitHub: [https://github.com/zhangr2021/LitMT_eval](https://github.com/zhangr2021/LitMT_eval).
* We investigate multiple human evaluation schemes through both students and professional translators, illustrated in Figure 1, where we find that error annotation including MQM, the de facto measure in standard human MT evaluation, is (at least partially) inadequate for literary translation; SQM's effectiveness for comparing high-quality outputs highly depends on the expertise of evaluators; and BWS proves best for comparing high-quality outputs.
* We examine the effectiveness of popular LLM-based metrics. GEMBA-MQM consistently outperforms other metrics. However, it primarily focuses on accuracy errors and struggles to differentiate between human translations and literal LLM outputs: it prefers human professional translations over LLM output at a level of 9.6%, compared to 94.4% for professional SQM, 81.1% for student BWS and 42% for student MQM and SQM.
* We compare system performance, where we find that (1) human translators produce the best translations, as judged by both professionals and students; (2) GPT-4o ranks second, followed by Google Translate and DeepL. LLM outputs are more literal and less diverse than human translations with recent models decisively better and more closely approaching human literary translation.

## 2 Related work

Human evaluation for literary textsLiterary MT evaluation primarily relies on human judgments of various types. MQM, the widely used guided error annotation scheme in the renowned MT venue WMT Wang et al. (2023); Karpinska and Iyyer (2023), has been shown effective for non-literary texts Freitag et al. (2021) but its suitability for literary translation is uncertain. However, Kocmi et al. (2024) indicate that the complex MQM error categorization system requires costly trained experts and may lead to low annotation agreement. BWS is also employed for literary texts, including poetry Thai et al. (2022); Belouadi and Eger (2023), but has limitations in comparing multiple systems simultaneously Kiritchenko and Mohammad (2017). Recently, Wu et al. (2024) claim that LLM-based multi-agent literary translations achieve parity with human translations, based on preference comparison by crowdsourced and LLM evaluator--though the expertise of human translators remains unclear. SQM, a Lik 

ert scale rating, is used in non-literary domains. To our knowledge, a comprehensive comparison of these schemes for literary MT is lacking, with different studies employing various methods ad libitum. Our work provides the first systematic comparison of these evaluation methods for literary texts.

**Human evaluation** of language models has been the subject of recent debate Thomson et al. (2024); Belz et al. (2023). Human evaluation is typically assessed via _consistency_, i.e., whether human annotations are similar. Besides consistency, we focus on _adequacy_: whether the implications of evaluation are plausible (e.g., whether human reference texts are ranked the highest) Toral et al. (2018).

**Automatic metrics** for MT have been extensively studied at sentence and document levels in non-literary domains. These include NLI-based MENLI Chen and Eger (2023), trained metrics like the COMET series Guerreiro et al. (2024), BERT-based scores such as BLEURT Sellam et al. (2020) and BERTScore Zhang et al. (2020), prompting-based metrics like GEMBA-MQM Kocmi and Federmann (2023) and MQMPAE Lu et al. (2025), in addition to documented BLONDE Jiang et al. (2022) and DiscoScore Zhao et al. (2023a). While some researchers have applied these metrics to literary texts Wang et al. (2023); Hansen and Esperancag-Rodier (2022), their effectiveness in literary domain remains understudied. Previous work has examined BLEU, BLEURT, and BLONDE for identifying top-performing systems Van Egdom et al. (2023); Thai et al. (2022), with limited correlation analysis (e.g., brief discussions in appendices) of COMET, COMET-QE, BLEURT, and BERTScore Karpinska and Iyyer (2023). Our study addresses this gap by evaluating the effectiveness and adequacy of recent LLM-based metrics for literary translation.

**Corpora for literary MT evaluation** Existing parallel corpora are insufficient for literary translation evaluation. Table 4 (appendix) summarizes the statistics. _BWB_Jiang et al. (2022) and _GuoFeng_Xu et al. (2022) focus on recent _Zh-En_ web novels with unclear translation origins, potentially post-edited MT outputs Kolb (2023). _PAR3_Thai et al. (2022) lacks crucial meta-information, such as the origin of human translations, and offers limited evaluation with small-scale A/B tests comparing outputs from only three generation systems Karpinska and Iyyer (2023) compare two systems Google Translate and GPT-3.5) with only contemporary works--usually easier to translate than classics and potentially involving post-edited MT. Our LitEval-Corpus addresses these limitations, containing over 2k paragraph-level segments with 13k sentences, comparing nine systems against verified published human translations with human evaluation under multiple schemes.

## 3 LiteEval-Corpus

We detail our LitEval-Corpus in this section, covering data construction and statistics.

### Dataset construction

Our data construction process consists of 3 steps: (1) preselect and construct paragraph-level parallel corpus; (2) select MT systems and collect translation hypotheses; (3) assess translation quality.

**Constructing parallel corpus** We consider three languages: English, Chinese, and German, covering translations between four different language pairs, i.e., English-Chinese (_E-Zn_), German-English (_De-En_)/English-German (_En-De_), and German-Chinese (_De-Zh_).2 We begin by collecting paragraph-level parallel corpus for each language pair, including both classic and contemporary works. Classics present more challenges due to their complex syntax, language change, and cultural references that may no longer be familiar to modern audiences. Notably, older translations are especially valuable, as they were created before the widespread use of MT and are thus free from concerns of being post-edited MT outputs. Additionally, including contemporary works helps mitigate potential data contamination in LLMs, as these models are more likely to have been pre-trained on widely available texts, including classic literature.

Footnote 2: The authors are native/proficient in these three languages and can thus verify the quality of the corpus and the evaluation.

To ensure the quality of our dataset, we derive both source and target paragraphs from verified publications, including books from Project Gutenberg Stroube (2003), reading samples from published works, and purchased materials. For classic works, the target paragraphs include at least two human translations, while contemporary works have at least one. For 

we utilize PAR3 samples (Thai et al., 2022).3 For contemporary _De-En_ works and the other three language pairs (_De-Zh_, _En-Zh_, and _En-De_), we build the corpus from scratch, where the first author manually aligns source and translation paragraph pairs.

Footnote 3: PAR3 includes 9k _De-En_ paragraphs aligned with human translations but has alignment issues for some paragraphs with unidentified translation origins. To address these, the first author proofread sampled source-translation pairs and verified the origin of each human translation.

Selecting MT systems and collecting translation hypothesesWe explore various-sized models, including cost-efficient alternatives to larger models:

* **Commercial systems** like Google Translate and DeepL. DeepL is particularly noteworthy for translators due to its widespread use within the translation community.
* Previous SOTA **transformer models** optimized for sentence-level translation, including NLLB-3.3b (Costa-jussa et al., 2022) and M2M_100-1.3b (Fan et al., 2021).
* Closed-source GPT-4o and four series of open-source **LLMs** of smaller sizes:4 (1) Llama 3 (AI@Meta, 2024), one of the strongest open-source models; (2) Owen 2 (Yang et al., 2024), specialized in Chinese; (3) Gemma 1.1 (Team et al., 2024), developed with the same technology as the closed-source Gemini model (Team et al., 2023); and (4) TowerInstruct (Alves et al., 2024), trained for translation-related tasks. We use identical prompts across all LLMs when requesting literary translations: _"Please translate the following literary texts from [source language] to [target language]. The texts are as follows: [texts]".5_ Footnote 4: For detailed parameter size, see Table 7 (appendix).

### Assessing translation quality

Our study examines three human evaluation schemes, i.e., guided/free error annotation, SQM, and BWS, using both student evaluators and professional translators. Figure 4 (appendix) summarizes the evaluation schemes. For **guided error annotation** (**MQM**), evaluators read the source/candidate translation pair, identify error spans, and categorize them based on the annotation guideline with six major categories: (1) Terminology; (2) Accuracy; (3) Fluency; (4) Style; (5) Locale-convention; and (6) Non-translation.6 Each error is then labeled as major or minor. The MQM score is thus computed with the following formula:

Footnote 6: See Section E.2 (appendix) for annotation details.

\[-\frac{(C_{\text{Non-translation}}\times 25+C_{\text{major}}\times 5+C_{\text{minor }}\times 1)}{\text{number of sentences per paragraph}}\]

\(C\) represents the count of errors. In contrast, **free error annotation** does not follow any guidelines. The evaluators highlight spans indicating erroneous/good translations and explain their highlights. We can thus compute span-level agreement with student error annotation.7 For **SQM**, evaluators provide a scale rating between 0-6 for the overall quality of the translation, especially the translation's stylistic and artistic qualities based on the source. Evaluators may scroll up and down to check all other translations for scoring comparison. **BWS** is a direct comparison method that identifies the best and worst translations.

Footnote 7: Our feedback from translators revealed that MQM error annotation is unfamiliar to the translation community. Free error annotation better reflects the editing process and enables us to assess MQM protocol by comparing error span matches.

Evaluator statisticsWe involve four student evaluators, one for each language pair, and four professional translators (one for _En-Zh_, two for _En-De_, and one for _De-En_). All evaluators must be native speakers of the target language. Three students are master students with bachelor degrees in linguistics (_De-Zh_), computational linguistics (_De-En_ and _En-De_) and one senior bachelor in translation studies (_En-Zh_). The student evaluators for _De-Zh_, _De-En_, and _En-Zh_ are female and the _En-De_ evaluator is male. For professional evaluators, we recruit two literary translators with verified publication records for ten hours each. The female translator for _En-Zh_ has published a translation of a classic work with a public rating of 9.3/10. The female _En-De_ translator is highly experienced, having published over 30 translated works. The other two translators volunteer to evaluate for six hours each. Both are female, recruited by universities, and highly experienced in academic translation.

We divide our human evaluation into three groups due to budget constraints: 

* Students conduct MQM and SQM evaluation for _all segments_ using Label Studio platform Tkachenko et al. (2020-2022).
* The same students perform BWS on _samples_, comparing translations from human translators, GPT-4o, DeepL, and Google Translate (Qwen for pairs involving Chinese) blindly.8 Footnote 7: Evaluating all systems in one entry with 10-12 segments may decrease the reliability of BWS Küritchenko and Mohammad (2017). We include 4-5 translation segments from the top-performing systems to achieve the best outcome.
* Professional translators perform free error annotation and SQM on _samples_ from human translators, GPT-4o, Google Translate, and DeepL (Qwen for pairs involving Chinese).9 Footnote 6: We focus on segments from top systems per feedback from professionals. Other translations often contain major issues, making them less valuable for professional assessment (e.g., highlighting several problematic sentences).

### LitEval-Corpus statistics

Table 1 summarizes the statistics of LitEval-Corpus. The dataset contains 188 source paragraphs, encompassing 2,188 translation segments at paragraph level including over 13k sentences. On average, source paragraphs contain 5.6 sentences, while target paragraphs have 6.1 sentences. To our knowledge, LitEval-Corpus is the most extensive dataset for literary translation evaluation, offering comprehensive information on both source and target translations. It uniquely features paragraph-level translations across 10 generation systems.10

Footnote 10: See Table 5 (appendix) for information on source and translation details.

## 4 Human evaluation

We use LitEval-Corpus to examine the consistency and adequacy of different schemes.

#### Consistency

Student vs. studentTable 2 (a) shows the annotation agreement between pairs of student evaluators across three language pairs: _De-En_, _En-De_, and _En-Zh_.11 We assess MQM and SQM using Kendall's Tau Puka (2011). For BWS and annotated error spans, we employ Cohen's kappa McHugh (2012). A span match indicates agreement on error spans regardless of labels, and a label match considers agreement on both spans and labels.12

Footnote 11: _De-En_ and _En-De_ evaluators annotate shared segments for both language pairs. A co-author, a native Chinese speaker with a double major in linguistics and fluent in English, evaluates _Ent-Zh_ samples. For _De-Zh_, we lack additional qualified evaluators.

The results show _moderate to high agreement for both MQM and SQM_. MQM agreement ranges from 0.434 for _En-De_ to 0.582 for _En-Zh_, while SQM scores the highest agreement for _En-Zh_ at 0.656 and the lowest for _En-De_ at 0.350. On average, MQM yields an agreement of 0.493, similar to SQM of 0.487.

In terms of error span agreement, _En-Zh_ achieves the highest span match at 0.452, while _En-De_ shows the highest label match at 0.283. Al

\begin{table}
\begin{tabular}{l c c c c c} \hline \multirow{2}{*}{**pair**} & \multirow{2}{*}{**\#paragraph**} & \multirow{2}{*}{**\#segments**} & \multicolumn{2}{c}{**fsentences**} & \multicolumn{2}{c}{**\#sent/para**} \\  & & & & **source** & **target** \\ \hline _De-En_ & 46 & 562 & 4310 & 7.1 & 7.7 \\ _En-De_ & 46 & 554 & 2790 & 4.4 & 5.0 \\ _De-Zh_ & 47 & 520 & 3093 & 5.3 & 5.8 \\ _Eu-Zh_ & 49 & 552 & 3162 & 5.5 & 5.7 \\ \cline{2-6} Total & 188 & 2188 & 13.01 & 5.6 & 6.1 \\ \hline \end{tabular}
\end{table}
Table 1: Characteristics of LitEval-Corpus. #sentences and #sent/para represent the total number of annotated sentences in translations and the number of sentences per paragraph, respectively. #paragraph and #segments represent the number of source paragraphs and translation segments from all systems, respectively.

 

though error span agreement is generally _lower than MQM and SQM, the results remain competitive_ with previous sentence-level multi-domain findings Blain et al. (2023); Freitag et al. (2021); Leiter et al. (2023). However, a direct comparison of agreement on paragraph-level annotations in literary domain has yet to be explored, underscoring the value of this study as a baseline for future research in literary translation evaluation.

Additionally, _BWS results indicate strong performance_, with _En-Zh_ achieving the highest agreement at 0.581, followed by _De-En_ at 0.578 and _En-De_ at 0.564 measured by Cohen's kappa. The overall mean BWS agreement is 0.574, showing a consistent agreement in selecting the best and worst translations.

**Professional translators vs. student evaluators** Table 2 (b) compares the annotation agreement between student and professional evaluators (S-P). For comparability, we also report the agreement between student evaluators (S-S) computed with the same segments. For SQM, S-P agreement is on average slightly higher than S-S agreement. For _En-Zh_, the agreement is 0.359 (S-S) versus 0.517 (S-P). One exception is _De-En_ pair, where S-S agreement is 0.528, while S-P agreement drops to 0.363. The agreement for _En-De_ is low for both groups. For error span match, the agreements between S-S and S-P both decline. Overall, S-S shows better agreement than S-P. This is understandable as student evaluators are trained with an annotation guideline, while professional translators perform free annotation without guidelines. The results indicate that both student and professional evaluators focus on joint error spans to some degree.13 While there is _consistency between the two groups_, particularly in certain language pairs, _discrepancies exist_. We discuss these in detail in Section 4.2.

Footnote 13: See Section E.3 for detailed examples.

Consistency in system rankingThe evaluation results in Table 7 (appendix) show strong consistency in system rankings between student and professional evaluators for both MQM and SQM. Human translation consistently ranks first in both scores when averaged across all segments in the four languages studied. This top ranking holds for both student and professional evaluators, indicating human translation's superiority. GPT-4o ranks second, followed by Google Translate and DeepL or Qwen. Smaller models such as M2M and NLLB consistently rank low. Additionally, according to student evaluators, there is a clear performance gap between the top 4 systems and the rest, i.e., 5.5 points drop from the fourth to fifth model in MQM and 1.1 points by SQM. Notably, professional translators rate human translations substantially higher than the second-place GPT-4o with a difference of 1.8 points in SQM, while this gap is comparatively smaller with 0.4 in student SQM.

### Adequacy

We assess the adequacy of translation evaluation schemes by evaluating their ability to differentiate high-quality human translations from machine outputs. We believe that experienced human translators outperform MT systems, based on recent studies for other domain texts Yan et al. (2024) and the inherent advantages of professional translators: access to full-text context, professional education, and the ability to research cultural backgrounds. These factors enable translators to develop a deeper understanding of the source material, intuitively resulting in more accurate and creative translations.

Table 3 shows the percentage of segments where human translations are preferred over other systems. The table presents two scenarios: (1) human translations compared to top-performing models like GPT-4o, DeepL, Google Translate, and Qwen, and (2) human translations compared to other MT systems, excluding top-performing models.

While student **MQM** demonstrates good performance in scenario (2), the bad performance in scenario (1) indicates that _MQM alone may not be adequate for evaluating translation quality effectively, especially for top systems_. The first reason is that MQM requires complex categorization and expects high expertise. This is evidenced by the inferior consistency score for label match in Table 2 (a) from student evaluators. The second reason is that MQM may be inherently unsuitable for literary translation evaluation. MQM struggles with evaluating top models. For example, Figure 5 (appendix) illustrates that the error distribution of top models closely resembles that of human translation. 16.6% of GPT-4o outputs are "error-free" but still fall short in wording or overall style compared to human translations. Professional translators also express concerns about 

the "error-counting" schemes, particularly when assessing high-quality literary translations, as they sometimes make intentional choices, such as omitting certain elements or slightly altering meanings, to enhance the expression in the target language.14

Footnote 14: See examples in Table 9 (appendix).

On average, **SQM** with students performs similarly to MQM but professionals show a substantially higher preference for human translation (100% in _De-En_ and _En-Zh_). Professional SQM outperforms student SQM by a large margin of nearly 40 points, because students--lacking sufficient competence or experience--may struggle to capture nuances through SQM rating as effectively as professionals do. This indicates that SQM's effectiveness heavily depends on the evaluator's experience and expertise. However, combining student MQM and SQM can empirically improve the adequacy by a maximum margin of 20 points for _De-En_ and nearly 5 points for _En-De_, _En-Zh_ and _De-Zh_ compared to SQM or MQM alone (see Figure 6 in Section D.4 for more details).

**BWS**, on the other hand, consistently shows high ratios among top systems, with students preferring human translations at 86.7% for _En-De_ and 95.0% for _De-Zn_. On average, BWS ranks second with 82.1% compared to \(\sim\)94% for professional SQM and \(\sim\)42% for student SQM and MQM. This suggests that BWS is more suitable for evaluating top systems, although it cannot provide the detailed error insights that MQM offers.

Summary:Student and professional evaluators consistently evaluate literary translations using guided/free error annotation and SQM at a moderate level with BWS showing slightly better agreement. Student MQM is not adequate to discriminate human translations from top LLM translations. Using it may yield false conclusions. The effectiveness of SQM heavily depends on the expertise of the evaluator. BWS excels at distinguishing human translations from top machine translations even among less-experienced evaluators, though this scheme offers less detailed feedback for further improvements such as post-editing.

## 5 Automatic metrics

We use LitEval-Corpus to assess recent automatic metrics. While human MQM and SQM may not fully capture the nuances of literary translation quality, as discussed in the previous section, it is still valuable to examine how consistently current metrics align with human MQM/SQM evaluations. This is particularly relevant given that the recent top-performing metrics are built on similar principles. We focus on reference-free metrics due to the scarcity of reference translations for the majority of literary works Rutherford et al. (2024) and measure the performance by segment-level correlation between automatic metrics and human evaluation.

We examine four SOTA automatic metrics that rely on LLMs: Prometheus 2 (7 billion parameters) Kim et al. (2024), an open-source model trained to evaluate other language models, which is queried using a tailored prompt (shown in Table 13 in the appendix) for SQM evaluation of literary translations; XCOMET-XL/XCOMET-XXL (3.5 and 10 billion parameters) Guerreiro et al. (2024), one of the strongest open-source MT metrics for standard MT, which is fine-tuned to assess quality, generating scores and error spans with severity labels;15 GEMBA-MQM Kocmi and Federmann (2023), a top-performing prompting-based metric, which is designed to de

\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{Human > Top Systems} & \multicolumn{6}{c}{Human > Other Systems} \\ \cline{2-10}
**pair** & \multicolumn{4}{c}{(GPT-do, DeepI, Google Translate, and Qwen)} & \multicolumn{4}{c}{(excluding Top systems)} \\ \cline{2-10}  & \multicolumn{3}{c}{**Student**} & \multicolumn{3}{c}{**Professional**} & \multicolumn{3}{c}{**GEMBA-MOM**} & \multicolumn{3}{c}{**Student**} & \multicolumn{3}{c}{**GEMBA-MOM**} \\ \cline{2-10}  & **MQM** & **SQM** & **BWS** & **SQM** & **Original** & **Literary** & **MQM** & **SQM** & **Original** & **Literary** \\ \hline \multirow{3}{*}{_De-En_} & 60.0\% & 60.0\% & 66.7\% & 100.0\% & 6.7\% & 0.0\% & 86.7\% & 80.0\% & 26.7\% & 33.3\% \\  & 35.0\% & 60.0\% & 86.7\% & 83.3\% & 0.0\% & 10.0\% & 95.0\% & 100.0\% & 70.0\% & 60.0\% \\  & 30.0\% & 25.0\% & 95.0\% & – & 15.0\% & 20.0\% & 95.0\% & 90.0\% & 45.0\% & 50.0\% \\  & 45.8\% & 25.0\% & 80.0\% & 100.0\% & 0.0\% & 8.3\% & 87.5\% & 79.2\% & 41.7\% & 50.0\% \\  & 42.7\% & 42.5\% & 82.1\% & 91.4\% & 5.4\% & 9.6\% & 91.0\% & 87.3\% & 45.8\% & 48.3\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Percentage of segments where human translations are preferred over machine translations per human evaluation schemes and GEMBA-MQM versions.

 

tect translation quality errors based on the MQM framework using LLMs.16 We modify GEMBA-MQM with the same error categories we use for student evaluators. We implement GEMBA-MQM (Original) using the original prompt template and GEMBA-MQM (Literary) adapted with domain knowledge and literary translation examples shown in Table 11.

Footnote 16: We utilize GPT-4o mini for GEMBA-MQM in this study due to cost reasons. See Section F.1 and F.2 (appendix) for more details.

**The best metric (GEMBA-MQM) correlates moderately with human MQM but it cannot distinguish human translation from LLM outputs.** Figure 2 (a) illustrates the segment-level correlation between human MQM and automatic metrics. GEMBA-MQM, particularly its Original version, consistently outperforms other metrics. GEMBA-MQM (Literary) performs similarly to GEMBA-MQM (Original), though slightly worse across the board. XCOMET-XL and XCOMET-XXL models rank second overall, demonstrating poor to moderate correlation with human evaluation. Their performance notably declines for more distant language pairs, such as _De-Zh_, possibly due to differences or limitations in their training data for this language direction. Moreover, we notice that XCOMET-XL and XCOMET-XXL are negatively correlated with the length of the target translation at -0.420 and -0.393, respectively. In contrast, the correlations between the length of the target and other metrics are all below \(0.05\) in absolute value. Additionally, Prometheus performs the worst, assigning a score of 4 (equivalent to "good" translation) to 54.9% of segments. Its correlation to human SQM ranks lowest among all automatic metrics, as shown in Figure 9 (appendix).

**Even the best automatic metric is not adequate for literary MT evaluation.**GEMBA-MQM, despite moderate correlation with human SQM and MQM, faces adequacy issues in literary translation evaluation. Table 3 shows the percentage of segments where GEMBA-MQM prefer human translations over MT. The results show GEMBA-MQM severely struggles to distinguish human translations from top LLM systems. For scenario (1), human translator vs. top systems, GEMBA-MQM (Literary)'s highest percentage (9.6%) is still 32.9 points lower than the worst human performance. This may partly stem from LLMs' tendency to act as narcissistic evaluators, favoring their own outputs--especially in reference-free scenarios Liu et al. (2024). In scenario (2), human translator vs. systems excluding top ones, despite higher percentages, GEMBA-MQM still lags behind human evaluators by at least 39 points. GEMBA-MQM (Literary) slightly outperforms the original version by an average of 4.2 points and 2.5 points for scenarios (1) and (2), indicating that literary-specific knowledge helps but remains insufficient for discriminating human and LLM outputs. As shown in Table 14 (appendix), other metrics also exhibit substantial gaps compared to human performance. XCOMET-XL scores 26.9 points lower and XCOMET-XXL 21.2 points lower than the worst human performance in scenario (1). In scenario (2), the gaps are even larger with 39.6 points for XCOMET-XL and 30.2 for XCOMET-XXL. While XCOMET-XL and XCOMET-XXL yield better results than GEMBA-MQM on adequacy, with average performances of 15.6% and 21.3% in scenario (1) respectively, they correlate worse with human MQM and SQM than GEMBA-MQM. This is particularly evident for _De-En_ and _De-Zh_, as shown in Figure 2 (a) and Figure 9 (appendix).

Additionally, our analysis reveals that GEMBA-MQM's performance is primarily driven by Accuracy aspects, with less reliability in evaluating Fluency, Style, and Terminology (see Figure 2 (b)).17 Accuracy shows the strongest correlations across all language pairs. Accuracy for _De-En_ and _De-Zh_ even surpasses the full correlation. Meanwhile, Fluency and Style

Figure 2: Segment level correlation measured by Kendall s \(\tau\) between (a) human MQM and automatic metrics and (b) human MQM and GEMBA-MQM (Literary) per error categories. See Figure 9 (appendix) for SQM results.

 

demonstrate poor to fair correlations. Notably, Terminology consistently shows near-zero correlations, which could be problematic for distant language pairs with more culture-specific terms, e.g., _En-Zh_ with Terminology as the second most frequent error category. These findings suggest that future metric development should focus on improving the assessment of Fluency, Style, and Terminology.

Human MQM vs. GEMBA-MQM: LLMs demonstrate a tendency towards more literal and less diverse translations.We further compare human MQM with GEMBA-MQM in system rankings considering characteristics of the translation systems. To assess the literalness of translation, we analyze the syntactic similarity between source-translation pairs. We utilize the recent _document-level_ tree kernel-based syntactic similarity metric FastKASSIM Chen et al. (2023). While high syntactic similarity preserves the source's form, it frequently sacrifices naturalness in the target language and hinders creativity in translations Guerberof-Arenas and Toral (2022); Al-Awawdeh (2021). Diversity is another concern in literary domain Chen et al. (2024); Zhang and Eger (2024). To measure lexical diversity of one system compared to all other systems, we calculate pairwise lexical overlap between pairs of translations from different systems and average them per system.18 A lower value suggests a more diverse vocabulary selection. Figure 3 shows results based on **all publications**, illustrating human MQM scores for 10 translation systems averaged over four language pairs (z-axis) against syntactic similarity (x-axis) and lexical overlap (y-axis) (see Figure 10 in the appendix for results on **contemporary works**). The rank of systems is marked as [rank per human MQM, rank per GEMBA-MQM ]. We show detailed examples in Table 15 and Table 16 (appendix).

Footnote 18: See Section F.6 (appendix) for more details.

Three major clusters emerge: (1) Human Translator stands alone, characterized by a high MQM score with the lowest syntactic similarity (0.21) and lexical overlap (18.9). While human translation ranks first per human MQM by a 0.4 margin over GPT-4o, GEMBA-MQM (Literary) prefers GPT-4o with a larger margin of 1.4 over human translators. Surprisingly, GEMBA-MQM (Literary) even rates Google Translate higher than human translations. (2) GPT-4o, DeepL, and Google Translate all exhibit high MQM scores and high lexical overlap. The three models share high lexical overlap, contributing to their remarkably high average scores. DeepL has the highest syntactic similarity of 0.27. (3) The third cluster demonstrates a performance gap to other clusters, with Qwen 2 and TowerInstruct performing slightly better. Qwen 2 paraphrases more for _De-Zn_ and _En-Zh_ pairs and sometimes includes Chinese characters in _De-En_ and _En-De_ outputs, which results in low syntactic similarity and lexical overlap but still higher than human.

Summary:GEMBA-MQM outperforms other current automatic metrics but struggles with non-Accuracy aspects. All metrics lack adequacy in distinguishing human and LLM translations. LLMs tend to produce more literal and less diverse translations showing higher syntactic similarity to the source and greater lexical overlap with other systems than human translations. Further, LLM as evaluator prefers more literal translations and is biased towards their outputs.

## 6 Conclusion

This paper introduces LitEval-Corpus and a comprehensive study on literary translation evaluation with human and automatic metrics.

Our study reveals the limitations of error annotation, particularly MQM when comparing top-performing systems. This may be attributed to three factors: (1) the complexity of MQM com

Figure 3: Distribution of human MQM (z-axis) against syntactic similarity between source-translation pairs and lexical overlap of the current system to all other systems. The system rankings are shown as [rank per human MQM, rank per GEMBA-MQM (Literary)]. See Figure 11 for a detailed heatmap.

 

pared to other schemes; (2) the current MQM framework failing to account for "errors" in literary translation intended for certain equivalence in the target language; (3) current leading systems producing fewer errors than previous MT systems, rendering MQM insufficient. While using BWS or experienced professionals yields more plausible assessments, BWS has limitations for further improvements like post-editing when compared to MQM. These results have substantial implications, particularly for determining human parity--a crucial issue in today's LLM landscape. They are especially relevant for professional translators, as naive translation companies might eliminate human translators due to misjudgments based on MQM or SQM evaluations by inexperienced practitioners. While we highlight MQM's limitations in literary MT evaluation, our current study does not propose specific improvements to the existing MQM guidelines. Future work can explore these aspects in depth.

Furthermore, our results indicate that automatic metrics development should focus on aspects such as Style and Terminology. LLMs still need to address the issue of overly literal outputs, and a substantial gap remains between LLM and human quality in literary translation, despite the clear advancements of recent models.

### Limitations

Several limitations of our current work warrant consideration:

Our study primarily focuses on prompting-based techniques with identical basic prompts across LLMs. While our initial exploration with GPT-4o showed no substantial improvement with complex prompts, we acknowledge that LLM-specific prompt optimization could potentially yield better results in literary translation. This aligns with the current understanding that prompting is a delicate process Leiter and Eger (2024); Mizrahi et al. (2024).

Other techniques such as fine-tuning with domain knowledge may further enhance LLM performance in this context. Conversely, data contamination may inadvertently improve translation quality for classic works already in the model's training data. However, classics unseen by the model or yet to be translated might exhibit lower translation quality.

Our focus on paragraph-level translation still does not fully capture the nuances of literary texts. The absence of evaluation on (consecutive) chapters limits our ability to comprehensively evaluate translation quality within a broader context. Although we conducted limited experiments with multi-paragraph samples, resource constraints and the complexity of the task on student evaluators prevented a more extensive exploration in this direction.

Our study covers four high-resource language pairs, each evaluated by a single evaluator except when computing agreements. We acknowledge that involving multiple evaluators with greater expertise could yield more robust results. To mitigate this limitation within our resource constraints, we implemented a rigorous selection process and provided comprehensive training for our evaluators. However, our dataset doesn't explore LLM performance in low-resource languages--an area that could potentially benefit the most from literary MT.

While we highlight the limitations of MQM in literary MT evaluation, we do not propose specific improvements to the current MQM guidelines. Professional translators agree that MQM is insufficient for literary translation but recognize that enhancing it requires extensive research and exploration. Future work should focus on improving upon MQM to better reflect aspects such as aesthetics, emotion, stylistic features, and the tone of both authors and characters.

## Ethical Considerations

LLMs in literary translation have sparked widespread public interest and concern. This topic has become a focal point for researchers, translators, and publishers, generating intense debate at various forums including workshops, interviews, and book fairs.19 The increased engagement underscores the need for responsible handling of MT outputs in general.

Footnote 19: We list some events below. Workshops: [https://ctt2824.ccl.kuleven.be/andhttps://bothranisations.com/blog/dual-t-end-of-project-event-recap](https://ctt2824.ccl.kuleven.be/andhttps://bothranisations.com/blog/dual-t-end-of-project-event-recap). Interviews & discussions: [https://www.goethe.de/ins/gb/en/kufl/lue/ail.html](https://www.goethe.de/ins/gb/en/kufl/lue/ail.html), [https://www.buthemsese.de/pressee/pressmetitleungen/2023-10-6eV-keunstliche-intelligenz](https://www.buthemsese.de/pressee/pressmetitleungen/2023-10-6eV-keunstliche-intelligenz), and [https://www.wifung-genshagen.de/veranaltaltungen/](https://www.wifung-genshagen.de/veranaltaltungen/) veransaltungen-2024/kku/default-a39e201a92/.

 

their awareness. LLMs may introduce or exacerbate issues of bias and toxicity in translated literature, impacting the audience, particularly children. The growing capabilities of LLMs in translation also threaten professional translators, raising concerns about job displacement and the devaluation of human expertise due to misconceptions about LLM quality. However, the careful use of LLMs or MT models as an auxiliary tool could democratize access to global literary works by improving the accessibility of literature across languages and cultures, especially beneficial for low-resource languages. LLMs could also empower translators, allowing them to focus on the challenging aspects of translation while leaving routine tasks to MT. Additionally, LLMs could serve as potential brainstorm tools, e.g., inspiring more creative wordplay alternatives.

Regarding dataset usage: We adhere to the guidelines specified in Germany's Copyright in Academic Work regulations.20 The dataset usage in this study has been approved by the university ethics committee.

Footnote 20: [https://www.bmpf.de/SharedDocs/Publikationen/DE/FS/31580_Urheberrecht_in_der_Wissenschaft_en.pdf](https://www.bmpf.de/SharedDocs/Publikationen/DE/FS/31580_Urheberrecht_in_der_Wissenschaft_en.pdf)

## Acknowledgements

We thank the anonymous reviewers for their thoughtful feedback that greatly improved the text. We also thank all evaluators for their dedication and appreciate the professional input from all translators involved. The NLLG Lab gratefully acknowledges support from the Federal Ministry of Education and Research (BMBF) via the research grant "Metrics4NLG" and the German Research Foundation (DFG) via the Heisenberg Grant EG 375/5-1.

## References

* A. Al-Awawadeh (2021)Translation between creativity and reproducing an equivalent original text. Psychology and Education Journal58 (1), pp. 2559-2564. External Links: ISSN 0003-69X Cited by: SS1.
* D. M. Alves, J. Pombal, N. M. Guerreiro, P. Martins, I. Alves, C. Farajian, B. Peters, R. Rei, P. Fernandes, S. Agrawal, P. Colombo, J. G. C. de Souza, and A. Martins (2024)Tower: An open multilingual large language model for translation-related tasks. External Links: Link Cited by: SS1.
* J. Belouadi and T. Eger (2023)ByGPT5: end-to-end style-conditioned poetry generation with token-free language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 7364-7381. External Links: Document, Link Cited by: SS1.
* A. Belz, C. Thomson, E. Reiter, G. Abercrombie, J. M. Alonso-Moral, M. Arvan, A. Braggaar, K. Cieliebak, E. Clark, K. van Deemter, T. Unkar, O. Deske, S. Fager, Q. Fang, M. Gao, A. Gatt, D. Gkatzia, J. Gonzalez-Corbelle, D. Hovy, M. Hurlimann, T. Ito, J. D. Kelleher, F. Klubicka, E. Krahmer, H. Lai, C. van der Lee, Y. Li, S. Mahamood, M. Mieskes, E. van Miltenburg, P. Mosteiro, M. Nissim, N. Parde, O. Platek, V. Rieser, J. Ruan, J. Tetreault, A. Torral, X. Wan, L. Wantson, and D. Yang (2023)Missing information, unresponsive authors, experimental flaws: The impossibility of assessing the reproducibility of previous human evaluations in NLP. In Proceedings of the Fourth Workshop on Insights from Negative Results in NLP, Dubrovnik, Croatia, pp. 1-10. External Links: ISBN 978-3-31-4653-3, Link Cited by: SS1.
* F. Blain, C. Zerva, R. Rei, N. M. Guerreiro, D. Kanoja, J. G. C. de Souza, B. Silva, T. Vaz, Y. Jingxuan, F. Azadi, C. Orasan, and A. Martins (2023)Findings of the wmt 2023 shared task on quality estimation. In Proceedings of the Eighth Conference on Machine Translation, Singapore, Singapore, pp. 629-653. External Links: Document, Link Cited by: SS1.
* M. Chen, C. Chen, X. Yu, and Z. Yu (2023)FastKASSIM: a fast tree kernel-based syntactic similarity metric. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, pp. 211-231. External Links: Link Cited by: SS1.
* Y. Chen and S. Eger (2023)Menli: Robust evaluation metrics from natural language inference. Transactions of the Association for Computational Linguistics11, pp. 804-825. External Links: Document, Link Cited by: SS1.
* Y. Chen, H. Groner, S. Zarriess, and S. Eger (2024)Evaluating diversity in automatic poetry generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, pp. 19671-19692. External Links: Link Cited by: SS1.
* M. R. Costa-jussa, J. Cross, O. Celebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, et al. (2022)No language left behind: scaling 

human-centered machine translation. _arXiv preprint arXiv:2207.04672_.
* Daems et al. (2024) Joke Daems, Paola Ruffo, and Lieve Macken. 2024. Impact of translation workflows with and without MT on textual characteristics in literary translation. In _Proceedings of the 1st Workshop on Creative-text Translation and Technology_, pages 57-64, Sheffield, United Kingdom. European Association for Machine Translation.
* Deutsch et al. (2023) Daniel Deutsch, Juraj Juraska, Mara Finkelstein, and Markus Freitag. 2023. Training and meta-evaluating machine translation evaluation metrics at the paragraph level. In _Proceedings of the Eighth Conference on Machine Translation_, pages 996-1013, Singapore. Association for Computational Linguistics.
* Fan et al. (2021) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. _J. Mach. Learn. Res._, 22(1).
* Freitag et al. (2021) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. _Transactions of the Association for Computational Linguistics_, 9:1460-1474.
* Graham et al. (2013) Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_, pages 33-41, Sofia, Bulgaria. Association for Computational Linguistics.
* Guerberof-Arenas and Toral (2022) Ana Guerberof-Arenas and Antonio Toral. 2022. Creativity in translation: Machine translation as a constraint for literary texts. _Translation Spaces_, 11(2):184-212.
* Guerreiro et al. (2024) Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andre F. T. Martins. 2024. xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection. _Transactions of the Association for Computational Linguistics_, 12:979-995.
* Hansen and Esperanca-Rodier (2022) Damien Hansen and Emmanuelle Esperanca-Rodier. 2022. Human-adapted mt for literary texts: Reality or fantasy? In _NeTTT 2022_, pages 178-190.
* Jiang et al. (2022) Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Minmaya Sachan, and Ming Zhou. 2022. BlonDe: An automatic evaluation metric for document-level machine translation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1550-1565, Seattle, United States. Association for Computational Linguistics.
* Karpinska and Ilyyer (2023) Marzena Karpinska and Mohit Ilyer. 2023. Large language models effectively leverage document-level context for literary translation, but critical errors persist. In _Proceedings of the Eighth Conference on Machine Translation_, pages 419-451.
* Kim et al. (2024) Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 4334-4353, Miami, Florida, USA. Association for Computational Linguistics.
* Kiritchenko and Mohammad (2017) Svetlana Kiritchenko and Saif Mohammad. 2017. Bestworst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 465-470, Vancouver, Canada. Association for Computational Linguistics.
* Kocmi and Federmann (2023) Tom Kocmi and Christian Federmann. 2023. GEMBAMQM: Detecting translation quality error spans with GPT-4. In _Proceedings of the Eighth Conference on Machine Translation_, pages 768-775, Singapore. Association for Computational Linguistics.
* Kocmi et al. (2024) Tom Kocmi, Vilem Zouhar, Elefterios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popovic, Mrinmaya Sachan, and Mariya Shmatova. 2024. Error span annotation: A balanced approach for human evaluation of machine translation. In _Proceedings of the Ninth Conference on Machine Translation_, pages 1440-1453, Miami, Florida, USA. Association for Computational Linguistics.
* Kolb (2023) Waltraud Kolb. 2023. i am a bit surprised: Literary translation and post-editing processes compared. In _Computer-Assisted Literary Translation_, pages 53-68. Routledge.
* Kolktevit-Intelligenz (2024) Koltektive-Intelligenz. 2024. Ki aber wie? Ubersetzertrag 2024.
* Leiter and Eger (2024) Christoph Leiter and Steffen Eger. 2024. Prexme! large scale prompt exploration of open source lllms for machine translation and summarization evaluation. _arXiv preprint arXiv:2406.18528_.
* Leiter et al. (2023) Christoph Leiter, Juri Opitz, Daniel Deutsch, Yang Gao, Rotem Dror, and Steffen Eger. 2023. The Eval4NLP 2023 shared task on prompting large language models as explainable metrics. In _Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems_, pages 117-138, Bali, Indonesia. Association for Computational Linguistics 

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. _Transactions of the Association for Computational Linguistics_, 8:726-742.
* Liu et al. (2024) Yiqi Liu, Nafise Moosavi, and Chenghua Lin. 2024. LLMs as narcissistic evaluators: When ego inflates evaluation scores. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 12688-12701, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Lommel et al. (2014) Arle Lommel, Aljoscha Burchardt, Maja Popovic, Kim Harris, Eleftherios Avramidis, and Hans Uszkoreit. 2014. Using a new analytic measure for the annotation and analysis of mt errors on real data. In _Proceedings of the 17th Annual conference of the European Association for Machine Translation_, pages 165-172.
* Lu et al. (2025) Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, and Dacheng Tao. 2025. MQM-APE: Toward high-quality error annotation predictors with automatic post-editing in LLM translation evaluators. pages 5570-5587.
* Macken (2024) Lieve Macken. 2024. Machine translation meets large language models: Evaluating ChatGPT's ability to automatically post-edit literary texts. In _Proceedings of the 1st Workshop on Creative-text Translation and Technology_, pages 65-81, Sheffield, United Kingdom. European Association for Machine Translation.
* Martin (2023) Ruth Martin. 2023. Reflections from an ai translation slam: (wo)man versus machine.
* Matusov (2019) Evgeny Matusov. 2019. The challenges of using neural machine translation for literature. In _Proceedings of the Qualities of Literary Machine Translation_, pages 10-19, Dublin, Ireland. European Association for Machine Translation.
* McHugh (2012) Mary L McHugh. 2012. Interrater reliability: the kappa statistic. _Biochemia medica_, 22(3):276-282.
* Mizrahi et al. (2024) Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. _Transactions of the Association for Computational Linguistics_, 12:933-949.
* Pfundmeier (2023) Monika Pfundmeier. Redaktion: Dorrit Bartel Nina George, Andre Hansen. 2023. Anwendungen von fortgeschrittener informatik und generativer systeme im buchsektor.
* Nizio (2024) Magdalena Nizio. 2024. Kunstliche intelligen z gefahr oder hilfe fur literarische Ubersetzung?
* Pang et al. (2024) Jianhui Pang, Fanghua Ye, Longyue Wang, Dian Yu, Derek F Wong, Shuming Shi, and Zhaopeng Tu. 2024. Salute the classic: Revisiting challenges of machine translation in the age of large language models. _arXiv preprint arXiv:2401.08350_.
* Patro (2015) S Patro. 2015. Normalization: A preprocessing stage. _arXiv preprint arXiv:1503.06462_.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
* Puka (2011) Llukan Puka. 2011. _Kendall's Tau_, pages 713-715. Springer Berlin Heidelberg, Berlin, Heidelberg.
* Resende and Hadley (2024) Natalia Resende and James Hadley. 2024. The translator's canvas: Using LLMs to enhance poetry translation. In _Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_, pages 178-189, Chicago, USA. Association for Machine Translation in the Americas.
* Rutherford et al. (2024) Markella B Rutherford, Peggy Levitt, and Erika Zhang. 2024. Where the 3 percent? How far have we come toward decentering americas literary preference? _Global Perspectives_, 5(1):93034.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Stroube (2003) Bryan Stroube. 2003. Literary freedom: Project gutenberg. _XRDS: Crossroads, The ACM Magazine for Students_, 10(1):3-3.
* Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. _arXiv preprint arXiv:2312.11805_.
* Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Suya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. _arXiv preprint arXiv:2403.08295_.
* Thai et al. (2022) Katherine Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray, Moira Inghilleri, John Wieting, and Mohit Iyyer. 2022. Exploring document-level literary machine translation with parallel paragraphs from world literature. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9882-9902.
* Thomson et al. (2024) Craig Thomson, Ehud Reiter, and Anya Belz. 2024. Common flaws in running human evaluation experiments in NLP. _Computational Linguistics_, 50(2):795-805.
* Tkachenko et al. (2022) Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2022. Label Studio: Data labeling software. Open source software available from [https://github.com/heartexlabs/label-studio](https://github.com/heartexlabs/label-studio) 

Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 113-123, Brussels, Belgium. Association for Computational Linguistics.
* Van Egdom et al. (2023) Gys-Walt van Egdom, Onno Kosters, and Christophe Declercq. 2023. The riddle of (literary) machine translation quality. _Tradumatica technologies de la traduccio_, (21):129-159.
* Vanroy et al. (2024) Bram Vanroy, Marie-Aude Lefer, Lieve Macken, and Paola Ruffo, editors. 2024. _Proceedings of the 1st Workshop on Creative-text Translation and Technology_. European Association for Machine Translation, Sheffield, United Kingdom.
* Voigt and Jurafsky (2012) Rob Voigt and Dan Jurafsky. 2012. Towards a literary machine translation: The role of referential cohesion. In _Proceedings of the NAACL-HLT 2012 Workshop on Computational Linguistics for Literature_, pages 18-25, Montreal, Canada. Association for Computational Linguistics.
* Wang et al. (2023) Longyue Wang, Zhaopeng Tu, Yan Gu, Siyou Liu, Dian Yu, Qingsong Ma, Chenyang Lyu, Litting Zhou, Chao-Hong Liu, Yufeng Ma, Weiyu Chen, Yvette Graham, Bonnie Webber, Philipp Koehn, Andy Way, Yulin Yuan, and Shuming Shi. 2023. Findings of the WMT 2023 shared task on discourse-level literary translation: A fresh or nb in the cosmos of LLMs. In _Proceedings of the Eighth Conference on Machine Translation_, pages 55-67, Singapore. Association for Computational Linguistics.
* Wu et al. (2024) Minghao Wu, Jiahao Xu, Yulin Yuan, Gholamreza Haffari, and Longyue Wang. 2024. (perhaps) beyond human translation: Harnessing multi-agent collaboration for translating ultra-long literary texts. _arXiv preprint arXiv:2405.11804_.
* Xu et al. (2022) Mingzhou Xu, Longyue Wang, Derek F. Wong, Hongye Liu, Linfeng Song, Lidia S. Chao, Shuming Shi, and Zhaopeng Tu. 2022. GuoFeng: A benchmark for zero pronoun recovery and translation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 11266-11278, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Yan et al. (2024) Jianhao Yan, Pingchuan Yan, Yulong Chen, Judy Li, Xianchao Zhu, and Yue Zhang. 2024. GPT-4 vs. human translators: A comprehensive evaluation of translation quality across languages, domains, and expertise levels. _arXiv preprint arXiv:2407.03658_.
* Yang et al. (2024) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. _arXiv preprint arXiv:2407.10671_.
* Zhang and Eger (2024) Ran Zhang and Steffen Eger. 2024. Llm-based multi-agent poetry generation in non-cooperative environments. _arXiv preprint arXiv:2409.03659_.
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.
* Zhao et al. (2023a) Wei Zhao, Michael Strube, and Steffen Eger. 2023a. Discsocore: Evaluating text generation with bert and discourse coherence. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 3865-3883.
* Zhao et al. (2023b) Zoie Zhao, Sophie Song, Bridget Duah, Jamie Macbreth, Scott Carter, Monica P Van, Nayeli Suseth Bravo, Matthew Klenk, Kate Sick, and Alexandre LS Filipowicz. 2023b. More human than human: Llm-generated narratives outperform human-llm interleaved narratives. In _Proceedings of the 15th Conference on Creativity and Cognition_, pages 368-370.

## Appendix A Corpora comparison

Table 4 summarizes the corpora for literary MT (evaluation). We report the annotated dataset and the entire MT parallel corpus separately



* [14] M. C. Bader, J. A. Kwiecinski, and R. D. Sorensen, "Theoretical evaluation of the performance of a new method for the determination of the number of parameters in a given set of experiments," _Journal of Computational Physics_**2013**(5):1-6, 2009.

[MISSING_PAGE_POST]

 

Corpus details

Table 5 summarizes the details of the publications and the corresponding translation versions.

## Appendix C Prompt experiments for literary translation

We use GPT-4o to experiment with four types of prompts. The results are shown in Table 6.

## Appendix D Human evaluation

### Summary of human evaluation schemes

Figure 4 shows human evaluation schemes with an example.

### System ranking

Table 7 shows the system ranking per evaluator and evaluation schemes.

### Error distribution

Figure 5 shows the distribution of mean error count per segment (i.e., the total error count divided by the number of segments) aggregated over error categories.

### Adequacy plot for scheme combination

Figure 6 shows the percentage change in distinguishing human translation from top system outputs by combining student SQM and MQM results. We use min-max scaling Patro [2015] to adjust MQM to the same range as SQM (0-6). The combined score is calculated as \((1-\alpha)\times S_{\text{MQM}}+\alpha\times S_{\text{SQM}}\), where S represents the score and \(\alpha\) represents the ratio between SQM and MQM.

## Appendix E Error annotation details

### Total cost

The four student evaluators each work 60 hours at a rate of 14 per hour. They are given two months to complete the task. Prior to annotation, the authors and students collaboratively refine the guidelines through several iterations to address any unclear points. Additionally, two professional translators work a total of 20 hours at a rate of approximately 50 per hour. In total, this comprehensive evaluation project amounts to nearly 4,500.

### MQM Annotation guidelines

We present our annotation guideline in this section and definition of translation errors in Table 8.

#### e.2.1 Annotation guideline

##### Introduction

For each annotation, you will be presented with two paragraphs: the source paragraph and the translation of the source paragraph. You need to read both paragraphs carefully and finish the following tasks.

_Task description_

**Task 1:****MQM error annotation**

Step 1: You will be given the definition of translation errors. You need to highlight the spans from the translation containing errors and choose the corresponding error categories.

Please consider following the order of questions during error annotation:

* Evaluate whether the sentence is comprehensible. If it is completely incomprehensible, mark it as "non-translation."
* Are there any untranslated texts?
* Is there any cultural-specific terminology mistranslation or inconsistency?
* Any accuracy errors with addition/omission/mismomer or mistranslation especially due to overly literal translation or temporal component?
* Are there any style errors?
* Are there any fluency errors that cause inconsistency in non-terminology usage and coherence errors?

Step 2: Rate the severity of errors Major Errors are errors that seriously distort the meaning of the source, in such a way that it becomes completely incomprehensible or that the essence of the message is lost. Another example is errors that may confuse or mislead the reader due to substantial changes in meaning.

Minor Errors dont lead to loss of meaning and wouldnnt confuse or mislead the reader but would be noticed, would decrease stylistic quality, fluency, or clarity, or make the content less appealing.

You only need to annotate once if the error is repeated several times.

**Task 2:****SQM assessment**

You need to rate the overall translation quality. How do you like the translation, especially in reflecting the stylistic, artistic, and emotional qualities of the original work? Give a score from 0-6 

* [10] M. A. B. Krizhevsky, S. G. Hinton, and R. H. Friedman, "Learning to predict the best model for a given dataset," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 257-268, 2009.
* [11] J. L. Friedman, "The generalization of deep learning with applications to image recognition," _IEEE Transactions on Image Processing_**34**, no. 1, 2015, pp. 1-12.
* [12] D. F. Kingma and P. Ba, "Adam: A method for stochastic optimization," in _Advances in neural information processing systems_, vol. 34, no. 2, p. 1353-1360, 2014.
* [13] T. Bengio, Y. Bengio, and C. Veness, "Classification and classification by jointly connected networks," _arXiv preprint arXiv:1412.6120_**, 2014.
* [14] I. Goodfellow, I. Guyon, and E. Rabinovich, "Imagenet classification with deep convolutional nets," in _NIPS conference on computer vision and pattern recognition_, volume 3, pages 1378-1389, 2014.
* [15] X. Jia, Z. Liu, W. Dong, and Q. Weinberger, "Deep residual learning for large scale visual data," _CoRR_**140**, 2016.

[MISSING_PAGE_POST]

 

* [10] M. C. Bresson, A. J. R. Krizhevsky, and D. S. Murray, "A review of the statistical learning approach to image recognition," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 235-246, 2009.
* [11] P. L. Barnett, G. E. Hinton, and I. V. Mnih, "Deep residual networks for large scale machine translation," _IEEE Transactions on neural information processing systems_, vol. 87, no. 1, pp. 1267-1270, 2009.
* [12] T. Bengio, Y. Bengio, and Z. Chen, "Generative adversarial nets: a new paradigm for stochastic optimization," _arXiv preprint arXiv:1412.6110_, 2014.
* [13] H. Ba, X. Li, and W. Teh, "Learning deep convolutional models with generative adversarial nets," _Journal of Machine Learning Research_, vol. 10, no. 1, pp. 1-10, 2015.
* [14] N. Bahdan, F. Bachrach, and A. Farhadi, "Improving deep convolutional neural networks by reducing the dimensionality of data," _arXiv preprint arXiv:1412.6110_, 2014.
* [15] D. Battou, A. Graves, O. Courville, and A. Rabinovich, "Very simple networks for tiny images," _Neural computation_, vol. 1, no. 1, p. 1090, Springer, 2014.

[MISSING_PAGE_POST]

 



#### e.2.2 Error definition

The definition of translation errors is shown in Table 8. We derive our category definitions primarily from the MQM official website21 and the study by Wang et al. (2023). We add two additional categories: **Non-translation** defined by Freitag et al. (2021) and **Temporal effect** defined by ourselves.

Footnote 21: [https://themqm.org/error-types-2/typology/](https://themqm.org/error-types-2/typology/)

#### e.2.3 Screenshot of error annotation platform

Figure 7 demonstrates the screenshot of the error annotation platform.

### Examples of human evaluation results

#### e.3.1 _De-En_

Figure 8 shows examples of SQM and error annotations by student and professional evaluators.

### Example of human translation labelled as "error" according to student MQM

Table 9 shows examples of MQM errors labeled by student evaluators due to cultural adaptation in the translation.

## Appendix F Automatic metrics

### Detail of automatic metrics

Table 10 summarizes the characteristics of automatic metrics. For cost reasons, we use GPT-4o-mini instead of GPT-4 used in the original paper. This may induce a slight performance reduction. But compared to the cost for GPT-4 (on average 0.12 per segment, approximately 250 for complete evaluation per prompt), GPT-4o-mini is over 200 times cheaper (1.2 for full evaluation per prompt).

### GEMBA-MQM detail

#### f.2.1 GEMBA-MQM prompt templates

Table 11 showcases the prompt templates.

#### f.2.2 Consistency analysis of GEMBA-MQM (Literary) under different temperature

Table 12 shows the consistency of GEMBACMQM (Literary) under different temperature. We evaluate the consistency of GEMBA-MQM (Literary) on the full _De-En_ dataset under varying temperatures, each temperature querying three times. From Table 12, we observe that even at temperature 0 (the deterministic setting), the resulting score shows slight variation while maintaining high overall consistency. The correlation coefficient between different queries decreases from 0.910 at temperature 0.0 to 0.856 at temperature 1.0. Similarly, the proportion of cases with small score differences (\(\delta\leq 1\)) drops from 78.8% to 73.3%, while larger differences (\(\delta>5\)) increase from 4.3% to 7.5% when temperature increases. These results demonstrate that the scores remain robust and consistent at temperature 0 and across low temperatures but variability increases with higher temperatures.

\begin{table}
\begin{tabular}{l c c c l} \hline \hline  & **aspect** & **intended** & **base model** & **fparameter** \\ \hline
**Prometheb 2** & & & pronethers2 & 7 billion \\
**XOMET-XL** & X & XLMR-XL & 3.5 billion \\
**SCOMER-XXI** & x & XLMR-XL & 10 billion \\
**GEMBA-MOM** & x & GPT-4o mini & ? \\ \hline \hline \end{tabular}
\end{table}
Table 10: Summary of characteristics for automatic metrics. _Aspect_ indicates whether the metric support evaluation per aspect such as error categories and #parameter indicates the size of the base model.



* [10] M. C. Dickson, A. K. Sohn, and R. J. Schuster, "A simple model for the social network of social networks: a review," in _Proceedings of the 27th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 35-46, 2009.
* [11] G. B. K. Sohn, and R. J. Schuster, "Social networks with applications to social networks," _The Journal of Social Sciences_, vol. 83, no. 1, pp. 103-114, 2009.
* [12] H. W. Lee, T. L. Kim, and Y. Zhang, "Social networks: The role of social networks," _Journal of Social Psychology_, vol. 10, no. 1, pp. 1-12, 2009.

[MISSING_PAGE_POST]

 


Figure 8: Annotation example for _De-En_

Figure 7: Screenshot of error annotation interface





### _Correlation between SQM and automatic metrics_

Figure 9 illustrates segment-level correlation between human SQM and automatic metrics.

### _Human vs. LLMs for metrics._

Table 14 shows the percentage of segments where human translations are preferred over machine translations per evaluation schemes and automatic metrics.

\begin{table}
\begin{tabular}{c c l} \hline \hline  & \multicolumn{2}{l}{**Score difference \(\delta\)**} \\ \cline{3-4}  & & \(\delta\leq 1\) & \(1<\delta\leq 5\) & \(\delta>5\) \\ \hline
0.0 & 0.910 & 78.8\% & 16.9\% & 4.3\% \\
0.1 & 0.902 & 76.9\% & 16.9\% & 6.2\% \\
0.3 & 0.882 & 74.7\% & 18.7\% & 6.6\% \\
0.5 & 0.887 & 71.9\% & 22.6\% & 5.5\% \\
0.7 & 0.874 & 72.2\% & 20.5\% & 7.3\% \\
1.0 & 0.856 & 73.3\% & 19.2\% & 7.5\% \\ \hline \hline \end{tabular}
\end{table}
Table 12: Consistency of GEMBA-MQM scores across different temperatures. We query three times at each temperature. Corr calculates the mean rank correlation between queries using Spearman correlation. The score difference shows the percentage of score changes under different thresholds. \(\delta=1\) represents one minor error in difference, while \(\delta=5\) represents one major error.

\begin{table}
\begin{tabular}{p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline pair & Source & Human Translation & Interpretation & \\ \hline _En-Zh_ & ... a subtermes of enrolling and shareings & \(-0\ldots\) & The text in brackets is marked as a mismanor (a subcategory of accuracy) over by the student evaluator. While the source text mentions griffins, the transaction simply uses “monster,” since in the target culture, griffins as mythical eratuures have no cultural significance. & \multirow{2}{*}{The text in brackets is marked as a misclassification (a subcategory) over decay) over by the student evaluator. Though the source text mentions’ Latin school,” the translation uses ”peristigious school” because Latin schools don’t exist in the target culture and therefore lack the same comnotation of academic pressure. & \multirow{2}{*}{
\begin{tabular}{} \end{tabular} } & The text in brackets is triversality translation based on the source texts?" \\ \cline{3-4}  & ... & & & \\ \cline{3-4}  & .



* [25] M. C. B. K., and R. J. S. G. Wilson, "A new approach to the design of a new class of neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-9.
* [26] A. D. Lerman, "The design of neural networks for learning from scratch," in _Proceedings of the IEEE International Conference on Machine Learning Systems (ICMLSP'08)_, pp. 347-358.
* [27] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with applications to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-10.
* [28] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [29] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [30] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [31] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [32] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [33] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [34] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [35] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [36] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [37] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [38] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [39] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [40] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [41] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [42] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [43] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [44] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [45] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [46] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [47] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [48] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [49] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [50] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [51] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [52] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [53] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [54] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [55] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [56] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [57] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [58] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [59] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [60] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [61] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [62] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [63] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [64] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [65] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [66] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [67] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [68] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [69] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [70] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [71] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [72] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [73] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [74] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [75] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [76] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [77] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08)_, pp. 1-1.
* [78] P. W. H. Lee, "Learning the structure of deep convolutional neural networks with application to machine translation," in _Proceedings of the IEEE Conference on Computer Vision and 

* [14] M. C. Bader, J. A. Harvey, and R. H. Soder, "Theoretical Population of the Universe: The Role of Life in the Early Universe," _Astrophys. J. Suppl._**207**(3), 59-68, 1994.
* [15] D. E. Kaplan, "A New Approach to the Evolution of the Universe," _Phys. Rev. Lett._**19**(2), 1018-1020, 1978.
* [16] F. L. Kane, "The Evolution of the Universe," _Ann. Phys._**19**(2), 103-118, 1980.

[MISSING_PAGE_POST]

 

Figure 1 is based on Table 3 and Table 14 (appendix). For the percentage computation of scheme professional error annotation, we first convert error span annotation counts to numerical scores using the formula Score \(=|\text{Good solution}|-|\text{Error}|\), then rank these scores to compute the percentage. The percentage of human > top systems averaged over three language pairs is 62.8%, with _De-En_ at 75.0%, _En-De_ at 33.3%, and _En-Zh_ 80.0%, respectively.

The professional BWS statistics shown in Figure 1 are based on the results from _En-Zh_, with 14 samples evaluated by the _En-Zh_ translator.

### Computation details for diversity measure

To assess a system's lexical diversity compared to others, we calculate the pairwise lexical overlap between translations from different systems. For each paragraph, we compute the lexical overlap between the current system's translation and those of all other systems. This overlap quantifies the vocabulary shared between the current system and others. We then average these overlaps for all paragraphs, resulting in an overall lexical overlap score for the system. A lower average overlap indicates that the system uses a more diverse and distinct vocabulary, while a higher score suggests greater similarity in word choice with other systems.

The mean lexical overlap for system \(j\) across all paragraphs is given by:

\[\text{Avg.overlap}_{j}=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{M-1}\underset {k\neq j}{\overset{l}{\sum}}S(T_{ij},T_{ik})\right)\]

where \(N\) is the total number of paragraphs. \(M\) is the total number of systems. \(T_{ij}\) is the translation of paragraph \(i\) by system \(j\). \(S(T_{ij},T_{ik})\) is the function, i.e., BLEU by SACREBLEU Post (2018), to compute lexical overlap between the translation from system \(j\) and system \(k\) for paragraph \(i\).

### Comparison of classic and contemporary works

Figure 10 shows the scatter plots of human MQM (z-axis) per average syntactic similarity between source and system translation and average lexical overlap of the current system to all other systems for all contemporary works after 2000. Human translation still holds the lowest lexical overlap (19.6) than other systems except for Qwen 2 (19.5). The syntactic similarity of human translation (0.267) ranks third slightly higher than Qwen2 (0.252) and NLLB (0.253).

Figure 11 shows the heatmap of pairwise lexical overlap between systems based on the entire LiteEval-Corpus. The results reveal exceptionally high lexical overlap among Google Translate, GPT-4o, and DeepL, while NLLB and M2M (all from META AI) also show notably high overlap. This pattern may stem from shared training corpora or training data distillation from other models.

## Appendix F

Figure 11: Heatmap for pairwise lexical overlap between translation systems (calculation based on the entire LiteEval-Corpus).

Figure 10: (Contemporary works after 2000) Scatter plots of human MQM (z-axis) per average syntactic similarity between source and system translation and average lexical overlap of the current system to all other systems. The rank of systems is marked as [rank per human MQM, rank per GEMBA-MQM (Literary)].

 



### Comparison of different human translation versions

Figure 12 compares different human translation versions over time of the same source per lexical overlap (upper) and syntactic similarity (lower). Versions 1-3 represent translations from the oldest to newest versions in our study. Our calculations are based on paragraphs from our LitEvalCorpus. For lexical overlap with other LLM systems (upper), 8 out of 12 books show that the newest translations show substantially higher overlap with other system translations (e.g., Wuthering Heights, Frankenstein, and Jane Eyre) compared to previous versions. However, some older translations (e.g., Wuthering Heights _En-Zh_ and Death in Venice _De-En_) also show high overlap, likely because they are part of the training data of the translation systems. Regarding syntactic similarity between source and translation (lower), we observe no consistent pattern across versions, with some translations like the Metamorphosis _De-Zh_ showing higher syntactic similarity while others remain lower or comparable.

### Example of system outputs with human and metric scores, syntactic similarity, and lexical overlap

Table 15 and 16 demonstrate a full example including all system outputs with evaluation from human and automatic metrics, syntactic similarity to the source, and lexical overlap to all other systems.

Figure 12: Comparison of lexical overlap and syntactic similarity for different human translation versions.





**Sources**

Ans diesem Traun erunckite der Heimesseuchte entreur, zentrauer und korlien das Danaen verfallen. Er schueit nacht mehr die bockouhenden Blicke des Menschen; oh er seich ihren Verdacht worsetze, kirmez han nicht. Auch flosen sie ja, reisten skollerreich Strehandliche Straindhne staden lore, de Bestetzung des Speziesaus bis erg85ere Lactichen suf, and in der Studi shann selchen ein Frenonde. The Whither stehtliche artenpreceller, the Paris, toxic tzzle Zemannismandke hertenesien, niel neiner Minamirkhanich. Aber fre Sun im Pritschmannlicht hat den Beneken, si es vell e Getzine katn hez an derhangen, oder zein dassen aus dem "stand teruchs war unmena nickelstuier. Tatiho habt und jenn, iester Untimprecht, wo e verscharen, als kome Flecht und Tod alles utremut Leben in der Donar einferen um or allehn mit dem Schum- and ferer land zurtiakbleenja, isern vom vomtiges maren. Block sivk, werserserserschaftlich, werserserschaft fur eufgen Begebrsche ruhr, weven der biischsten Tape drurch Gascon, in denurs vechnimkthewerees das die Stetson umerging.

\begin{table}
\begin{tabular}{p{42.7pt} p{193.6pt} p{193.6pt} p{193.6pt} p{193.6pt} p{193.6pt}} \hline \multirow{2}{*}{**Model**} & \multicolumn{1}{c|}{**Translation**} \\  & & & & & & \\  & & & & & \\  & When be awazened from the affiliation of this dream he was unsured, datered, and hydrolysis under the power of the demon: It does not bear avoided the intuitive glunes of other people. he did not care if he was existing their suggestions. And as a matter of fact they were fleeting, travelling elsewhere. Numerous bathings showed today, the occupants of the damling medium became more and more scattered, and in the city now one merely saw a foreigner. The truth seemed to have leaked out; the panic, despite the evidence of those whose interests were missed, seconder no longer available. But the woman with the pearls remained with the family, either because the rumours had yet reached her, or became she was too rapid and fetches to bed them. Trâton remained. And to Avochelson, his institutions, it seemed attumes through flight death death death removal entirely of the disturbing elements of life around them, and he stay here during the way. By, by the sea in the forension when he's eyes tested heavily, irrespectively, unwittingly on the thing he correctly, or when, as they draw swages, he followed lambesty after him through streets where the hideouts held in secrett such times the stereos seemed to thin rich in possibilities, and laws of meatfly had dropped away. & & & & & & & \\ Human Translator & The strikscan man make home has drawn unserved and shaken, pancreas in the demands trail file to be abogter shamed the conditions places of the people around him: whether he accused their migration would so or to concern to him They were before, they nor, letting Victory man of the chouns were vacant, a number of tubes in the along room unoccupied, and in the city there was scarcely a foreigner to be seen. Then study used to have leaked out despite the tensions stability among international paths in their attempt to ward off past. But the women in penuts stayed on Sunday toward her cheovehes use (0,157) into round and features to head them. Thus traders avoid end, and Aschbach, in his teleological state, occasionally had he feeding that fight and death night tag eliminate all five standing his wis and leave lung alone on this shall with the powerful boy, indeed, when he sat in the morning by the sea, his respiratory, individuals, and firekeeping on the object for this desire, even at, when as evening fell, resumed his undersupified pursuit through the narrow streets alternatively banned by bathsome dying, things sometimes seemed superpositions and then travel cool until and void. & & & & & & & & \\ Human Transistor & From that dextran, the walkerer noise, mestilted, location, and without strength, networking to the dome. He no longer faced the watchlists passper: it didn’t holder than if they annulated their suspicious leaders. Indeed, how do he had not let, many been host stood empty, the things half indicated greater twenty, and all its alive, one seldom swim anymore. The truth seemed to spread. Not despite the stubborn only of the interested parties, panic could see he buck. But the woman advanced & & & & & & & \\ Own & With beads remained, whether shielded from the sunnotes or topa road and featries to evade them. ”Taidio” shaped. In his era, us sometimes for ten though flight and death control removed all directions from the circle, leaving him follow with the heavy, whightness of the island. Indeed, when, in the early morning, his gaze, deliberate, an accountable, and unravancing, raised on the desired object across the site, or as he followed the lethal fish looking streets during descending days, feeling unperturbed, he found the dyusal allergic increublites and the most law. & & & & & & & & \\ \hline \end{tabular} \begin{tabular}{p{42.7pt} p{193.6pt} p{193.6pt} p{193.6pt} p{193.6pt} p{193.6pt} p{193.6pt}} \end{tabular} \begin{tabular}{p{42.7pt} p{193.6pt} p{193.6pt} p{193.6pt} p{193.6pt}} \end{tabular} \begin{tabular}{p{42.7pt} p{193.6pt} p{193.6pt} p{193.6pt}} \end{tabular} \begin{tabular}{p{42.7pt} p{193.6pt} p{193.6pt}} \end{tabular} \begin{tabular}{p{42.7pt} p{193.6pt} p{193.6pt}} \end{tabular} 
\begin{tabular}{p{42.





\begin{table}
\begin{tabular}{p{42.7pt} p{190.3pt} p{56.9pt} p{85.4pt} p{85.4pt} p{85.4pt} p{85.4pt} p{85.4pt}}
**Model** & **Translation** & **Lexical validity (Sec-Tgt)** & **Human MQM (Sec-Tgt)** \\ \hline  & The rounded unkoned from his dream, unsured, disconnected and adult before the demon. He no longer faced the watchful eyes of the people; whether he should read their symptoms entered under. They were freeing too, traveling away: many head heads already changed, the dining hall was spursuedly regulated, and the dry held scarcely a stranger. Then seemed to have drained away, qualifying to consetion itself despite the departure clogging of the inverted particle. But her own in an event partly stopped with him, whenther the rumers had not reached her, we became the sea two peoand fondest raules to yold them. Tadrios remained, and in his isolation. Is sometimes screened for that flight and death could cross all raw as disturbing its floor and leave him after the twentieth on this disintles, even if this morning, at the sea they can, give him being heavily, impossible and undetected. On the reduced movie, while he thought in, in the whipering gantre, death happened in silent motor, making him unwectively of its pursuit. & From this dram the non-seeking man who e nervous, crushed, and progressively fallen to the devid. In one looster faced the observing overs of the people; whether he engomed himself to other suspicion, he did not care. There also do he, yet wear anymore more hands which attuse energy, the acquisition of the dining room showed greater epses, and in the city drew new as strange. The truth seemed to hase tracked, the panic, dropping the right chicken of the imported patters, so longer holds back. But the woman in the past jewardy remained what her owns, either because the names of data come lost or, because she was to spend and fearless to evade them. Mario remained, but him, in his inspection, it was something as if every endn and donth control myself all the disturbing life in the circle and then remained alone with the healthed on this listings, where he an incoming at the sea has lock tested heavy, irresponsible, unrelated to the desired, when he in the sitting day by the streets, is nysthetically vigged off, followed him unravibly, so the moutsors seemed to him night night and fall the law.