GenAI4ELab/papercli-papers huggingface dataset:
Targeted Source Text Editing for Machine Translation: Exploiting Quality Estimators and Large Language Models
Authors: Hyuga Koretaka; Atsushi Fujita; Tomoyuki Kajiwara
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: To improve the translation quality of “black-box” machine translation (MT) systems,we focus on the automatic editing of source texts to be translated.In addition to the use of a large language model (LLM) to implement robust and accurate editing,we investigate the usefulness of targeted editing, i.e., instructing the LLM with a text span to be edited.Our method determines such source text spans using a span-level quality estimator, which identifies actual translation errors caused by the MT system of interest, and a word aligner, which identifies alignments between the tokens in the source text and translation hypothesis.Our empirical experiments with eight MT systems and ten test datasets for four translation directionsconfirmed the efficacy of our method in improving translation quality.Through analyses, we identified several characteristics of our method andthat the segment-level quality estimator is a vital component of our method.
Self-Retrieval from Distant Contexts for Document-Level Machine Translation
Authors: Ziqian Peng; Rachel Bawden; François Yvon
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: Document-level machine translation is a challenging task, as it requires modeling both short-range and long-range dependencies to maintain the coherence and cohesion of the generated translation. However, these dependencies are sparse, and most context-augmented translation systems resort to two equally unsatisfactory options: either to include maximally long contexts, hoping that the useful dependencies are not lost in the noise; or to use limited local contexts, at the risk of missing relevant information. In this work, we study a self-retrieval-augmented machine translation framework (Self-RAMT), aimed at informing translation decisions with informative local and global contexts dynamically extracted from the source and target texts. We examine the effectiveness of this method using three large language models, considering three criteria for context selection. We carry out experiments on TED talks as well as parallel scientific articles, considering three translation directions. Our results show that integrating distant contexts with Self-RAMT improves translation quality as measured by reference-based scores and consistency metrics.
Using Encipherment to Isolate Conditions for the Successful Fine-tuning of Massively Multilingual Translation Models
Authors: Carter Louchheim; Denis Sotnichenko; Yukina Yamaguchi; Mark Hopkins
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: When fine-tuning massively multilingual translation models for low-resource languages, practitioners often include auxiliary languages to improve performance, but factors determining successful auxiliary language selection remain unclear. This paper investigates whether syntactic similarity or lexical overlap is more important for effective multilingual fine-tuning. We use encipherment to create controlled experimental conditions that disentangle these confounded factors, generating novel languages with identical syntax but no lexical overlap, and conversely, languages that preserve lexical overlap. Through extensive NLLB-200 fine-tuning experiments across Europarl and AmericasNLP datasets, we demonstrate that lexical overlap is the dominant factor. Syntactically identical auxiliary languages provide negligible benefits ( 1.0 ChrF), while languages with significant lexical overlap provide substantial improvements ( 5.0 ChrF), with effectiveness strongly correlated to KL-divergence between token distributions (r = -0.47, p .001). Our findings provide clear guidance: when selecting auxiliary languages for multilingual fine-tuning, prioritize lexical overlap over syntactic similarity.
Translate, Then Detect: Leveraging Machine Translation for Cross-Lingual Toxicity Classification
Authors: Samuel Bell; Eduardo Sánchez; David Dale; Pontus Stenetorp; Mikel Artetxe; Marta R. Costa-Jussà
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: Multilingual toxicity detection remains a significant challenge due to the scarcity of training data and resources for many languages. While prior work has leveraged the translate-test paradigm to support cross-lingual transfer across a range of classification tasks, the utility of translation in supporting toxicity detection at scale remains unclear.In this work, we conduct a comprehensive comparison of translation-based and language-specific/multilingual classification pipelines.We find that translation-based pipelines consistently outperform out-of-distribution classifiers in 81.3% of cases (13 of 16 languages), with translation benefits strongly correlated with both the resource level of the target language and the quality of the machine translation (MT) system.Our analysis reveals that traditional classifiers continue to outperform LLM-based judgment methods, with this advantage being particularly pronounced for low-resource languages, where translate-classify methods dominate translate-judge approaches in 6 out of 7 cases.We show that MT-specific fine-tuning on LLMs yields lower refusal rates compared to standard instruction-tuned models, but it can negatively impact toxicity detection accuracy for low-resource languages.These findings offer actionable guidance for practitioners developing scalable multilingual content moderation systems.
Feeding Two Birds or Favoring One? Adequacy–Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translation
Authors: Behzad Shayegh; Jan-Thorsten Peter; David Vilar; Tobias Domhan; Juraj Juraska; Markus Freitag; Lili Mou
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We investigate the tradeoff between adequacy and fluency in machine translation. We show the severity of this tradeoff at the evaluation level and analyze where popular metrics fall within it. Essentially, current metrics generally lean toward adequacy, meaning that their scores correlate more strongly with the adequacy of translations than with fluency. More importantly, we find that this tradeoff also persists at the meta-evaluation level, and that the standard WMT meta-evaluation favors adequacy-oriented metrics over fluency-oriented ones. We show that this bias is partially attributed to the composition of the systems included in the meta-evaluation datasets. To control this bias, we propose a method that synthesizes translation systems in meta-evaluation. Our findings highlight the importance of understanding this tradeoff in meta-evaluation and its impact on metric rankings.
DocHPLT: A Massively Multilingual Document-Level Translation Dataset
Authors: Dayyán O’Brien; Bhavitvya Malik; Ona de Gibert; Pinzhen Chen; Barry Haddow; Jörg Tiedemann
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: Existing document-level machine translation resources are only available for a handful of languages, mostly high-resourced ones. To facilitate the training and evaluation of document-level translation and, more broadly, long-context modeling for global communities, we create DocHPLT, the largest publicly available document-level translation dataset to date. It contains 124 million aligned document pairs across 50 languages paired with English, comprising 4.26 billion sentences. By adding pivoted alignments, practitioners can obtain 2500 additional pairs not involving English. Unlike previous reconstruction-based approaches that piece together documents from sentence-level data, we modify an existing web extraction pipeline to preserve complete document integrity from the source, retaining all content, including unaligned portions. After our preliminary experiments identify the optimal training context strategy for document-level translation, we demonstrate that LLMs fine-tuned on DocHPLT substantially outperform off-the-shelf instruction-tuned baselines, with particularly dramatic improvements for under-resourced languages. We open-source the dataset under a permissive license, providing essential infrastructure for advancing multilingual document-level translation.
SONAR-SLT: Multilingual Sign Language Translation via Language-Agnostic Sentence Embedding Supervision
Authors: Yasser Hamidullah; Shakib Yazdani; Cennet Oguz; Josef Van Genabith; Cristina España-Bonet
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: Sign language translation (SLT) is typically trained with text in a single spoken language, which limits scalability and cross-language generalization. Earlier approaches have replaced gloss supervision with text-based sentence embeddings, but up to now, these remain tied to a specific language and modality. In contrast, here we employ language-agnostic, multimodal embeddings trained on text and speech from multiple languages to supervise SLT, enabling direct multilingual translation. To address data scarcity, we propose a coupled augmentation method that combines multilingual target augmentations (i.e. translations into many languages) with video-level perturbations, improving model robustness. Experiments show consistent BLEURT gains over text-only sentence embedding supervision, with larger improvements in low-resource settings. Our results demonstrate that language-agnostic embedding supervision, combined with coupled augmentation, provides a scalable and semantically robust alternative to traditional SLT training.
GAMBIT+: A Challenge Set for Evaluating Gender Bias in Machine Translation Quality Estimation Metrics
Authors: George Filandrianos; Orfeas Menis Mastromichalakis; Wafaa Mohammed; Giuseppe Attanasio; Chrysoula Zerva
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: Gender bias in machine translation (MT) systems has been extensively documented, but bias in automatic quality estimation (QE) metrics remains comparatively underexplored. Existing studies suggest that QE metrics can also exhibit gender bias, yet most analyses are limited by small datasets, narrow occupational coverage, and restricted language variety. To address this gap, we introduce a large-scale challenge set specifically designed to probe the behavior of QE metrics when evaluating translations containing gender-ambiguous occupational terms. Building on the GAMBIT corpus of English texts with gender-ambiguous occupations, we extend coverage to three source languages that are genderless or natural-gendered, and eleven target languages with grammatical gender, resulting in 33 source–target language pairs. Each source text is paired with two target versions differing only in the grammatical gender of the occupational term(s) (masculine vs. feminine), with all dependent grammatical elements adjusted accordingly. An unbiased QE metric should assign equal or near-equal scores to both versions. The dataset’s scale, breadth, and fully parallel design, where the same set of texts is aligned across all languages, enables fine-grained bias analysis by occupation and systematic comparisons across languages.
Implementing and Evaluating Multi-source Retrieval-Augmented Translation
Authors: Tommi Nieminen; Jörg Tiedemann; Sami Virpioja
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: In recent years, neural machine translation (NMT) systems have been integrated with external databases with the aim of improving machine translation (MT) quality and enforcing domain-specific terminology and other conventions in the MT output. Most of the work in incorporating external knowledge with NMT has concentrated on integrating a single source of information, usually either a terminology database or a translation memory. However, in real-life translation scenarios, all relevant knowledge sources should be used in parallel. In this article, we evaluate different methods of integrating external knowledge from multiple sources in a single NMT system. In addition to training single models trained to utilize multiple kinds of information, we also ensemble models that have been trained to utilize a single type of information. We evaluate our models against state-of-the-art LLMs using an extensive purpose-built English to Finnish test suite.
A Cross-Lingual Perspective on Neural Machine Translation Difficulty
Authors: Esther Ploeger; Johannes Bjerva; Jörg Tiedemann; Robert Östling
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: Intuitively, machine translation (MT) between closely related languages, such as Swedish and Danish, is easier than MT between more distant pairs, such as Finnish and Danish. Yet, the notions of ‘closely related’ languages and ‘easier’ translation have so far remained underspecified. Moreover, in the context of neural MT, this assumption was almost exclusively evaluated in scenarios where English was either the source or target language, leaving a broader cross-lingual view unexplored. In this work, we present a controlled study of language similarity and neural MT difficulty for 56 European translation directions. We test a range of language similarity metrics, some of which are reasonable predictors of MT difficulty. On a text-level, we reassess previously introduced indicators of MT difficulty, and find that they are not well-suited to our domain, or neural MT more generally. Ultimately, we hope that this work inspires further cross-lingual investigations of neural MT difficulty
Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets
Authors: Tom Kocmi; Ekaterina Artemova; Eleftherios Avramidis; Rachel Bawden; Ondřej Bojar; Konstantin Dranch; Anton Dvorkovich; Sergey Dukanov; Mark Fishel; Markus Freitag; Thamme Gowda; Roman Grundkiewicz; Barry Haddow; Marzena Karpinska; Philipp Koehn; Howard Lakougna; Jessica Lundin; Christof Monz; Kenton Murray; Masaaki Nagata; Stefano Perrella; Lorenzo Proietti; Martin Popel; Maja Popović; Parker Riley; Mariya Shmatova; Steinthór Steingrímsson; Lisa Yankovskaya; Vilém Zouhar
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: This paper presents the results of the General Machine Translation Task organized as part of the 2025 Conference on Machine Translation (WMT). Participants were invited to build systems for any of 30 language pairs. For half of these pairs, we conducted a human evaluation on test sets spanning four to five different domains.We evaluated 60 systems in total: 36 submitted by participants and 24 for which we collected translations from large language models (LLMs) and popular online translation providers.This year, we focused on creating challenging test sets by developing a difficulty sampling technique and using more complex source data. We evaluated system outputs with professional annotators using the Error Span Annotation (ESA) protocol, except for two language pairs, for which we used Multidimensional Quality Metrics (MQM) instead.We continued the trend of increasingly moving towards document-level translation, providing the source texts as whole documents containing multiple paragraphs.
Findings of the WMT25 Multilingual Instruction Shared Task: Persistent Hurdles in Reasoning, Generation, and Evaluation
Authors: Tom Kocmi; Sweta Agrawal; Ekaterina Artemova; Eleftherios Avramidis; Eleftheria Briakou; Pinzhen Chen; Marzieh Fadaee; Markus Freitag; Roman Grundkiewicz; Yupeng Hou; Philipp Koehn; Julia Kreutzer; Saab Mansour; Stefano Perrella; Lorenzo Proietti; Parker Riley; Eduardo Sánchez; Patricia Schmidtova; Mariya Shmatova; Vilém Zouhar
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: The WMT25 Multilingual Instruction Shared Task (MIST) introduces a benchmark to evaluate large language models (LLMs) across 30 languages. The benchmark covers five types of problems: machine translation, linguistic reasoning, open-ended generation, cross-lingual summarization, and LLM-as-a-judge.We provide automatic evaluation and collect human annotations, which highlight the limitations of automatic evaluation and allow further research into metric meta-evaluation. We run on our benchmark a diverse set of open- and closed-weight LLMs, providing a broad assessment of the multilingual capabilities of current LLMs. Results highlight substantial variation across sub-tasks and languages, revealing persistent challenges in reasoning, cross-lingual generation, and evaluation reliability. This work establishes a standardized framework for measuring future progress in multilingual LLM development.
Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help
Authors: Alon Lavie; Greg Hanneman; Sweta Agrawal; Diptesh Kanojia; Chi-Kiu Lo; Vilém Zouhar; Frederic Blain; Chrysoula Zerva; Eleftherios Avramidis; Sourabh Deoghare; Archchana Sindhujan; Jiayi Wang; David Ifeoluwa Adelani; Brian Thompson; Tom Kocmi; Markus Freitag; Daniel Deutsch
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: The WMT25 Shared Task on Automated Translation Evaluation Systems evaluates metrics and quality estimation systems that assess the quality of language translation systems. This task unifies and consolidates the separate WMT shared tasks on Machine Translation Evaluation Metrics and Quality Estimation from previous years. Our primary goal is to encourage the development and assessment of new state-of-the-art translation quality evaluation systems. The shared task this year consisted of three subtasks: (1) segment-level quality score prediction, (2) span-level translation error annotation, and (3) quality-informed segment-level error correction. The evaluation data for the shared task were provided by the General MT shared task and were complemented by “challenge sets” from both the organizers and participants. Task 1 results indicate the strong performance of large LLMs at the system level, whilereference-based baseline metrics outperform LLMs at the segment level. Task 2 results indicate that accurate error detection and balancing precision and recall are persistent challenges. Task 3 results show that minimal editing is challenging even when informed by quality indicators. Robustness across the broad diversity of languages remains a major challenge across all three subtasks.
Findings of the WMT 2025 Shared Task on Model Compression: Early Insights on Compressing LLMs for Machine Translation
Authors: Marco Gaido; Roman Grundkiewicz; Thamme Gowda; Matteo Negri
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We present the results of the first edition of the Model Compression shared task, organized as part of the 10th Conference on Machine Translation (WMT25). The task challenged participants to compress Large Language Models (LLMs) toward enabling practical deployment in resource-constrained scenarios, while minimizing loss in translation performance. In this edition, participants could choose to compete in either a constrained track, which required compressing a specific model (Aya Expanse 8B) evaluated on a limited set of language pairs (Czech→German, Japanese→Chinese, and English→Arabic), or an unconstrained track, which placed no restrictions on the model and allowed submissions for any of the 15 language directions covered by the General MT task (GenMT). We received 12 submissions from three teams, all in the constrained track. They proposed different compression solutions and covered various language combinations: all targeted Czech→German, while one covered all the language pairs. Evaluation was conducted separately for each language, measuring translation quality using COMET and MetricX, model size, and inference speed on an Nvidia A100 GPU.
Findings of the WMT 2025 Shared Task of the Open Language Data Initiative
Authors: David Dale; Laurie Burchell; Jean Maillard; Idris Abdulmumin; Antonios Anastasopoulos; Isaac Caswell; Philipp Koehn
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We present the results of the WMT 2025 shared task of the Open Language Data Initiative. Participants were invited to contribute to the massively multilingual open datasets (FLORES+, MT Seed, WMT24++) or create new such resources. We accepted 8 submissions, including 7 extensions or revisions of the existing datasets and one submission with a new parallel training dataset, SMOL.
Findings of the WMT 2025 Shared Task LLMs with Limited Resources for Slavic Languages: MT and QA
Authors: Shu Okabe; Daryna Dementieva; Marion Di Marco; Lukas Edman; Katharina Haemmerl; Marko Měškank; Anita Hendrichowa; Alexander Fraser
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We present the findings of the WMT 2025 Shared Task LLMs with Limited Resources for Slavic Languages. This shared task focuses on training LLMs using limited data and compute resources for three Slavic languages: Upper Sorbian (hsb), Lower Sorbian (dsb), and Ukrainian (uk), with the objective to develop and improve LLMs for these languages. We consider two tasks which are to be evaluated jointly: Machine Translation (MT) and Multiple-Choice Question Answering (QA). In total, three teams participated in this shared task, with submissions from all three teams for the Sorbian languages and one submission for Ukrainian. All submissions led to an improvement compared to the baseline Qwen2.5-3B model through varying fine-tuning strategies. We note, however, that training purely on MT degrades original QA capabilities. We also report further analyses on the submissions, including MT evaluation using advanced neural metrics for Ukrainian, as well as manual annotation and comparison to the current Sorbian machine translator.
Findings of the First Shared Task for Creole Language Machine Translation at WMT25
Authors: Nathaniel Robinson; Claire Bizon Monroc; Rasul Dent; Stefan Watson; Kenton Murray; Raj Dabre; Andre Coy; Heather Lent
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: Efforts towards better machine translation (MT) for Creole languages have historically been isolated, due to Creole languages’ geographic and linguistic diversity. However, most speakers of Creole languages stand to benefit from improved MT for low-resource languages. To galvanize collaboration for Creole MT across the NLP community, we introduce the First Shared Task for Creole Language Machine Translation at WMT25. This Shared Task consists of two systems tracks and one data track, for which we received submissions from five participating teams. Participants experimented with a wide variety of systems development techniques. Our evaluation campaign gave rise to improvements in MT performance in several languages, and particularly large improvements in new testing genres, though some participants found that reusing subsets of pretraining data for specialized post-training did not yield significant improvements. Our campaign also yielded new test sets for Mauritian Creole and a vast expansion of public training data for two Creole languages of Latin America.