GenAI4ELab/papercli-papers huggingface dataset:
MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task
Authors: Juraj Juraska; Tobias Domhan; Mara Finkelstein; Tetsuji Nakagawa; Geza Kovacs; Daniel Deutsch; Pidong Wang; Markus Freitag
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: In this paper, we present our submissions to the unified WMT25 Translation Evaluation Shared Task. For the Quality Score Prediction subtask, we create a new generation of MetricX with improvements in the input format and the training protocol, while for the Error Span Detection subtask we develop a new model, GemSpanEval, trained to predict error spans along with their severities and categories. Both systems are based on the state-of-the-art multilingual open-weights model Gemma 3, fine-tuned on publicly available WMT data. We demonstrate that MetricX-25, adapting Gemma 3 to an encoder-only architecture with a regression head on top, can be trained to effectively predict both MQM and ESA quality scores, and significantly outperforms its predecessor. Our decoder-only GemSpanEval model, on the other hand, we show to be competitive in error span detection with xCOMET, a strong encoder-only sequence-tagging baseline. With error span detection formulated as a generative task, we instruct the model to also output the context for each predicted error span, thus ensuring that error spans are identified unambiguously.
HW-TSC’s Submissions to the WMT 2025 Segment-level Quality Score Prediction Task
Authors: Yuanchang Luo; Jiaxin Guo; Daimeng Wei; Hengchao Shang; Zongyao Li; Zhiqiang Rao; Jinlong Yang; Zhanglin Wu; Xiaoyu Chen; Hao Yang
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: This paper presents the submissions of Huawei Translate Services Center (HW-TSC) to the WMT 2025 Segment-level quality score prediction Task. We participate in 16 language pairs. For the prediction of translation quality scores for long multi-sentence text units, we propose an automatic evaluation framework based on alignment algorithms. Our approach integrates sentence segmentation tools and dynamic programming to construct sentence-level alignments between source and translated texts, then adapts sentence-level evaluation models to document-level assessment via sliding-window aggregation. Our submissions achieved competitive results in the final evaluations of all language pairs we participated in.
UvA-MT at WMT25 Evaluation Task: LLM Uncertainty as a Proxy for Translation Quality
Authors: Di Wu; Christof Monz
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: This year, we focus exclusively on using the uncertainty quantification as a proxy for translation quality. While this has traditionally been regarded as a form of unsupervised quality estimation, such signals have been overlooked in the design of the current metric models—we show their value in the context of LLMs. More specifically, in contrast to conventional unsupervised QE methods, we apply recent calibration technology to adjust translation likelihoods to better align with quality signals, and we use the single resulting model to participate in both the general translation and QE tracks at WMT25.Our offline experiments show some advantages: 1) uncertainty signals extracted from LLMs, like Tower or Gemma-3, provide accurate quality predictions; and 2) calibration technology further improves this QE performance, sometimes even surpassing certain metric models that were trained with human annotations, such as CometKiwi. We therefore argue that uncertainty quantification (confidence), especially from LLMs, can serve as a strong and complementary signal for the metric design, particularly when human-annotated data are lacking. However, we also identify limitations, such as its tendency to assign disproportionately higher scores to hypotheses generated by the model itself.
Submission for WMT25 Task 3
Authors: Govardhan Padmanabhan
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: The paper presents two approaches submitted to the WMT 2025 Automated Translation Quality Evaluation Systems Task 3 - Quality Estimation (QE)-informed Segment-level Error Correction. While jointly training QE systems with Automatic Post-Editing (APE) has shown improved performance for both tasks, APE systems are still known to overcorrect the output of Machine Translation (MT), leading to a degradation in performance. We investigate a simple training-free approach - QE-informed Retranslation, and compare it with another within the same training-free paradigm. Our winning approach selects the highest-quality translation from multiple candidates generated by different LLMs. The second approach, more akin to APE, instructs an LLM to replace error substrings as specified in the provided QE explanation(s). A conditional heuristic was employed to minimise the number of edits, with the aim of maximising the Gain-to-Edit ratio. The two proposed approaches achieved a ∆COMET scoreof 0.0201 and −0.0108, respectively, leading the first approach to achieve the winning position on the subtask leaderboard.
RankedCOMET: Elevating a 2022 Baseline to a Top-5 Finish in the WMT 2025 QE Task
Authors: Sujal Maharjan; Astha Shrestha
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: This paper presents rankedCOMET, a lightweight per-language-pair calibration applied to the publicly available Unbabel/wmt22-comet-da model that yields a competitive Quality Estimation (QE) system for the WMT 2025 shared task. This approach transforms raw model outputs into per-language average ranks and min–max normalizes those ranks to [0,1], maintaining intra-language ordering while generating consistent numeric ranges across language pairs. Applied to 742,740 test segments and submitted to Codabench, this unsupervised post-processing enhanced the aggregated Pearson correlation on the preliminary snapshot and led to a 5th-place finish. We provide detailed pseudocode, ablations (including a negative ensemble attempt), and a reproducible analysis pipeline providing Pearson, Spearman, and Kendall correlations with bootstrap confidence intervals.
Quality-Informed Segment-Level Error Correction Using Natural Language Explanations from xTower and Large Language Models
Authors: Prashant Sharma
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: This paper describes our submission to the WMT25 Automated Translation Quality Evaluation Systems Task 3 - QE-informed Segment-level Error Correction. We propose a two-step approach for Automatic Post-Editing (APE) that leverages natural language explanations of translation errors. Our method first utilises the xTower model to generate a descriptive explanation of the errors present in a machine-translated segment, given the source text, the machine translation, and quality estimation annotations. This explanation is then provided as a prompt to a powerful Large Language Model, Gemini 1.5 Pro, which generates the final, corrected translation. This approach is inspired by recent work in edit-based APE and aims to improve the interpretability and performance of APE systems. We Evaluated across six language pairs (EN→ZH, EN→CS, EN→IS, EN→JA, EN→RU, EN→UK), our approach demonstrates promising results, especially in cases requiring fine-grained edits.
TASER: Translation Assessment via Systematic Evaluation and Reasoning
Authors: Monishwaran Maheswaran; Marco Carini; Christian Federmann; Tony Diaz
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We introduce TASER (Translation Assessment via Systematic Evaluation and Reasoning), a metric that uses Large Reasoning Models (LRMs) for automated translation quality assessment. TASER harnesses the explicit reasoning capabilities of LRMs to conduct systematic, step-by-step evaluation of translation quality. We evaluate TASER on the WMT24 Metrics Shared Task across both reference-based and reference-free scenarios, demonstrating state-of-the-art performance. In system-level evaluation, TASER achieves the highest soft pairwise accuracy in both reference-based and reference-free settings, outperforming all existing metrics. At the segment level, TASER maintains competitive performance with our reference-free variant ranking as the top-performing metric among all reference-free approaches. Our experiments reveal that structured prompting templates yield superior results with LRMs compared to the open-ended approaches that proved optimal for traditional LLMs. We evaluate o3, a large reasoning model from OpenAI, with varying reasoning efforts, providing insights into the relationship between reasoning depth and evaluation quality. The explicit reasoning process in LRMs offers interpretability and visibility, addressing a key limitation of existing automated metrics. Our results demonstrate that Large Reasoning Models show a measurable advancement in translation quality assessment, combining improved accuracy with transparent evaluation across diverse language pairs.
Vicomtech@WMT 2025: Evolutionary Model Compression for Machine Translation
Authors: David Ponce; Harritxu Gete; Thierry Etchegoyhen
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We describe Vicomtech’s participation in the WMT 2025 Shared Task on Model Compression. We addressed all three language pairs of the constrained task, namely Czech to German, English to Arabic and Japanese to Chinese, using the Aya Expanse 8B model as our base model. Our approach centers on GeLaCo, an evolutionary method for LLM compression via layer collapse operations, which efficiently explores the compression solution space through population-based search and a module-wise similarity fitness function that captures attention, feed-forward, and hidden state representations. We systematically evaluated compression at three different ratios (0.25, 0.50, and 0.75) and applied targeted post-training techniques to recover performance through fine-tuning and knowledge distillation over translation instructions. Additionally, we explored quantization techniques to achieve further model size reduction. Our experimental results demonstrate that the combination of evolutionary layer compression, targeted post-training, and quantization can achieve substantial model size reduction while maintaining competitive translation quality across all language pairs.
Iterative Layer Pruning for Efficient Translation Inference
Authors: Yasmin Moslem; Muhammad Hazim Al Farouq; John Kelleher
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: Large language models (LLMs) have transformed many areas of natural language processing, including machine translation. However, efficient deployment of LLMs remains challenging due to their intensive computational requirements. In this paper, we address this challenge and present our submissions to the Model Compression track at the Conference on Machine Translation (WMT 2025). In our experiments, we investigate iterative layer pruning guided by layer importance analysis. We evaluate this method using the Aya-Expanse-8B model for translation from Czech to German, and from English to Egyptian Arabic. Our approach achieves substantial reductions in model size and inference time, while maintaining the translation quality of the baseline models.
Expanding the WMT24++ Benchmark with Rumantsch Grischun, Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader
Authors: Jannis Vamvas; Ignacio Pérez Prat; Not Soliva; Sandra Baltermia-Guetg; Andrina Beeli; Simona Beeli; Madlaina Capeder; Laura Decurtins; Gian Peder Gregori; Flavia Hobi; Gabriela Holderegger; Arina Lazzarini; Viviana Lazzarini; Walter Rosselli; Bettina Vital; Anna Rutkiewicz; Rico Sennrich
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: The Romansh language, spoken in Switzerland, has limited resources for machine translation evaluation. In this paper, we present a benchmark for six varieties of Romansh: Rumantsch Grischun, a supra-regional variety, and five regional varieties: Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader. Our reference translations were created by human translators based on the WMT24++ benchmark, which ensures parallelism with more than 55 other languages. An automatic evaluation of existing MT systems and LLMs shows that translation out of Romansh into German is handled relatively well for all the varieties, but translation into Romansh is still challenging.
A French Version of the OLDI Seed Corpus
Authors: Malik Marmonier; Benoît Sagot; Rachel Bawden
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We present the first French partition of the OLDI Seed Corpus, our submission to the WMT 2025 Open Language Data Initiative (OLDI) shared task. We detail its creation process, which involved using multiple machine translation systems and a custom-built interface for post-editing by qualified native speakers. We also highlight the unique translation challenges presented by the source data, which combines highly technical, encyclopedic terminology with the stylistic irregularities characteristic of user-generated content taken from Wikipedia. This French corpus is not an end in itself, but is intended as a crucial pivot resource to facilitate the collection of parallel corpora for the under-resourced regional languages of France.
Bringing Ladin to FLORES+
Authors: Samuel Frontull; Thomas Ströhle; Carlo Zoli; Werner Pescosta; Ulrike Frenademez; Matteo Ruggeri; Daria Valentin; Karin Comploj; Gabriel Perathoner; Silvia Liotto; Paolo Anvidalfarei
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: Recent advances in neural machine translation (NMT) have opened new possibilities for developing translation systems also for smaller, so-called low-resource, languages. The rise of large language models (LLMs) has further revolutionized machine translation by enabling more flexible and context-aware generation. However, many challenges remain for low-resource languages, and the availability of high-quality, validated test data is essential to support meaningful development, evaluation, and comparison of translation systems. In this work, we present an extension of the FLORES+ dataset for two Ladin variants, Val Badia and Gherdëina, as a submission to the Open Language Data Initiative Shared Task 2025. To complement existing resources, we additionally release two parallel datasets for Gherdëina–Val Badia and Gherdëina–Italian. We validate these datasets by evaluating state-of-the-art LLMs and NMT systems on this test data, both with and without leveraging the newly released parallel data for fine-tuning and prompting. The results highlight the considerable potential for improving translation quality in Ladin, while also underscoring the need for further research and resource development, for which this contribution provides a basis.
Correcting the Tamazight Portions of FLORES+ and OLDI Seed Datasets
Authors: Alp Oktem; Mohamed Aymane Farhi; Brahim Essaidi; Naceur Jabouja; Farida Boudichat
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We present the manual correction of the Tamazight portions of the FLORES+ and OLDI Seed datasets to improve the quality of open machine translation resources for the language. These widely used reference corpora contained numerous issues, including mistranslations, orthographic inconsistencies, overuse of loanwords, and non-standard transliterations. Overall, 36% of FLORES+ and 40% of Seed sentences were corrected by expert linguists, with average token divergence of 19% and 25% among changed items. Evaluation of multiple MT systems, including NLLB models and commercial LLM services, showed consistent gains in automated evaluation metrics when using the corrected data. Fine-tuning NLLB-600M on the revised Seed corpus yielded improvements of +6.05 chrF (en→zgh) and +2.32 (zgh→en), outperforming larger parameter models and LLM providers in en→zgh direction.
Filling the Gap for Uzbek: Creating Translation Resources for Southern Uzbek
Authors: Mukhammadsaid Mamasaidov; Azizullah Aral; Abror Shopulatov; Mironshoh Inomjonov
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: Southern Uzbek (uzs) is a Turkic language variety spoken by around 5 million people in Afghanistan and differs significantly from Northern Uzbek (uzn) in phonology, lexicon, and orthography. Despite the large number of speakers, Southern Uzbek is underrepresented in natural language processing. We present new resources for Southern Uzbek machine translation, including a 997-sentence FLORES+ dev set, 39,994 parallel sentences from dictionary, literary, and web sources, and a fine-tuned NLLB-200 model (lutfiy). We also propose a post-processing method for restoring Arabic-script half-space characters, which improves handling of morphological boundaries. All datasets, models, and tools are released publicly to support future work on Southern Uzbek and other low-resource languages.
The Kyrgyz Seed Dataset Submission to the WMT25 Open Language Data Initiative Shared Task
Authors: Murat Jumashev; Alina Tillabaeva; Aida Kasieva; Turgunbek Omurkanov; Akylai Musaeva; Meerim Emil Kyzy; Gulaiym Chagataeva; Jonathan Washington
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We present a Kyrgyz language seed dataset as part of our contribution to the WMT25 Open Language Data Initiative (OLDI) shared task. This paper details the process of collecting and curating English–Kyrgyz translations, highlighting the main challenges encountered in translating into a morphologically rich, low-resource language. We demonstrate the quality of the dataset through fine-tuning experiments, showing consistent improvements in machine translation performance across multiple models. Comparisons with bilingual and MNMT Kyrgyz-English baselines reveal that, for some models, our dataset enables performance surpassing pretrained baselines in both English–Kyrgyz and Kyrgyz–English translation directions. These results validate the dataset’s utility and suggest that it can serve as a valuable resource for the Kyrgyz MT community and other related low-resource languages.
SMOL: Professionally Translated Parallel Data for 115 Under-represented Languages
Authors: Isaac Caswell; Elizabeth Nielsen; Jiaming Luo; Colin Cherry; Geza Kovacs; Hadar Shemtov; Partha Talukdar; Dinesh Tewari; Baba Mamadi Diane; Djibrila Diane; Solo Farabado Cissé; Koulako Moussa Doumbouya; Edoardo Ferrante; Alessandro Guasoni; Christopher Homan; Mamadou K. Keita; Sudhamoy DebBarma; Ali Kuzhuget; David Anugraha; Muhammad Ravi Shulthan Habibi; Sina Ahmadi; Anthony Munthali; Jonathan Mingfei Liu; Jonathan Eng
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We open-source SMOL (Set of Maximal Over-all Leverage), a suite of training data to un-lock machine translation for low-resource languages (LRLs). SMOL has been translated into123 under-resourced languages (125 language pairs), including many for which there exist no previous public resources, for a total of 6.1M translated tokens. SMOL comprises two sub-datasets, each carefully chosen for maximum impact given its size: SMOLSENT, a set of sentences chosen for broad unique token coverage, and SMOLDOC, a document-level source focusing on a broad topic coverage. They join the already released GATITOS for a trifecta of paragraph, sentence, and token-level content. We demonstrate that using SMOL to prompt or fine-tune Large Language Models yields robust chrF improvements. In addition to translation, we provide factuality ratings and rationales for all documents in SMOLDOC, yielding the first factuality datasets for most of these languages.
Improved Norwegian Bokmål Translations for FLORES
Authors: Petter Mæhlum; Anders Næss Evensen; Yves Scherrer
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: FLORES+ is a collection of parallel datasets obtained by translation from originally English source texts. FLORES+ contains Norwegian translations for the two official written variants of Norwegian: Norwegian Bokmål and Norwegian Nynorsk. However, the earliest Bokmål version contained non-native-like mistakes, and even after a later revision, the dataset contained grammatical and lexical errors. This paper aims at correcting unambiguous mistakes, and thus creating a new version of the Bokmål dataset. At the same time, we provide a translation into Radical Bokmål, a sub-variety of Norwegian which is closer to Nynorsk in some aspects, while still being within the official norms for Bokmål. We discuss existing errors and differences in the various translations and the corrections that we provide.
NRC Systems for the WMT2025-LRSL Shared Task
Authors: Samuel Larkin; Chi-Kiu Lo; Rebecca Knowles
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: We describe the NRC team systems for the WMT25 Shared Tasks on Large Language Models (LLMs) with Limited Resources for Slavic Languages. We participate in the Lower Sorbian and Upper Sorbian Machine Translation and Question Answering tasks. On the machine translation tasks, our primary focus, our systems rank first according to the automatic MT evaluation metric (chrF). Our systems underperform on the QA tasks.
TartuNLP at WMT25 LLMs with Limited Resources for Slavic Languages Shared Task
Authors: Taido Purason; Mark Fishel
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: This paper describes the TartuNLP submission to the Upper Sorbian (hsb) and Lower Sorbian (dsb) tracks of the WMT25 LLMs with Limited Resources for Slavic Languages shared task, which jointly targets machine translation (MT) and question answering (QA). We develop a single multilingual model based on Qwen2.5-3B-Instruct by continuing pretraining on Sorbian monolingual and parallel data together with general instruction datasets, combining language acquisition and instruction-following in a single step. The resulting model delivers substantial improvements over the baseline Qwen2.5-3B-Instruct model and also achieves the highest ranking for both tasks in the hsb and dsb shared task tracks.
JGU Mainz’s Submission to the WMT25 Shared Task on LLMs with Limited Resources for Slavic Languages: MT and QA
Authors: Hossain Shaikh Saadi; Minh Duc Bui; Mario Sanz-Guerrero; Katharina Von Der Wense
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: This paper presents the JGU Mainz submission to the WMT25 Shared Task on LLMs with Limited Resources for Slavic Languages: Machine Translation and Question Answering, focusing on Ukrainian, Upper Sorbian, and Lower Sorbian. For each language, we jointly fine-tune a Qwen2.5-3B-Instruct model for both tasks with parameter-efficient finetuning. Our pipeline integrates additional translation and multiple-choice question answering (QA) data. For Ukrainian QA, we further use retrieval-augmented generation. We also apply ensembling for QA in Upper and Lower Sorbian. Experiments show that our models outperform the baseline on both tasks.
Krey-All WMT 2025 CreoleMT System Description: Language Agnostic Strategies for Low-Resource Translation
Authors: Ananya Ayasi
Venue: EMNLP 2025 (Proceedings of the Tenth Conference on Machine Translation)

Abstract: The paper is a submission to the WMT 2025 Creole MT Task, aimed at improving translation between Seychellois Creole and English by using data from other closely related Creole languages. The approach treats all Creoles as variations of the same language, allowing the model to share knowledge across them. This strategy led to better translation quality than using separate language settings. The work highlights that related languages can effectively support each other in low-resource settings, while also noting challenges such as data imbalance, differences in language domains, and the absence of human evaluation.