GenAI4ELab/papercli-papers huggingface dataset:
Results of the WAT/WMT 2024 Shared Task on Patent Translation
Authors: Shohei Higashiyama
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the results of the patent translation shared task at the 11th Workshop on Asian Translation and 9th Conference on Machine Translation. Two teams participated in this task, and their submitted translation results for one or more of the six language directions were automatically and manually evaluated. The evaluation results demonstrate the strong performance of large language model-based systems from both participants.
Findings of the WMT 2024 Biomedical Translation Shared Task: Test Sets on Abstract Level
Authors: Mariana Neves; Cristian Grozea; Philippe Thomas; Roland Roller; Rachel Bawden; Aurélie Névéol; Steffen Castle; Vanessa Bonato; Giorgio Maria Di Nunzio; Federica Vezzani; Maika Vicente Navarro; Lana Yeganova; Antonio Jimeno Yepes
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: We present the results of the ninth edition of the Biomedical Translation Task at WMT’24. We released test sets for six language pairs, namely, French, German, Italian, Portuguese, Russian, and Spanish, from and into English. Eachtest set consists of 50 abstracts from PubMed. Differently from previous years, we did not split abstracts into sentences. We received submissions from five teams, and for almost all language directions. We used a baseline/comparison system based on Llama 3.1 and share the source code at https://github.com/cgrozea/wmt24biomed-ref.
MSLC24 Submissions to the General Machine Translation Task
Authors: Samuel Larkin; Chi-Kiu Lo; Rebecca Knowles
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: The MSLC (Metric Score Landscape Challenge) submissions for English-German, English-Spanish, and Japanese-Chinese are constrained systems built using Transformer models for the purpose of better evaluating metric performance in the WMT24 Metrics Task. They are intended to be representative of the performance of systems that can be built relatively simply using constrained data and with minimal modifications to the translation training pipeline.
IOL Research Machine Translation Systems for WMT24 General Machine Translation Shared Task
Authors: Wenbo Zhang
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper illustrates the submission system of the IOL Research team for the WMT24 General Machine Translation shared task. We submitted translations for all translation directions in the general machine translation task. According to the official track categorization, our system qualifies as an open system due to the utilization of open-source resources in developing our machine translation model. With the growing prevalence of large language models (LLMs) as a conventional approach for managing diverse NLP tasks, we have developed our machine translation system by leveraging the capabilities of LLMs. Overall, we first performed continued pretraining using the open-source LLMs with tens of billions of parameters to enhance the model’s multilingual capabilities. Subsequently, we employed open-source Large Language Models, equipped with hundreds of billions of parameters, to generate synthetic data. This data was then blended with a modest quantity of additional open-source data for precise supervised fine-tuning. In the final stage, we also used ensemble learning to improve translation quality. Based on the official automated evaluation metrics, our system excelled by securing the top position in 8 out of the total 11 translation directions, spanning both open and constrained system categories.
Choose the Final Translation from NMT and LLM Hypotheses Using MBR Decoding: HW-TSC’s Submission to the WMT24 General MT Shared Task
Authors: Zhanglin Wu; Daimeng Wei; Zongyao Li; Hengchao Shang; Jiaxin Guo; Shaojun Li; Zhiqiang Rao; Yuanchang Luo; Ning Xie; Hao Yang
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the submission of Huawei Translate Services Center (HW-TSC) to the WMT24 general machine translation (MT) shared task, where we participate in the English to Chinese (en→zh) language pair. Similar to previous years’ work, we use training strategies such as regularized dropout, bidirectional training, data diversification, forward translation, back translation, alternated training, curriculum learning, and transductive ensemble learning to train the neural machine translation (NMT) model based on the deep Transformer-big architecture. The difference is that we also use continue pre-training, supervised fine-tuning, and contrastive preference optimization to train the large language model (LLM) based MT model. By using Minimum Bayesian risk (MBR) decoding to select the final translation from multiple hypotheses for NMT and LLM-based MT models, our submission receives competitive results in the final evaluation.
CycleGN: A Cycle Consistent Approach for Neural Machine Translation
Authors: Sören Dreano; Derek Molloy; Noel Murphy
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: CycleGN is a fully self-supervised Neural Machine Translation framework relying on the Transformer architecture that does not require parallel data. Its approach is similar to a Discriminator-less CycleGAN, hence the “non-adversarial” name, specifically tailored for non-parallel text datasets. The foundational concept of our research posits that in an ideal scenario, retro-translations of generated translations should revert to the original source sentences. Consequently, a pair of models can be trained using a Cycle Consistency Loss (CCL) only, with one model translating in one direction and the second model in the opposite direction.In the context of this research, two sub-categories of non-parallel datasets are introduced. A “permuted” dataset is defined as a parallel dataset wherein the sentences of one language have been systematically rearranged. Consequently, this results in a non-parallel corpus where it is guaranteed that each sentence has a corresponding translation located at an unspecified index within the dataset. A “non-intersecting” dataset is a non-parallel dataset for which it is guaranteed that no sentence has an exact translation.Masked Language Modeling (MLM) is a pre-training strategy implemented in BERT, where a specified proportion of the input tokens are substituted with a unique $mask$ token. The objective of the neural network under this paradigm is to accurately reconstruct the original sentence from this degraded input.In inference mode, Transformers are able to generate sentences without labels. Thus, the first step is to generate pseudo-labels in inference, that are then used as labels during training. However, the models consistently converge towards a trivial solution in which the input, the generated pseudo-labels and the output are identical, achieving an optimal outcome on the CCL function, registering a value of zero. CycleGN demonstrates how MLM pre-training can be leveraged to move away from this trivial path and perform actual text translation.As a contribution to the WMT24 challenge, this study explores the efficacy of the CycleGN architectural framework in learning translation tasks across eleven language pairs under the permuted condition and four under the non-intersecting condition. Moreover, two additional language pairs from the previous WMT edition were trained and the evaluations demonstrate the robust adaptability of CycleGN in learning translation tasks.
UvA-MT’s Participation in the WMT24 General Translation Shared Task
Authors: Shaomu Tan; Di Wu; David Stap; Seth Aycock; Christof Monz
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: Fine-tuning Large Language Models (FT-LLMs) with parallel data has emerged as a promising paradigm in recent machine translation research. In this paper, we explore the effectiveness of FT-LLMs and compare them to traditional encoder-decoder Neural Machine Translation (NMT) systems under the WMT24 general MT shared task for English to Chinese direction. We implement several techniques, including Quality Estimation (QE) data filtering, supervised fine-tuning, and post-editing that integrate NMT systems with LLMs. We demonstrate that fine-tuning LLaMA2 on a high-quality but relatively small bitext dataset (100K) yields COMET results comparable to much smaller encoder-decoder NMT systems trained on over 22 million bitexts. However, this approach largely underperforms on surface-level metrics like BLEU and ChrF. We further control the data quality using the COMET-based quality estimation method. Our experiments show that 1) filtering low COMET scores largely improves encoder-decoder systems, but 2) no clear gains are observed for LLMs when further refining the fine-tuning set. Finally, we show that combining NMT systems with LLMs via post-editing generally yields the best performance for the WMT24 official test set.
Tower v2: Unbabel-IST 2024 Submission for the General MT Shared Task
Authors: Ricardo Rei; Jose Pombal; Nuno M. Guerreiro; João Alves; Pedro Henrique Martins; Patrick Fernandes; Helena Wu; Tania Vaz; Duarte Alves; Amin Farajian; Sweta Agrawal; Antonio Farinhas; José G. C. De Souza; André Martins
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: In this work, we present Tower v2, an improved iteration of the state-of-the-art open-weight Tower models, and the backbone of our submission to the WMT24 General Translation shared task. Tower v2 introduces key improvements including expanded language coverage, enhanced data quality, and increased model capacity up to 70B parameters. Our final submission combines these advancements with quality-aware decoding strategies, selecting translations based on multiple translation quality signals. The resulting system demonstrates significant improvement over previous versions, outperforming closed commercial systems like GPT-4o, Claude 3.5, and DeepL even at a smaller 7B scale.
TSU HITS’s Submissions to the WMT 2024 General Machine Translation Shared Task
Authors: Vladimir Mynka; Nikolay Mikhaylovskiy
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper describes the TSU HITS team’s submission system for the WMT’24 general translation task. We focused on exploring the capabilities of discrete diffusion models for the English-to-{Russian, German, Czech, Spanish} translation tasks in the constrained track. Our submission system consists of a set of discrete diffusion models for each language pair. The main advance is using a separate length regression model to determine the length of the output sequence more precisely.
Document-level Translation with LLM Reranking: Team-J at WMT 2024 General Translation Task
Authors: Keito Kudo; Hiroyuki Deguchi; Makoto Morishita; Ryo Fujii; Takumi Ito; Shintaro Ozaki; Koki Natsumi; Kai Sato; Kazuki Yano; Ryosuke Takahashi; Subaru Kimura; Tomomasa Hara; Yusuke Sakai; Jun Suzuki
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: We participated in the constrained track for English-Japanese and Japanese-Chinese translations at the WMT 2024 General Machine Translation Task. Our approach was to generate a large number of sentence-level translation candidates and select the most probable translation using minimum Bayes risk (MBR) decoding and document-level large language model (LLM) re-ranking. We first generated hundreds of translation candidates from multiple translation models and retained the top 30 candidates using MBR decoding. In addition, we continually pre-trained LLMs on the target language corpora to leverage document-level information. We utilized LLMs to select the most probable sentence sequentially in context from the beginning of the document.
DLUT and GTCOM’s Neural Machine Translation Systems for WMT24
Authors: Hao Zong; Chao Bei; Huan Liu; Conghu Yuan; Wentao Chen; Degen Huang
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the submission from Global Tone Communication Co., Ltd. and Dalian University of Technology for the WMT24 shared general Machine Translation (MT) task at the Conference on Empirical Methods in Natural Language Processing (EMNLP). Our participation encompasses two language pairs: English to Japanese and Japanese to Chinese. The systems are developed without particular constraints or requirements, facilitating extensive research in machine translation. We emphasize back-translation, utilize multilingual translation models, and apply fine-tuning strategies to improve performance. Additionally, we integrate both human-generated and machine-generated data to fine-tune our models, leading to enhanced translation accuracy. The automatic evaluation results indicate that our system ranks first in terms of BLEU score for the Japanese to Chinese translation.
CUNI at WMT24 General Translation Task: LLMs, (Q)LoRA, CPO and Model Merging
Authors: Miroslav Hrabal; Josef Jon; Martin Popel; Nam Luu; Danil Semin; Ondřej Bojar
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the contributions of Charles University teams to the WMT24 General Translation task (English to Czech, German and Russian, and Czech to Ukrainian), and the WMT24 Translation into Low-Resource Languages of Spain task.Our most elaborate submission, CUNI-MH for en2cs, is the result of fine-tuning Mistral 7B v0.1 for translation using a three-stage process: Supervised fine-tuning using QLoRA, Contrastive Preference Optimization, and merging of model checkpoints. We also describe the CUNI-GA, CUNI-Transformer and CUNI-DocTransformer submissions, which are based on our systems from the previous year.Our en2ru system CUNI-DS uses a similar first stage as CUNI-MH (QLoRA for en2cs) and follows with transferring to en2ru.For en2de (CUNI-NL), we experimented with a LLM-based speech translation system, to translate without the speech input.For the Translation into Low-Resource Languages of Spain task, we performed QLoRA fine-tuning of a large LLM on a small amount of synthetic (backtranslated) data.
From General LLM to Translation: How We Dramatically Improve Translation Quality Using Human Evaluation Data for LLM Finetuning
Authors: Denis Elshin; Nikolay Karpachev; Boris Gruzdev; Ilya Golovanov; Georgy Ivanov; Alexander Antonov; Nickolay Skachkov; Ekaterina Latypova; Vladimir Layner; Ekaterina Enikeeva; Dmitry Popov; Anton Chekashev; Vladislav Negodin; Vera Frantsuzova; Alexander Chernyshev; Kirill Denisov
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: In this paper, we present the methodology employed by the NLP team at Yandex LLC for participating in the WMT 2024 General MT Translation track, focusing on English-to-Russian translation. Our approach involves training a YandexGPT LLM-based model for translation tasks using a multi-stage process to ensure high-quality and contextually accurate translations.Initially, we utilize a pre-trained model, trained on a large corpus of high-quality monolingual texts in various languages, crawled from various open sources, not limited to English and Russian. This extensive pre-training allows the model to capture a broad spectrum of linguistic nuances and structures. Following this, the model is fine-tuned on a substantial parallel corpus of high-quality texts collected from diverse open sources, including websites, books, and subtitles. These texts are meticulously aligned at both the sentence and paragraph levels to enhance the model’s contextual understanding and translation accuracy.In the subsequent stage, we employ p-tuning on an internal high-quality corpus of paragraph-aligned data. This step ensures that the model is finely adjusted to handle complex paragraph-level translations with greater fluency and coherence.Next, we apply the Contrastive Pretraining Objective (CPO) method, as described in the paper CPO, using a human-annotated translation corpus. This stage focuses on refining the model’s performance based on metrics evaluated at the paragraph level, emphasizing both the accuracy of the translation and the fluency of the resulting texts. The CPO method helps the model to better distinguish between subtle contextual differences, thereby improving translation quality.In the final stage, we address the importance of preserving the content structure in translations, which is crucial for the General MT test set. To achieve this, we introduce a synthetic corpus based on web pages and video subtitles, and use it during HE markup finetune training. This encourages the model to maintain the original text’s tag structure. This step ensures that the translated output retains the structural integrity of the source web pages, providing a seamless user experience.Our multi-stage approach, combining extensive pre-training, targeted fine-tuning, advanced p-tuning, and structure-preserving techniques, ensures that our model delivers high-quality, fluent, and structurally consistent translations suitable for practical applications and competitive benchmarks.
Cogs in a Machine, Doing What They’re Meant to Do – the AMI Submission to the WMT24 General Translation Task
Authors: Atli Jasonarson; Hinrik Hafsteinsson; Bjarki Ármannsson; Steinþór Steingrímsson
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the submission of the Arni Magnusson Institute’s team to the WMT24 General translation task. We work on the English→Icelandic translation direction. Our system comprises four translation models and a grammar correction model. For training our systems we carefully curate our datasets, aggressively filtering out sentence pairs that may detrimentally affect the quality of our systems output. Some of our data are collected from human translations and some are synthetically generated. A part of the synthetic data is generated using an LLM, and we find that it increases the translation capability of our system significantly.
IKUN for WMT24 General MT Task: LLMs Are Here for Multilingual Machine Translation
Authors: Baohao Liao; Christian Herold; Shahram Khadivi; Christof Monz
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper introduces two multilingual systems, IKUN and IKUN-C, developed for the general machine translation task in WMT24. IKUN and IKUN-C represent an open system and a constrained system, respectively, built on Llama-3-8b and Mistral-7B-v0.3. Both systems are designed to handle all 11 language directions using a single model. According to automatic evaluation metrics, IKUN-C achieved 6 first-place and 3 second-place finishes among all constrained systems, while IKUN secured 1 first-place and 2 second-place finishes across both open and constrained systems. These encouraging results suggest that large language models (LLMs) are nearing the level of proficiency required for effective multilingual machine translation. The systems are based on a two-stage approach: first, continuous pre-training on monolingual data in 10 languages, followed by fine-tuning on high-quality parallel data for 11 language directions. The primary difference between IKUN and IKUN-C lies in their monolingual pre-training strategy. IKUN-C is pre-trained using constrained monolingual data, whereas IKUN leverages monolingual data from the OSCAR dataset. In the second phase, both systems are fine-tuned on parallel data sourced from NTREX, Flores, and WMT16-23 for all 11 language pairs.
NTTSU at WMT2024 General Translation Task
Authors: Minato Kondo; Ryo Fukuda; Xiaotian Wang; Katsuki Chousa; Masato Nishimura; Kosei Buma; Takatomo Kano; Takehito Utsuro
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: The NTTSU team’s submission leverages several large language models developed through a training procedure that includes continual pre-training and supervised fine-tuning. For paragraph-level translation, we generated synthetic paragraph-aligned data and utilized this data for training.In the task of translating Japanese to Chinese, we particularly focused on the speech domain translation. Specifically, we built Whisper models for Japanese automatic speech recognition (ASR). We used YODAS dataset for Whisper training. Since this data contained many noisy data pairs, we combined the Whisper outputs using ROVER for polishing the transcriptions. Furthermore, to enhance the robustness of the translation model against errors in the transcriptions, we performed data augmentation by forward translation from audio, using both ASR and base translation models.To select the best translation from multiple hypotheses of the models, we applied Minimum Bayes Risk decoding + reranking, incorporating scores such as COMET-QE, COMET, and cosine similarity by LaBSE.
SCIR-MT’s Submission for WMT24 General Machine Translation Task
Authors: Baohang Li; Zekai Ye; Yichong Huang; Xiaocheng Feng; Bing Qin
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper introduces the submission of SCIR research center of Harbin Institute of Technology participating in the WMT24 machine translation evaluation task of constrained track for English to Czech. Our approach involved a rigorous process of cleaning and deduplicating both monolingual and bilingual data, followed by a three-stage model training recipe. During the testing phase, we used the beam serach decoding method to generate a large number of candidate translations. Furthermore, we employed COMET-MBR decoding to identify optimal translations.
AIST AIRC Systems for the WMT 2024 Shared Tasks
Authors: Matiss Rikters; Makoto Miwa
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: At WMT 2024 AIST AIRC participated in the General Machine Translation shared task and the Biomedical Translation task. We trained constrained track models for translation between English, German, and Japanese. Before training the final models, we first filtered the parallel data, then performed iterative back-translation as well as parallel data distillation. We experimented with training baseline Transformer models, Mega models, and fine-tuning open-source T5 and Gemma model checkpoints using the filtered parallel data. Our primary submissions contain translations from ensembles of two Mega model checkpoints and our contrastive submissions are generated by our fine-tuned T5 model checkpoints.
Occiglot at WMT24: European Open-source Large Language Models Evaluated on Translation
Authors: Eleftherios Avramidis; Annika Grützner-Zahn; Manuel Brack; Patrick Schramowski; Pedro Ortiz Suarez; Malte Ostendorff; Fabio Barth; Shushen Manakhimova; Vivien Macketanz; Georg Rehm; Kristian Kersting
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This document describes the submission of the very first version of the Occiglot open-source large language model to the General MT Shared Task of the 9th Conference of Machine Translation (WMT24). Occiglot is an open-source, community-based LLM based on Mistral-7B, which went through language-specific continual pre-training and subsequent instruction tuning, including instructions relevant to machine translation.We examine the automatic metric scores for translating the WMT24 test set and provide a detailed linguistically-motivated analysis.Despite Occiglot performing worse than many of the other system submissions, we observe that it performs better than Mistral7B, which has been based upon, which indicates the positive effect of the language specific continual-pretraining and instruction tuning. We see the submission of this very early version of the model as a motivation to unite community forces and pursue future LLM research on the translation task.