GenAI4ELab/papercli-papers huggingface dataset:
Robustness of Fine-Tuned Models for Machine Translation with Varying Noise Levels: Insights for Asturian, Aragonese and Aranese
Authors: Martin Bär; Elisa Forcada Rodríguez; María García-Abadillo Velasco
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: We present the LCT-LAP proposal for the shared task on Translation into Low-Resource Languages of Spain at WMT24 within the constrained submission category. Our work harnesses encoder-decoder models pretrained on higher-resource Iberian languages to facilitate MT model training for Asturian, Aranese and Aragonese. Furthermore, we explore the robustness of these models when fine-tuned on datasets with varying levels of alignment noise. We fine-tuned a Spanish-Galician model using Asturian data filtered by BLEU score thresholds of 5, 15, 30 and 60, identifying BLEU 15 as the most effective. This threshold was then applied to the Aranese and Aragonese datasets. Our findings indicate that filtering the corpora reduces computational costs and improves performance compared to using nearly raw data or data filtered with language identification. However, it still falls short of the performance achieved by the rule-based system Apertium in Aranese and Aragonese.
Training and Fine-Tuning NMT Models for Low-Resource Languages Using Apertium-Based Synthetic Corpora
Authors: Aleix Sant; Daniel Bardanca; José Ramom Pichel Campos; Francesca De Luca Fornaciari; Carlos Escolano; Javier Garcia Gilabert; Pablo Gamallo; Audrey Mash; Xixian Liao; Maite Melero
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: In this paper, we present the two strategies employed for the WMT24 Shared Task on Translation into Low-Resource Languages of Spain. We participated in the language pairs of Spanish-to-Aragonese, Spanish-to-Aranese, and Spanish-to-Asturian, developing neural-based translation systems and moving away from rule-based approaches for these language directions. To create these models, two distinct strategies were employed. The first strategy involved a thorough cleaning process and curation of the limited provided data, followed by fine-tuning the multilingual NLLB-200-600M model (Constrained Submission). The other strategy involved training a transformer from scratch using a vast amount of synthetic data (Open Submission). Both approaches relied on generated synthetic data and resulted in high ChrF and BLEU scores. However, given the characteristics of the task, the strategy used in the Constrained Submission resulted in higher scores that surpassed the baselines across the three translation directions, whereas the strategy employed in the Open Submission yielded slightly lower scores than the highest baseline.
Vicomtech@WMT 2024: Shared Task on Translation into Low-Resource Languages of Spain
Authors: David Ponce; Harritxu Gete; Thierry Etchegoyhen
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: We describe Vicomtech’s participation in the WMT 2024 Shared Task on translation into low-resource languages of Spain. We addressed all three languages of the task, namely Aragonese, Aranese and Asturian, in both constrained and open settings. Our work mainly centred on exploiting different types of corpora via data filtering, selection and combination methods, along with synthetic data generated with translation models based on rules, neural sequence-to-sequence or large language models. We improved or matched the best baselines in all three language pairs and present complementary results on additional test sets.
SJTU System Description for the WMT24 Low-Resource Languages of Spain Task
Authors: Tianxiang Hu; Haoxiang Sun; Ruize Gao; Jialong Tang; Pei Zhang; Baosong Yang; Rui Wang
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: We participate in the translation task on Spanish to Aragonese, Spanish to Aranese and Spanish to Asturian. Initially, we conduct preliminary experiments to assess the basic translation capabilities of various models and evaluate the impact of fine-tuning with different data types. We then choose to fine-tune the Qwen2-0.5B model using a forward synthesized pseudo-corpus from the Apertium translation system to replicate its fundamental performance. Building on this distillation model, we explore three optimization strategies across the three language directions: (1) Assembling the provided FLORES+ dev sets into a 5-shot format translation training dataset and performing few-shot fine-tuning to enhance model performance. (2) Utilizing the FLORES+ dev sets as training data and applying the Contrastive Preference Optimization (CPO) strategy for further refinement. (3) Retrieving the 20 most similar translation examples from the FLORES+ dev sets using the BM25 algorithm and performing 20-shot translations with the Claude 3.5-sonnet model. After evaluating these strategies, we select the best-performing approach for each language pair as our submission result.
Multilingual Transfer and Domain Adaptation for Low-Resource Languages of Spain
Authors: Yuanchang Luo; Zhanglin Wu; Daimeng Wei; Hengchao Shang; Zongyao Li; Jiaxin Guo; Zhiqiang Rao; Shaojun Li; Jinlong Yang; Yuhao Xie; Zheng Jiawei; Bin Wei; Hao Yang
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This article introduces the submission status of the Translation into Low-Resource Languages of Spain task at (WMT 2024) by Huawei Translation Service Center (HW-TSC). We participated in three translation tasks: spanish to aragonese (es2arg), spanish to aranese (es2arn), and spanish to asturian (es2ast). For these three translation tasks, we use training strategies such as multilingual transfer, regularized dropout, forward translation and back translation, labse denoising, transduction ensemble learning and other strategies to neural machine translation (NMT) model based on training deep transformer-big architecture. By using these enhancement strategies, our submission achieved a competitive result in the final evaluation.
TRIBBLE - TRanslating IBerian languages Based on Limited E-resources
Authors: Igor Kuzmin; Piotr Przybyła; Euan Mcgill; Horacio Saggion
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: In this short overview paper, we describe our system submission for the language pairs Spanish to Aragonese (spa-arg), Spanish to Aranese (spa-arn), and Spanish to Asturian (spa-ast). We train a unified model for all language pairs in the constrained scenario. In addition, we add two language control tokens for Aragonese and Aranese Occitan, as there is already one present for Asturian. We take the distilled NLLB-200 model with 600M parameters and extend special tokens with 2 tokens that denote target languages (arn_Latn, arg_Latn) because Asturian was already presented in NLLB-200 model. We adapt the model by training on a special regime of data augmentation with both monolingual and bilingual training data for the language pairs in this challenge.
CloudSheep System for WMT24 Discourse-Level Literary Translation
Authors: Lisa Liu; Ryan Liu; Angela Tsai; Jingbo Shang
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper describes the CloudSheep translation system for WMT24 Discourse-Level Literary Translation shared task. We participated in the Chinese-English direction on the unconstrained track. Our approach to the task used a pipeline of different tools in order to maximize the translation accuracy and flow of the text by combining the strengths of each tool. In particular, our focus was to translate names consistently and idioms correctly. To achieve consistent names throughout a text, a custom name dictionary was generated for each text, containing person and place names, along with their translations. A common honorific dictionary was applied for consistency with titles, especially in historical or cultivation novels. The names were found and translated with GPT 3.5-turbo. To achieve accurate and concise translations of idioms, which are often translated literally and verbosely, we integrated the CC-CEDICT library to provide official definitions. Then, we used GPT-4 to pick the best dictionary definition that fit the context and rephrase it to fit grammatically within a sentence. For the translation of non-name and non-idiom terms, we used Google Translate. We compared our approach’s performance with Google Translate as a baseline using BLEU, chrF, and COMET, as well as A/B testing.
Final Submission of SJTULoveFiction to Literary Task
Authors: Haoxiang Sun; Tianxiang Hu; Ruize Gao; Jialong Tang; Pei Zhang; Baosong Yang; Rui Wang
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper describes Shanghai Jiao Tong University (SJTU LoveFiction) Discourse-Level Literary Translation systems for the WMT24shared task. We participate in the literary translation task on Chinese → English, Chinese →German and Chinese → Russian with uncon-strained tack.Check our paper for detail.
Context-aware and Style-related Incremental Decoding Framework for Discourse-Level Literary Translation
Authors: Yuanchang Luo; Jiaxin Guo; Daimeng Wei; Hengchao Shang; Zongyao Li; Zhanglin Wu; Zhiqiang Rao; Shaojun Li; Jinlong Yang; Hao Yang
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This report outlines our approach for the WMT24 Discourse-Level Literary Translation Task, focusing on the Chinese-English language pair in the Constrained Track. Translating literary texts poses significant challenges due to the nuanced meanings, idiomatic expressions, and intricate narrative structures inherent in such works. To address these challenges, we leveraged the Chinese-Llama2 model, specifically enhanced for this task through a combination of Continual Pre-training (CPT) and Supervised Fine-Tuning (SFT). Our methodology includes a novel Incremental Decoding framework, which ensures that each sentence is translated with consideration of its broader context, maintaining coherence and consistency throughout the text. This approach allows the model to capture long-range dependencies and stylistic elements, producing translations that faithfully preserve the original literary quality. Our experiments demonstrate significant improvements in both sentence-level and document-level BLEU scores, underscoring the effectiveness of our proposed framework in addressing the complexities of document-level literary translation.
NovelTrans: System for WMT24 Discourse-Level Literary Translation
Authors: Yuchen Liu; Yutong Yao; Runzhe Zhan; Yuchu Lin; Derek F. Wong
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper describes our submission system, NovelTrans, from NLP²CT and DeepTranx for the WMT24 Discourse-Level Literary Translation Task in Chinese-English, Chinese-German, and Chinese-Russian language pairs under unconstrained conditions. For our primary system, three translations are done by GPT4o using three different settings of additional information and a terminology table generated by online models. The final result is composed of sentences that have the highest xCOMET score compared with the corresponding sentences in other results. Our system achieved an xCOMET score of 79.14 which is higher than performing a direct chapter-level translation on our dataset.
LinChance-NTU for Unconstrained WMT2024 Literary Translation
Authors: Kechen Li; Yaotian Tao; Hongyi Huang; Tianbo Ji
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: The rapid growth of deep learning has spurred significant advancements across industries, par- ticularly in machine translation through large language models (LLMs). However, translat- ing literary still presents challenges, including cross-cultural nuances, complex language struc- tures, metaphorical expressions, and cultural differences. To address these issues, this study utilizes the Llama and Phi models using both LoRA and full-parameter techniques, along-side a prompt-based translation system. Full-parameter tuning of the Llama-3-Chinese-8B-Instruct model was unsuccessful due to mem-ory constraints. In terms of the WMT task, the fully fine-tuned Phi 3 model was selected for submission due to its more natural and flu-ent translations. Nonetheless, results showed that LoRA and the prompt-based system sig- nificantly improved the Llama3 model’s perfor- mance, surpassing other models in BLEU and ROUGE evaluations.
Improving Context Usage for Translating Bilingual Customer Support Chat with Large Language Models
Authors: Jose Pombal; Sweta Agrawal; André Martins
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper describes Unbabel+IT’s submission to the Chat Shared Task held at the Workshop of Machine Translation 2024. The task focuses on translating customer support chats between agents and customers communicating in different languages. We present two strategies for adapting state-of-the-art language models to better utilize contextual information when translating such conversations. Our training strategy involves finetuning the model on chat datasets with context-augmented instructions, resulting in a specialized model, TOWERCHAT. For inference, we propose a novel quality-aware decoding approach that leverages a context-aware metric, CONTEXTCOMET, to select the optimal translation from a pool of candidates. We evaluate our proposed approach on the official shared task datasets for ten language pairs, showing that our submission consistently outperforms baselines on all and competing systems on 8 out of 10 language pairs across multiple automated metrics. Remarkably, TOWERCHAT outperforms our contrastive submission based on the much larger TOWER-V2-70B model while being 10× smaller. According to human evaluation, our system outperforms all other systems and baselines across all language pairs. These results underscore the importance of context-aware training and inference in handling complex bilingual dialogues.
Optimising LLM-Driven Machine Translation with Context-Aware Sliding Windows
Authors: Xinye Yang; Yida Mu; Kalina Bontcheva; Xingyi Song
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper describes SheffieldGATE’s submission to WMT 2024 Chat Shared Translation Task. We participate in three language pairs: English-German, English-Dutch, and English-Portuguese (Brazil). In this work, we introduce a context-aware sliding window decoding method to track dependencies between chat messages. We fine-tune a large pre-trained language model based on the training data provided by the shared task Our experiments (i) compare the model performance between multilingual and bilingual fine-tuning and (ii) assess the impact of different window sizes. Our experimental results demonstrate that utilising contextual information yields superior performance in document-level translation compared to translating documents as isolated text segments, and that models fine-tuned with multilingual data perform better than those fine-tuned with bilingual data.
Context-Aware LLM Translation System Using Conversation Summarization and Dialogue History
Authors: Mingi Sung; Seungmin Lee; Jiwon Kim; Sejoon Kim
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: Translating conversational text, particularly in customer support contexts, presents unique challenges due to its informal and unstructured nature. We propose a context-aware LLM translation system that leverages conversation summarization and dialogue history to enhance translation quality for the English-Korean language pair. Our approach incorporates the two most recent dialogues as raw data and a summary of earlier conversations to manage context length effectively. We demonstrate that this method significantly improves translation accuracy, maintaining coherence and consistency across conversations. This system offers a practical solution for customer support translation tasks, addressing the complexities of conversational text.
Enhancing Translation Quality: A Comparative Study of Fine-Tuning and Prompt Engineering in Dialog-Oriented Machine Translation Systems. Insights from the MULTITAN-GML Team
Authors: Lichao Zhu; Maria Zimina; Behnoosh Namdarzadeh; Nicolas Ballier; Jean-Baptiste Yunès
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: For this shared task, we have used several machine translation engines to produce translations (en ⇔ fr) by fine-tuning a dialog-oriented NMT engine and having NMT baseline translations post-edited with prompt engineering. Our objectives are to test the effectiveness of a fine-tuning strategy with help of a robust NMT model, to draw out a from-translation-to-post-editing pipeline, and to evaluate the strong and weak points of NMT systems.
The SETU-ADAPT Submissions to WMT 2024 Chat Translation Tasks
Authors: Maria Zafar; Antonio Castaldo; Prashanth Nayak; Rejwanul Haque; Andy Way
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the SETU-ADAPT submissions to the WMT24 Chat Translation Task. Large language models (LLM) currently provides the state-of-the-art solutions in many natural language processing (NLP) problems including machine translation (MT). For the WMT24 Chat Translation Task we leveraged LLMs for their MT capabilities. In order to adapt the LLMs for a specific domain of interest, we explored different fine-tuning and prompting strategies. We also employed efficient data retrieval methods to curate the data used for fine-tuning. We carried out experiments for two language pairs: German-to-English and French-to-English. Our MT models were evaluated using three metrics: BLEU, chrF and COMET. In this paper we describes our experiments including training setups, results and findings.
Exploring the Traditional NMT Model and Large Language Model for Chat Translation
Authors: Jinlong Yang; Hengchao Shang; Daimeng Wei; Jiaxin Guo; Zongyao Li; Zhanglin Wu; Zhiqiang Rao; Shaojun Li; Yuhao Xie; Yuanchang Luo; Zheng Jiawei; Bin Wei; Hao Yang
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper describes the submissions of Huawei Translation Services Center(HW-TSC) to WMT24 chat translation shared task on English↔Germany (en-de) bidirection. The experiments involved fine-tuning models using chat data and exploring various strategies, including Minimum Bayesian Risk (MBR) decoding and self-training. The results show significant performance improvements in certain directions, with the MBR self-training method achieving the best results. The Large Language Model also discusses the challenges and potential avenues for further research in the field of chat translation.
Graph Representations for Machine Translation in Dialogue Settings
Authors: Lea Krause; Selene Baez Santamaria; Jan-Christoph Kalo
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: In this paper, we present our approach to the WMT24 - Chat Task, addressing the challenge of translating chat conversations.Chat conversations are characterised by their informal, ungrammatical nature and strong reliance on context posing significant challenges for machine translation systems. To address these challenges, we augment large language models with explicit memory mechanisms designed to enhance coherence and consistency across dialogues. Specifically, we employ graph representations to capture and utilise dialogue context, leveraging concept connectivity as a compressed memory. Our approach ranked second place for Dutch and French, and third place for Portuguese and German, based on COMET-22 scores and human evaluation.
Reducing Redundancy in Japanese-to-English Translation: A Multi-Pipeline Approach for Translating Repeated Elements in Japanese
Authors: Qiao Wang; Yixuan Huang; Zheng Yuan
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents a multi-pipeline Japanese-to-English machine translation (MT) system designed to address the challenge of translating repeated elements from Japanese into fluent and lexically diverse English. The system is developed as part of the Non-Repetitive Translation Task at WMT24, which focuses on minimizing redundancy while maintaining high translation quality. Our approach utilizes MeCab, the de facto NLP tool for Japanese, for the identification of repeated elements, and Claude Sonnet 3.5, a large language model (LLM), for translation and proofreading. The system effectively accomplishes the shared task by identifying and translating in a diversified manner 89.79% of the 470 repeated instances in the testing dataset, and achieving an average translation quality score of 4.60 out of 5, significantly surpassing the baseline score of 3.88. Analysis also revealed the challenges encountered, particularly in identifying standalone noun-suffix elements and occasional cases of consistent translations or mistranslations.
SYSTRAN @ WMT24 Non-Repetitive Translation Task
Authors: Marko Avila; Josep Crego
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: Many contemporary NLP systems rely on neural decoders for text generation, which demonstrate an impressive ability to generate text approaching human fluency levels. However, in the case of neural machine translation networks, they often grapple with the production of repetitive content, also known as repetitive diction or word repetition, an aspect they weren’t explicitly trained to address. While not inherently negative, this repetition can make writing seem monotonous or awkward if not used intentionally for emphasis or stylistic purposes. This paper presents our submission to the WMT 2024 Non-Repetitive Translation Task, for which we adopt a repetition penalty method applied at learning inspired by the principles of label smoothing. No additional work is needed at inference time. We modify the ground-truth distribution to steer the model towards discouraging repetitions. Experiments show the ability of the proposed methods in reducing repetitions within neural machine translation engines, without compromising efficiency or translation quality.
Mitigating Metric Bias in Minimum Bayes Risk Decoding
Authors: Geza Kovacs; Daniel Deutsch; Markus Freitag
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as any improvement might simply be due to reward hacking rather than reflecting real quality improvements. In this work we demonstrate that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.