GenAI4ELab/papercli-papers huggingface dataset:
Correcting FLORES Evaluation Dataset for Four African Languages
Authors: Idris Abdulmumin; Sthembiso Mkhwanazi; Mahlatse Mbooi; Shamsuddeen Hassan Muhammad; Ibrahim Said Ahmad; Neo Putini; Miehleketo Mathebula; Matimba Shingange; Tajuddeen Gwadabe; Vukosi Marivate
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper describes the corrections made to the FLORES evaluation (dev and devtest) dataset for four African languages, namely Hausa, Northern Sotho (Sepedi), Xitsonga, and isiZulu. The original dataset, though groundbreaking in its coverage of low-resource languages, exhibited various inconsistencies and inaccuracies in the reviewed languages that could potentially hinder the integrity of the evaluation of downstream tasks in natural language processing (NLP), especially machine translation. Through a meticulous review process by native speakers, several corrections were identified and implemented, improving the dataset’s overall quality and reliability. For each language, we provide a concise summary of the errors encountered and corrected and also present some statistical analysis that measures the difference between the existing and corrected datasets. We believe that our corrections enhance the linguistic accuracy and reliability of the data and, thereby, contribute to a more effective evaluation of NLP tasks involving the four African languages. Finally, we recommend that future translation efforts, particularly in low-resource languages, prioritize the active involvement of native speakers at every stage of the process to ensure linguistic accuracy and cultural relevance.
Expanding FLORES+ Benchmark for More Low-Resource Settings: Portuguese-Emakhuwa Machine Translation Evaluation
Authors: Felermino Dario Mario Ali; Henrique Lopes Cardoso; Rui Sousa-Silva
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: As part of the Open Language Data Initiative shared tasks, we have expanded the FLORES+ evaluation set to include Emakhuwa, a low-resource language widely spoken in Mozambique. We translated the dev and devtest sets from Portuguese into Emakhuwa, and we detail the translation process and quality assurance measures used. Our methodology involved various quality checks, including post-editing and adequacy assessments. The resulting datasets consist of multiple reference sentences for each source. We present baseline results from training a Neural Machine Translation system and fine-tuning existing multilingual translation models. Our findings suggest that spelling inconsistencies remain a challenge in Emakhuwa. Additionally, the baseline models underperformed on this evaluation set, underscoring the necessity for further research to enhance machine translation quality for Emakhuwa.The data is publicly available at https://huggingface.co/datasets/LIACC/Emakhuwa-FLORES
Enhancing Tuvan Language Resources through the FLORES Dataset
Authors: Ali Kuzhuget; Airana Mongush; Nachyn-Enkhedorzhu Oorzhak
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: FLORES is a benchmark dataset designed for evaluating machine translation systems, partic- ularly for low-resource languages. This paper, conducted as a part of Open Language Data Ini- tiative (OLDI) shared task, presents our contri- bution to expanding the FLORES dataset with high-quality translations from Russian to Tu- van, an endangered Turkic language. Our ap- proach combined the linguistic expertise of na- tive speakers to ensure both accuracy and cul- tural relevance in the translations. This project represents a significant step forward in support- ing Tuvan as a low-resource language in the realm of natural language processing (NLP) and machine translation (MT).
Machine Translation Evaluation Benchmark for Wu Chinese: Workflow and Analysis
Authors: Hongjian Yu; Yiming Shi; Zherui Zhou; Christopher Haberland
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: We introduce a FLORES+ dataset as an evaluation benchmark for modern Wu Chinese machine translation models and showcase its compatibility with existing Wu data. Wu Chinese is mutually unintelligible with other Sinitic languages such as Mandarin and Yue (Cantonese), but uses a set of Hanzi (Chinese characters) that profoundly overlaps with others. The population of Wu speakers is the second largest among languages in China, but the language has been suffering from significant drop in usage especially among the younger generations. We identify Wu Chinese as a textually low-resource language and address challenges for its machine translation models. Our contributions include: (1) an open-source, manually translated dataset, (2) full documentations on the process of dataset creation and validation experiments, (3) preliminary tools for Wu Chinese normalization and segmentation, and (4) benefits and limitations of our dataset, as well as implications to other low-resource languages.
Open Language Data Initiative: Advancing Low-Resource Machine Translation for Karakalpak
Authors: Mukhammadsaid Mamasaidov; Abror Shopulatov
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This study presents several contributions for the Karakalpak language: a FLORES+ devtest dataset translated to Karakalpak, parallel corpora for Uzbek-Karakalpak, Russian-Karakalpak and English-Karakalpak of 100,000 pairs each and open-sourced fine-tuned neural models for translation across these languages. Our experiments compare different model variants and training approaches, demonstrating improvements over existing baselines. This work, conducted as part of the Open Language Data Initiative (OLDI) shared task, aims to advance machine translation capabilities for Karakalpak and contribute to expanding linguistic diversity in NLP technologies.
FLORES+ Translation and Machine Translation Evaluation for the Erzya Language
Authors: Isai Gordeev; Sergey Kuldin; David Dale
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper introduces a translation of the FLORES+ dataset into the endangered Erzya language, with the goal of evaluating machine translation between this language and any of the other 200 languages already included into FLORES+. This translation was carried out as a part of the Open Language Data shared task at WMT24. We also present a benchmark of existing translation models bases on this dataset and a new translation model that achieves the state-of-the-art quality of translation into Erzya from Russian and English.
Spanish Corpus and Provenance with Computer-Aided Translation for the WMT24 OLDI Shared Task
Authors: Jose Cols
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the Seed-CAT submission to the WMT24 Open Language Data Initiative shared task. We detail our data collection method, which involves a computer-aided translation tool developed explicitly for translating Seed corpora. We release a professionally translated Spanish corpus and a provenance dataset documenting the translation process. The quality of the data was validated on the FLORES+ benchmark with English-Spanish neural machine translation models, achieving an average chrF++ score of 34.9.
Efficient Terminology Integration for LLM-based Translation in Specialized Domains
Authors: Sejoon Kim; Mingi Sung; Jeonghwan Lee; Hyunkuk Lim; Jorge Gimenez Perez
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: Traditional machine translation methods typically involve training models directly on large parallel corpora, with limited emphasis on specialized terminology. However, In specialized fields such as patents, finance, biomedical domains, terminology is crucial for translation, with many terminologies that should not be translated based on semantics of the sentence but should be translated following agreed-upon conventions. In this paper we introduce a methodology that efficiently trains models with a smaller amount of data while preserving the accuracy of terminology translation. The terminology extraction model generates a glossary from existing training datasets and further refines the LLM by instructing it to effectively incorporate these terms into translations. We achieve this through a systematic process of term extraction and glossary creation using the Trie Tree algorithm, followed by data reconstruction to teach the LLM how to integrate these specialized terms. This methodology enhances the model’s ability to handle specialized terminology and ensures high-quality translations, particularly in fields where term consistency is crucial. Our approach has demonstrated exceptional performance, achieving the highest translation score among participants in the WMT patent task to date, showcasing its effectiveness and broad applicability in specialized translation domains where general methods often fall short.
Rakuten’s Participation in WMT 2024 Patent Translation Task
Authors: Ohnmar Htun; Alberto Poncelas
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper introduces our machine transla- tion system (team sakura), developed for the 2024 WMT Patent Translation Task. Our sys- tem focuses on translations between Japanese- English, Japanese-Korean, and Japanese- Chinese. As large language models have shown good results for various natural language pro- cessing tasks, we have adopted the RakutenAI- 7B-chat model, which has demonstrated effec- tiveness in English and Japanese. We fine-tune this model with patent-domain parallel texts and translate using multiple prompts.
The SETU-ADAPT Submission for WMT 24 Biomedical Shared Task
Authors: Antonio Castaldo; Maria Zafar; Prashanth Nayak; Rejwanul Haque; Andy Way; Johanna Monti
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This system description paper presents SETU-ADAPT’s submission to the WMT 2024 Biomedical Shared Task, where we participated for the language pairs English-to-French and English-to-German. Our approach focused on fine-tuning Large Language Models, using in-domain and synthetic data, employing different data augmentation and data retrieval strategies. We introduce a novel MT framework, involving three autonomous agents: a Translator Agent, an Evaluator Agent and a Reviewer Agent. We present our findings and report the quality of the outputs.
Findings of WMT 2024 Shared Task on Low-Resource Indic Languages Translation
Authors: Partha Pakray; Santanu Pal; Advaitha Vetagiri; Reddi Krishna; Arnab Kumar Maji; Sandeep Dash; Lenin Laitonjam; Lyngdoh Sarah; Riyanka Manna
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the results of the low-resource Indic language translation task, organized in conjunction with the Ninth Conference on Machine Translation (WMT) 2024. In this edition, participants were challenged to develop machine translation models for four distinct language pairs: English–Assamese, English-Mizo, English-Khasi, and English-Manipuri. The task utilized the enriched IndicNE-Corp1.0 dataset, which includes an extensive collection of parallel and monolingual corpora for northeastern Indic languages. The evaluation was conducted through a comprehensive suite of automatic metrics—BLEU, TER, RIBES, METEOR, and ChrF—supplemented by meticulous human assessment to measure the translation systems’ performance and accuracy. This initiative aims to drive advancements in low-resource machine translation and make a substantial contribution to the growing body of knowledge in this dynamic field.
Findings of WMT 2024’s MultiIndic22MT Shared Task for Machine Translation of 22 Indian Languages
Authors: Raj Dabre; Anoop Kunchukuttan
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the findings of the WMT 2024’s MultiIndic22MT Shared Task, focusing on Machine Translation (MT) of 22 Indian Languages. In this task, we challenged participants with building MT systems which could translate between any or all of 22 Indian languages in the 8th schedule of the Indian constitution and English. For evaluation, we focused on automatic metrics, namely, chrF, chrF++ and BLEU.
Findings of WMT2024 English-to-Low Resource Multimodal Translation Task
Authors: Shantipriya Parida; Ondřej Bojar; Idris Abdulmumin; Shamsuddeen Hassan Muhammad; Ibrahim Said Ahmad
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the results of the English-to-Low Resource Multimodal Translation shared tasks from the Ninth Conference on Machine Translation (WMT2024). This year, 7 teams submitted their translation results for the automatic and human evaluation.
Findings of the WMT 2024 Shared Task Translation into Low-Resource Languages of Spain: Blending Rule-Based and Neural Systems
Authors: Felipe Sánchez-Martínez; Juan Antonio Perez-Ortiz; Aaron Galiano Jimenez; Antoni Oliver
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the results of the Ninth Conference on Machine Translation (WMT24) Shared Task “Translation into Low-Resource Languages of Spain”’. The task focused on the development of machine translation systems for three language pairs: Spanish-Aragonese, Spanish-Aranese, and Spanish-Asturian. 17 teams participated in the shared task with a total of 87 submissions. The baseline system for all language pairs was Apertium, a rule-based machine translation system that still performs competitively well, even in an era dominated by more advanced non-symbolic approaches. We report and discuss the results of the submitted systems, highlighting the strengths of both neural and rule-based approaches.
Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation
Authors: Longyue Wang; Siyou Liu; Chenyang Lyu; Wenxiang Jiao; Xing Wang; Jiahao Xu; Zhaopeng Tu; Yan Gu; Weiyu Chen; Minghao Wu; Liting Zhou; Philipp Koehn; Andy Way; Yulin Yuan
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: Translating literary works has perennially stood as an elusive dream in machine translation (MT), a journey steeped in intricate challenges. To foster progress in this domain, we hold a new shared task at WMT 2023, the second edition of the Discourse-Level Literary Translation. First, we (Tencent AI Lab and China Literature Ltd.) release a copyrighted and document-level Chinese-English web novel corpus. Furthermore, we put forth an industry-endorsed criteria to guide human evaluation process. This year, we totally received 10 submissions from 5 academia and industry teams. We employ both automatic and human evaluations to measure the performance of the submitted systems. The official ranking of the systems is based on the overall human judgments. In addition, our extensive analysis reveals a series of interesting findings on literary and discourse-aware MT. We release data, system outputs, and leaderboard at https://www2.statmt.org/wmt24/literary-translation-task.html.
Findings of the WMT 2024 Shared Task on Chat Translation
Authors: Wafaa Mohammed; Sweta Agrawal; Amin Farajian; Vera Cabarrão; Bryan Eikema; Ana C Farinha; José G. C. De Souza
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the findings from the third edition of the Chat Translation Shared Task. As with previous editions, the task involved translating bilingual customer support conversations, specifically focusing on the impact of conversation context in translation quality and evaluation. We also include two new language pairs: English-Korean and English-Dutch, in addition to the set of language pairs from previous editions: English-German, English-French, and English-Brazilian Portuguese.We received 22 primary submissions and 32 contrastive submissions from eight teams, with each language pair having participation from at least three teams. We evaluated the systems comprehensively using both automatic metrics and human judgments via a direct assessment framework.The official rankings for each language pair were determined based on human evaluation scores, considering performance in both translation directions—agent and customer. Our analysis shows that while the systems excelled at translating individual turns, there is room for improvement in overall conversation-level translation quality.
Findings of the WMT 2024 Shared Task on Non-Repetitive Translation
Authors: Kazutaka Kinugawa; Hideya Mino; Isao Goto; Naoto Shirai
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: The repetition of words in an English sentence can create a monotonous or awkward impression. In such cases, repetition should be avoided appropriately. To evaluate the performance of machine translation (MT) systems in avoiding such repetition and outputting more polished translations, we presented the shared task of controlling the lexical choice of MT systems. From Japanese–English parallel news articles, we collected several hundred sentence pairs in which the source sentences containing repeated words were translated in a style that avoided repetition. Participants were required to encourage the MT system to output tokens in a non-repetitive manner while maintaining translation quality. We conducted human and automatic evaluations of systems submitted by two teams based on an encoder-decoder Transformer and a large language model, respectively. From the experimental results and analysis, we report a series of findings on this task.
A3-108 Controlling Token Generation in Low Resource Machine Translation Systems
Authors: Saumitra Yadav; Ananya Mukherjee; Manish Shrivastava
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: Translating for languages with limited resources poses a persistent challenge due to the scarcity of high-quality training data. To enhance translation accuracy, we explored controlled generation mechanisms, focusing on the importance of control tokens. In our experiments, while training, we encoded the target sentence length as a control token to the source sentence, treating it as an additional feature for the source sentence. We developed various NMT models using transformer architecture and conducted experiments across 8 language directions (English = Assamese, Manipuri, Khasi, and Mizo), exploring four variations of length encoding mechanisms. Through comparative analysis against the baseline model, we submitted two systems for each language direction. We report our findings for the same in this work.
Samsung R&D Institute Philippines @ WMT 2024 Indic MT Task
Authors: Matthew Theodore Roque; Carlos Rafael Catalan; Dan John Velasco; Manuel Antonio Rufino; Jan Christian Blaise Cruz
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper presents the methodology developed by the Samsung R&D Institute Philippines (SRPH) Language Intelligence Team (LIT) for the WMT 2024 Shared Task on Low-Resource Indic Language Translation. We trained standard sequence-to-sequence Transformer models from scratch for both English-to-Indic and Indic-to-English translation directions. Additionally, we explored data augmentation through backtranslation and the application of noisy channel reranking to improve translation quality. A multilingual model trained across all language pairs was also investigated. Our results demonstrate the effectiveness of the multilingual model, with significant performance improvements observed in most language pairs, highlighting the potential of shared language representations in low-resource translation scenarios.
DLUT-NLP Machine Translation Systems for WMT24 Low-Resource Indic Language Translation
Authors: Chenfei Ju; Junpeng Liu; Kaiyu Huang; Degen Huang
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: This paper describes the submission systems of DLUT-NLP team for the WMT24 low-resource Indic language translation shared task. We participated in the translation task of four language pairs, including en-as, en-mz, en-kha, en-mni.
SRIB-NMT’s Submission to the Indic MT Shared Task in WMT 2024
Authors: Pranamya Patil; Raghavendra Hr; Aditya Raghuwanshi; Kushal Verma
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: In the context of the Indic Low Resource Ma-chine Translation (MT) challenge at WMT-24, we participated in four language pairs:English-Assamese (en-as), English-Mizo (en-mz), English-Khasi (en-kh), and English-Manipuri (en-mn). To address these tasks,we employed a transformer-based sequence-to-sequence architecture (Vaswani et al., 2017).In the PRIMARY system, which did not uti-lize external data, we first pretrained languagemodels (low resource languages) using avail-able monolingual data before finetuning themon small parallel datasets for translation. Forthe CONTRASTIVE submission approach, weutilized pretrained translation models like In-dic Trans2 (Gala et al., 2023) and appliedLoRA Fine-tuning (Hu et al., 2021) to adaptthem to smaller, low-resource languages, aim-ing to leverage cross-lingual language transfercapabilities (CONNEAU and Lample, 2019).These approaches resulted in significant im-provements in SacreBLEU scores(Post, 2018)for low-resource languages.
MTNLP-IIITH: Machine Translation for Low-Resource Indic Languages
Authors: Abhinav P M; Ketaki Shetye; Parameswari Krishnamurthy
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: Machine Translation for low-resource languages presents significant challenges, primarily due to limited data availability. We have a baseline model and a primary model. For the baseline model, we first fine-tune the mBART model (mbart-large-50-many-to-many-mmt) for the language pairs English-Khasi, Khasi-English, English-Manipuri, and Manipuri-English. We then augment the dataset by back-translating from Indic languages to English. To enhance data quality, we fine-tune the LaBSE model specifically for Khasi and Manipuri, generating sentence embeddings and applying a cosine similarity threshold of 0.84 to filter out low-quality back-translations. The filtered data is combined with the original training data and used to further fine-tune the mBART model, creating our primary model. The results show that the primary model slightly outperforms the baseline model, with the best performance achieved by the English-to-Khasi (en-kh) primary model, which recorded a BLEU score of 0.0492, a chrF score of 0.3316, and a METEOR score of 0.2589 (on a scale of 0 to 1), with similar results for other language pairs.
Exploration of the CycleGN Framework for Low-Resource Languages
Authors: Sören Dreano; Derek Molloy; Noel Murphy
Venue: EMNLP 2024 (Proceedings of the Ninth Conference on Machine Translation)

Abstract: CycleGN is a Neural Machine Translation framework relying on the Transformer architecture. The foundational concept of our research posits that in an ideal scenario, retro-translations of generated translations should revert to the original source sentences. Consequently, a pair of models can be trained using a Cycle Consistency Loss only, with one model translating in one direction and the second model in the opposite direction.