GenAI4ELab/papercli-papers huggingface dataset:
RoCS-MT: Robustness Challenge Set for Machine Translation
Authors: Rachel Bawden; Benoît Sagot
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: RoCS-MT, a Robust Challenge Set for Machine Translation (MT), is designed to test MT systems’ ability to translate user-generated content (UGC) that displays non-standard characteristics, such as spelling errors, devowelling, acronymisation, etc. RoCS-MT is composed of English comments from Reddit, selected for their non-standard nature, which have been manually normalised and professionally translated into five languages: French, German, Czech, Ukrainian and Russian. In the context of the WMT23 test suite shared task, we analyse the models submitted to the general MT task for all from-English language pairs, offering some insights into the types of problems faced by state-of-the-art MT models when dealing with non-standard UGC texts. We compare automatic metrics for MT quality, including quality estimation to see if the same conclusions can be drawn without references. In terms of robustness, we find that many of the systems struggle with non-standard variants of words (e.g. due to phonetically inspired spellings, contraction, truncations, etc.), but that this depends on the system and the amount of training data, with the best overall systems performing better across all phenomena. GPT4 is the clear front-runner. However we caution against drawing conclusions about generalisation capacity as it and other systems could be trained on the source side of RoCS and also on similar data.
Multifaceted Challenge Set for Evaluating Machine Translation Performance
Authors: Xiaoyu Chen; Daimeng Wei; Zhanglin Wu; Ting Zhu; Hengchao Shang; Zongyao Li; Jiaxin Guo; Ning Xie; Lizhi Lei; Hao Yang; Yanfei Jiang
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: Machine Translation Evaluation is critical to Machine Translation research, as the evaluation results reflect the effectiveness of training strategies. As a result, a fair and efficient evaluation method is necessary. Many researchers have raised questions about currently available evaluation metrics from various perspectives, and propose suggestions accordingly. However, to our knowledge, few researchers has analyzed the difficulty level of source sentence and its influence on evaluation results. This paper presents HW-TSC’s submission to the WMT23 MT Test Suites shared task. We propose a systematic approach for construing challenge sets from four aspects: word difficulty, length difficulty, grammar difficulty and model learning difficulty. We open-source two Multifaceted Challenge Sets for Zh→En and En→Zh. We also present results of participants in this year’s General MT shared task on our test sets.
Linguistically Motivated Evaluation of the 2023 State-of-the-art Machine Translation: Can ChatGPT Outperform NMT?
Authors: Shushen Manakhimova; Eleftherios Avramidis; Vivien Macketanz; Ekaterina Lapshinova-Koltunski; Sergei Bagdasarov; Sebastian Möller
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper offers a fine-grained analysis of the machine translation outputs in the context of the Shared Task at the 8th Conference of Machine Translation (WMT23). Building on the foundation of previous test suite efforts, our analysis includes Large Language Models and an updated test set featuring new linguistic phenomena. To our knowledge, this is the first fine-grained linguistic analysis for the GPT-4 translation outputs. Our evaluation spans German-English, English-German, and English-Russian language directions. Some of the phenomena with the lowest accuracies for German-English are idioms and resultative predicates. For English-German, these include mediopassive voice, and noun formation(er). As for English-Russian, these included idioms and semantic roles. GPT-4 performs equally or comparably to the best systems in German-English and English-German but falls in the second significance cluster for English-Russian.
IIIT HYD’s Submission for WMT23 Test-suite Task
Authors: Ananya Mukherjee; Manish Shrivastava
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper summarizes the results of our test suite evaluation on 12 machine translation systems submitted at the Shared Task of the 8th Conference of Machine Translation (WMT23) for English-German (en-de) language pair. Our test suite covers five specific domains (entertainment, environment, health, science, legal) and spans five distinct writing styles (descriptive, judgments, narrative, reporting, technical-writing). We present our analysis through automatic evaluation methods, conducted with a focus on domain-specific and writing style-specific evaluations.
Test Suites Task: Evaluation of Gender Fairness in MT with MuST-SHE and INES
Authors: Beatrice Savoldi; Marco Gaido; Matteo Negri; Luisa Bentivogli
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: As part of the WMT-2023 “Test suites” shared task, in this paper we summarize the results of two test suites evaluations: MuST-SHEWMT23 and INES. By focusing on the en-de and de-en language pairs, we rely on these newly created test suites to investigate systems’ ability to translate feminine and masculine gender and produce gender-inclusive translations. Furthermore we discuss metrics associated with our test suites and validate them by means of human evaluations. Our results indicate that systems achieve reasonable and comparable performance in correctly translating both feminine and masculine gender forms for naturalistic gender phenomena. Instead, the generation of inclusive language forms in translation emerges as a challenging task for all the evaluated MT models, indicating room for future improvements and research on the topic. We make MuST-SHEWMT23 and INES freely available.
Biomedical Parallel Sentence Retrieval Using Large Language Models
Authors: Sheema Firdous; Sadaf Abdul Rauf
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: We have explored the effect of in domain knowledge during parallel sentence filtering from in domain corpora. Models built with sentences mined from in domain corpora without domain knowledge performed poorly, whereas model performance improved by more than 2.3 BLEU points on average with further domain centric filtering. We have used Large Language Models for selecting similar and domain aligned sentences. Our experiments show the importance of inclusion of domain knowledge in sentence selection methodologies even if the initial comparable corpora are in domain.
The Path to Continuous Domain Adaptation Improvements by HW-TSC for the WMT23 Biomedical Translation Shared Task
Authors: Zhanglin Wu; Daimeng Wei; Zongyao Li; Zhengzhe Yu; Shaojun Li; Xiaoyu Chen; Hengchao Shang; Jiaxin Guo; Yuhao Xie; Lizhi Lei; Hao Yang; Yanfei Jiang
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper presents the domain adaptation methods adopted by Huawei Translation Service Center (HW-TSC) to train the neural machine translation (NMT) system on the English↔German (en↔de) language pair of the WMT23 biomedical translation task. Our NMT system is built on deep Transformer with larger parameter sizes. Based on the biomedical NMT system trained last year, we leverage Curriculum Learning, Data Diversification, Forward translation, Back translation, and Transductive Ensemble Learning to further improve system performance. Overall, we believe our submission can achieve highly competitive result in the official final evaluation.
Investigating Techniques for a Deeper Understanding of Neural Machine Translation (NMT) Systems through Data Filtering and Fine-tuning Strategies
Authors: Lichao Zhu; Maria Zimina; Maud Bénard; Behnoosh Namdar; Nicolas Ballier; Guillaume Wisniewski; Jean-Baptiste Yunès
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: In the context of this biomedical shared task, we have implemented data filters to enhance the selection of relevant training data for fine- tuning from the available training data sources. Specifically, we have employed textometric analysis to detect repetitive segments within the test set, which we have then used for re- fining the training data used to fine-tune the mBart-50 baseline model. Through this approach, we aim to achieve several objectives: developing a practical fine-tuning strategy for training biomedical in-domain fr<>en models, defining criteria for filtering in-domain training data, and comparing model predictions, fine-tuning data in accordance with the test set to gain a deeper insight into the functioning of Neural Machine Translation (NMT) systems.
MAX-ISI System at WMT23 Discourse-Level Literary Translation Task
Authors: Li An; Linghao Jin; Xuezhe Ma
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper describes our translation systems for the WMT23 shared task. We participated in the discourse-level literary translation task - constrained track. In our methodology, we conduct a comparative analysis between the conventional Transformer model and the recently introduced MEGA model, which exhibits enhanced capabilities in modeling long-range sequences compared to the traditional Transformers. To explore whether language models can more effectively harness document-level context using paragraph-level data, we took the approach of aggregating sentences into paragraphs from the original literary dataset provided by the organizers. This paragraph-level data was utilized in both the Transformer and MEGA models. To ensure a fair comparison across all systems, we employed a sentence-alignment strategy to reverse our translation results from the paragraph-level back to the sentence-level alignment. Finally, our evaluation process encompassed sentence-level metrics such as BLEU, as well as two document-level metrics: d-BLEU and BlonDe.
The MAKE-NMTVIZ System Description for the WMT23 Literary Task
Authors: Fabien Lopez; Gabriela González; Damien Hansen; Mariam Nakhle; Behnoosh Namdarzadeh; Nicolas Ballier; Marco Dinarelli; Emmanuelle Esperança-Rodier; Sui He; Sadaf Mohseni; Caroline Rossi; Didier Schwab; Jun Yang; Jean-Baptiste Yunès; Lichao Zhu
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper describes the MAKE-NMTVIZ Systems trained for the WMT 2023 Literary task. As a primary submission, we used Train, Valid1, test1 as part of the GuoFeng corpus (Wang et al., 2023) to fine-tune the mBART50 model with Chinese-English data. We followed very similar training parameters to (Lee et al. 2022) when fine-tuning mBART50. We trained for 3 epochs, using gelu as an activation function, with a learning rate of 0.05, dropout of 0.1 and a batch size of 16. We decoded using a beam search of size 5. For our contrastive1 submission, we implemented a fine-tuned concatenation transformer (Lupo et al., 2023). The training was developed in two steps: (i) a sentence-level transformer was implemented for 10 epochs trained using general, test1, and valid1 data (more details in contrastive2 system); (ii) second, we fine-tuned at document-level using 3-sentence concatenation for 4 epochs using train, test2, and valid2 data. During the fine-tuning, we used ReLU as an activation function, with an inverse square root learning rate, dropout of 0.1, and a batch size of 64. We decoded using a beam search of size. Four our contrastive2 and last submission, we implemented a sentence-level transformer model (Vaswani et al., 2017). The model was trained with general data for 10 epochs using general-purpose, test1, and valid 1 data. The training parameters were an inverse square root scheduled learning rate, a dropout of 0.1, and a batch size of 64. We decoded using a beam search of size 4. We then compared the three translation outputs from an interdisciplinary perspective, investigating some of the effects of sentence- vs document-based training. Computer scientists, translators and corpus linguists discussed the linguistic remaining issues for this discourse-level literary translation.
DUTNLP System for the WMT2023 Discourse-Level Literary Translation
Authors: Anqi Zhao; Kaiyu Huang; Hao Yu; Degen Huang
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper describes the submission of DUTNLP Lab submission to WMT23 Discourse-Level Literary Translation in the Chinese to English translation direction under unconstrained conditions. Our primary system aims to leverage a large language model with various prompt strategies, which can fully investigate the potential capabilities of large language models for discourse-level neural machine translation. Moreover, we test a widely used discourse-level machine translation model, G-transformer, with different training strategies. In our experimental results, the method with large language models achieves a BLEU score of 28.16, while the fine-tuned method scores 25.26. These findings indicate that selecting appropriate prompt strategies based on large language models can significantly improve translation performance compared to traditional model training methods.
HW-TSC’s Submissions to the WMT23 Discourse-Level Literary Translation Shared Task
Authors: Yuhao Xie; Zongyao Li; Zhanglin Wu; Daimeng Wei; Xiaoyu Chen; Zhiqiang Rao; Shaojun Li; Hengchao Shang; Jiaxin Guo; Lizhi Lei; Hao Yang; Yanfei Jiang
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper introduces HW-TSC’s submission to the WMT23 Discourse-Level Literary Translation shared task. We use standard sentence-level transformer as a baseline, and perform domain adaptation and discourse modeling to enhance discourse-level capabilities. Regarding domain adaptation, we employ Back-Translation, Forward-Translation and Data Diversification. For discourse modeling, we apply strategies such as Multi-resolutional Document-to-Document Translation and TrAining Data Augmentation.
TJUNLP:System Description for the WMT23 Literary Task in Chinese to English Translation Direction
Authors: Shaolin Zhu; Deyi Xiong
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper introduces the overall situation of the Natural Language Processing Laboratory of Tianjin University participating in the WMT23 machine translation evaluation task from Chinese to English. For this evaluation, the base model used is a Transformer based on a Mixture of Experts (MOE) model. During the model’s construction and training, a basic dense model based on Transformer is first trained on the training set. Then, this model is used to initialize the MOE-based translation model, which is further trained on the training corpus. Since the training dataset provided for this translation task is relatively small, to better utilize sparse models to enhance translation, we employed a data augmentation technique for alignment. Experimental results show that this method can effectively improve neural machine translation performance.
Machine Translation for Nko: Tools, Corpora, and Baseline Results
Authors: Moussa Doumbouya; Baba Mamadi Diané; Solo Farabado Cissé; Djibrila Diané; Abdoulaye Sow; Séré Moussa Doumbouya; Daouda Bangoura; Fodé Moriba Bayo; Ibrahima Sory Conde; Kalo Mory Diané; Chris Piech; Christopher Manning
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: Currently, there is no usable machine translation system for Nko, a language spoken by tens of millions of people across multiple West African countries, which holds significant cultural and educational value. To address this issue, we present a set of tools, resources, and baseline results aimed towards the development of usable machine translation systems for Nko and other languages that do not currently have sufficiently large parallel text corpora available. (1) Fria∥el: A novel collaborative parallel text curation software that incorporates quality control through copyedit-based workflows. (2) Expansion of the FLoRes-200 and NLLB-Seed corpora with 2,009 and 6,193 high-quality Nko translations in parallel with 204 and 40 other languages. (3) nicolingua-0005: A collection of trilingual and bilingual corpora with 130,850 parallel segments and monolingual corpora containing over 3 million Nko words. (4) Baseline bilingual and multilingual neural machine translation results with the best model scoring 30.83 English-Nko chrF++ on FLoRes-devtest.
TTIC’s Submission to WMT-SLT 23
Authors: Marcelo Sandoval-Castaneda; Yanhong Li; Bowen Shi; Diane Brentari; Karen Livescu; Gregory Shakhnarovich
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: In this paper, we describe TTIC’s submission to WMT 2023 Sign Language Translation task on the Swiss-German Sign Language (DSGS) to German track. Our approach explores the advantages of using large-scale self-supervised pre-training in the task of sign language translation, over more traditional approaches that rely heavily on supervision, along with costly labels such as gloss annotations. The proposed model consists of a VideoSwin transformer for image encoding, and a T5 model adapted to receive VideoSwin features as input instead of text. In WMT-SLT 22’s development set, this system achieves 2.03 BLEU score, a 59% increase over the previous best reported performance. In the official test set, our primary submission achieves 1.1 BLEU score and 17.0 chrF score.
KnowComp Submission for WMT23 Sign Language Translation Task
Authors: Baixuan Xu; Haochen Shi; Tianshi Zheng; Qing Zong; Weiqi Wang; Zhaowei Wang; Yangqiu Song
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: Sign Language Translation (SLT) is a complex task that involves accurately interpreting sign language gestures and translating them into spoken or written language and vice versa. Its primary objective is to facilitate communication between individuals with hearing difficulties using deep learning systems. Existing approaches leverage gloss annotations of sign language gestures to assist the model in capturing the movement and differentiating various gestures. However, constructing a large-scale gloss-annotated dataset is both expensive and impractical to cover multiple languages, and pre-trained generative models cannot be efficiently used due to the lack of textual source context in SLT. To address these challenges, we propose a gloss-free framework for the WMT23 SLT task. Our system primarily consists of a visual extractor for extracting video embeddings and a generator responsible for producing the translated text. We also employ an embedding alignment block that is trained to align the embedding space of the visual extractor with that of the generator. Despite undergoing extensive training and validation, our system consistently falls short of meeting the baseline performance. Further analysis shows that our model’s poor projection rate prevents it from learning diverse visual embeddings. Our codes and model checkpoints are available at https://github.com/HKUST-KnowComp/SLT.
A Fast Method to Filter Noisy Parallel Data WMT2023 Shared Task on Parallel Data Curation
Authors: Nguyen-Hoang Minh-Cong; Nguyen Van Vinh; Nguyen Le-Minh
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: The effectiveness of a machine translation (MT) system is intricately linked to the quality of its training dataset. In an era where websites offer an extensive repository of translations such as movie subtitles, stories, and TED Talks, the fundamental challenge resides in pinpointing the sentence pairs or documents that represent accurate translations of each other. This paper presents the results of our submission to the shared task WMT2023 (Sloto et al., 2023), which aimed to evaluate parallel data curation methods for improving the MT system. The task involved alignment and filtering data to create high-quality parallel corpora for training and evaluating the MT models. Our approach leveraged a combination of dictionary and rule-based methods to ensure data quality and consistency. We achieved an improvement with the highest 1.6 BLEU score compared to the baseline system. Significantly, our approach showed consistent improvements across all test sets, suggesting its efficiency.
A Sentence Alignment Approach to Document Alignment and Multi-faceted Filtering for Curating Parallel Sentence Pairs from Web-crawled Data
Authors: Steinthor Steingrimsson
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper describes the AST submission to the WMT23 Shared Task on Parallel Data Curation. We experiment with two approaches for curating data from the provided web-scraped texts. We use sentence alignment to identify document alignments in the data and extract parallel sentence pairs from the aligned documents. All other sentences, not aligned in that step, are paired based on cosine similarity before we apply various different filters. For filtering, we use language detection, fluency classification, word alignments, cosine distance as calculated by multilingual sentence embedding models, and Bicleaner AI. Our best model outperforms the baseline by 1.9 BLEU points on average over the four provided evaluation sets.
Document-Level Language Models for Machine Translation
Authors: Frithjof Petrick; Christian Herold; Pavel Petrushkov; Shahram Khadivi; Hermann Ney
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: Despite the known limitations, most machine translation systems today still operate on the sentence-level. One reason for this is, that most parallel training data is only sentence-level aligned, without document-level meta information available. In this work, we set out to build context-aware translation systems utilizing document-level monolingual data instead. This can be achieved by combining any existing sentence-level translation model with a document-level language model. We improve existing approaches by leveraging recent advancements in model combination. Additionally, we propose novel weighting techniques that make the system combination more flexible and significantly reduce computational overhead. In a comprehensive evaluation on four diverse translation tasks, we show that our extensions improve document-targeted scores significantly and are also computationally more efficient. However, we also find that in most scenarios, back-translation gives even better results, at the cost of having to re-train the translation system. Finally, we explore language model fusion in the light of recent advancements in large language models. Our findings suggest that there might be strong potential in utilizing large language models via model combination.