GenAI4ELab/papercli-papers huggingface dataset:
A Hybrid of Rule-based and Transformer-based Approaches for Relation Extraction in Biodiversity Literature
Authors: Roselyn Gabud; Portia Lapitan; Vladimir Mariano; Eduardo Mendoza; Nelson Pampolina; Maria Art Antonette Clariño; Riza Batista-Navarro
Venue: EMNLP 2023 (Proceedings of the 2nd Workshop on Pattern-based Approaches to NLP in the Age of Deep Learning)

Abstract: Relation extraction (RE) is one of the tasks behind many relevant natural language processing (NLP) applications. Exploiting the information hidden in millions of scholarly articles by leveraging NLP, specifically RE, systems could benefit studies in specialized domains, e.g. biomedicine and biodiversity. Although deep learning (DL)-based methods have shown state-of-the-art performance in many NLP tasks including RE, DL for domain-specific RE systems has been hindered by the lack of expert-labeled datasets which are typically required to train such methods. In this paper, we take advantage of the zero-shot (i.e., not requiring any labeled data) capability of pattern-based methods for RE using a rule-based approach, combined with templates for natural language inference (NLI) transformer models. We present our hybrid method for RE that exploits the advantages of both methods, i.e., interpretability of rules and transferability of transformers. Evaluated on a corpus of biodiversity literature with annotated relations, our hybrid method demonstrated an improvement of up to 15 percentage points in recall and best performance over solely rule-based and transformer-based methods with F1-scores ranging from 89.61% to 96.75% for reproductive condition - temporal expression relations, and ranging from 85.39% to 89.90% for habitat - geographic location relations.
Dialogue-based generation of self-driving simulation scenarios using Large Language Models
Authors: Antonio Valerio Miceli Barone; Craig Innes; Alex Lascarides
Venue: EMNLP 2023 (Proceedings of the 3rd Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics (SpLU-RoboNLP 2023))

Abstract: Simulation is an invaluable tool for developing and evaluating controllers for self-driving cars. Current simulation frameworks are driven by highly-specialist domain specific languages, and so a natural language interface would greatly enhance usability. But there is often a gap, consisting of tacit assumptions the user is making, between a concise English utterance and the executable code that captures the user’s intent. In this paper we describe a system that addresses this issue by supporting an extended multimodal interaction: the user can follow up prior instructions with refinements or revisions, in reaction to the simulations that have been generated from their utterances so far. We use Large Language Models (LLMs) to map the user’s English utterances in this interaction into domain-specific code, and so we explore the extent to which LLMs capture the context sensitivity that’s necessary for computing the speaker’s intended message in discourse.
Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here but Not Quite There Yet
Authors: Tom Kocmi; Eleftherios Avramidis; Rachel Bawden; Ondřej Bojar; Anton Dvorkovich; Christian Federmann; Mark Fishel; Markus Freitag; Thamme Gowda; Roman Grundkiewicz; Barry Haddow; Philipp Koehn; Benjamin Marie; Christof Monz; Makoto Morishita; Kenton Murray; Masaaki Nagata; Toshiaki Nakazawa; Martin Popel; Maja Popović; Mariya Shmatova; Jun Suzuki
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper presents the results of the General Machine Translation Task organised as part of the 2023 Conference on Machine Translation (WMT). In the general MT task, participants were asked to build machine translation systems for any of 8 language pairs (corresponding to 14 translation directions), to be evaluated on test sets consisting of up to four different domains. We evaluate system outputs with professional human annotators using a combination of source-based Direct Assessment and scalar quality metric (DA+SQM).
Findings of the WMT 2023 Biomedical Translation Shared Task: Evaluation of ChatGPT 3.5 as a Comparison System
Authors: Mariana Neves; Antonio Jimeno Yepes; Aurélie Névéol; Rachel Bawden; Giorgio Maria Di Nunzio; Roland Roller; Philippe Thomas; Federica Vezzani; Maika Vicente Navarro; Lana Yeganova; Dina Wiemann; Cristian Grozea
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: We present an overview of the Biomedical Translation Task that was part of the Eighth Conference on Machine Translation (WMT23). The aim of the task was the automatic translation of biomedical abstracts from the PubMed database. It included twelve language directions, namely, French, Spanish, Portuguese, Italian, German, and Russian, from and into English. We received submissions from 18 systems and for all the test sets that we released. Our comparison system was based on ChatGPT 3.5 and performed very well in comparison to many of the submissions.
Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs
Authors: Longyue Wang; Zhaopeng Tu; Yan Gu; Siyou Liu; Dian Yu; Qingsong Ma; Chenyang Lyu; Liting Zhou; Chao-Hong Liu; Yufeng Ma; Weiyu Chen; Yvette Graham; Bonnie Webber; Philipp Koehn; Andy Way; Yulin Yuan; Shuming Shi
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: Translating literary works has perennially stood as an elusive dream in machine translation (MT), a journey steeped in intricate challenges. To foster progress in this domain, we hold a new shared task at WMT 2023, the first edition of the Discourse-Level Literary Translation. First, we (Tencent AI Lab and China Literature Ltd.) release a copyrighted and document-level Chinese-English web novel corpus. Furthermore, we put forth an industry-endorsed criteria to guide human evaluation process. This year, we totally received 14 submissions from 7 academia and industry teams. We employ both automatic and human evaluations to measure the performance of the submitted systems. The official ranking of the systems is based on the overall human judgments. In addition, our extensive analysis reveals a series of interesting findings on literary and discourse-aware MT. We release data, system outputs, and leaderboard at http://www2.statmt.org/wmt23/literary-translation-task.html.
Findings of the Second WMT Shared Task on Sign Language Translation (WMT-SLT23)
Authors: Mathias Müller; Malihe Alikhani; Eleftherios Avramidis; Richard Bowden; Annelies Braffort; Necati Cihan Camgöz; Sarah Ebling; Cristina España-Bonet; Anne Göhring; Roman Grundkiewicz; Mert Inan; Zifan Jiang; Oscar Koller; Amit Moryossef; Annette Rios; Dimitar Shterionov; Sandra Sidler-Miserez; Katja Tissi; Davy Van Landuyt
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper presents the results of the Second WMT Shared Task on Sign Language Translation (WMT-SLT23; https://www.wmt-slt.com/). This shared task is concerned with automatic translation between signed and spoken languages. The task is unusual in the sense that it requires processing visual information (such as video frames or human pose estimation) beyond the well-known paradigm of text-to-text machine translation (MT). The task offers four tracks involving the following languages: Swiss German Sign Language (DSGS), French Sign Language of Switzerland (LSF-CH), Italian Sign Language of Switzerland (LIS-CH), German, French and Italian. Four teams (including one working on a baseline submission) participated in this second edition of the task, all submitting to the DSGS-to-German track. Besides a system ranking and system papers describing state-of-the-art techniques, this shared task makes the following scientific contributions: novel corpora and reproducible baseline systems. Finally, the task also resulted in publicly available sets of system outputs and more human evaluation scores for sign language translation.
Findings of the WMT 2023 Shared Task on Parallel Data Curation
Authors: Steve Sloto; Brian Thompson; Huda Khayrallah; Tobias Domhan; Thamme Gowda; Philipp Koehn
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: Building upon prior WMT shared tasks in document alignment and sentence filtering, we posed the open-ended shared task of finding the best subset of possible training data from a collection of Estonian-Lithuanian web data. Participants could focus on any portion of the end-to-end data curation pipeline, including alignment and filtering. We evaluated results based on downstream machine translation quality. We release processed Common Crawl data, along with various intermediate states from a strong baseline system, which we believe will enable future research on this topic.
Samsung R&D Institute Philippines at WMT 2023
Authors: Jan Christian Blaise Cruz
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: In this paper, we describe the constrained submission systems of Samsung R&D Institute Philippines to the WMT 2023 General Translation Task for two directions: en->he and he->en. Our systems comprise of Transformer-based sequence-to-sequence models that are trained with a mix of best practices: comprehensive data preprocessing pipelines, synthetic backtranslated data, and the use of noisy channel reranking during online decoding. Our models perform comparably to, and sometimes outperform, strong baseline unconstrained systems such as mBART50 M2M and NLLB 200 MoE despite having significantly fewer parameters on two public benchmarks: FLORES-200 and NTREX-128.
NAIST-NICT WMT’23 General MT Task Submission
Authors: Hiroyuki Deguchi; Kenji Imamura; Yuto Nishida; Yusuke Sakai; Justin Vasselli; Taro Watanabe
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: In this paper, we describe our NAIST-NICT submission to the WMT’23 English ↔ Japanese general machine translation task. Our system generates diverse translation candidates and reranks them using a two-stage reranking system to find the best translation. First, we generated 50 candidates each from 18 translation methods using a variety of techniques to increase the diversity of the translation candidates. We trained seven models per language direction using various combinations of hyperparameters. From these models we used various decoding algorithms, ensembling the models, and using kNN-MT (Khandelwal et al., 2021). We processed the 900 translation candidates through a two-stage reranking system to find the most promising candidate. In the first step, we compared 50 candidates from each translation method using DrNMT (Lee et al., 2021) and returned the candidate with the best score. We ranked the final 18 candidates using COMET-MBR (Fernandes et al., 2022) and returned the best score as the system output. We found that generating diverse translation candidates improved translation quality using the well-designed reranker model.
CUNI at WMT23 General Translation Task: MT and a Genetic Algorithm
Authors: Josef Jon; Martin Popel; Ondřej Bojar
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper presents the contributions of Charles University teams to the WMT23 General translation task (English to Czech and Czech to Ukrainian translation directions). Our main submission, CUNI-GA, is a result of applying a novel n-best list reranking and modification method on translation candidates produced by the two other submitted systems, CUNI-Transformer and CUNI-DocTransformer (document-level translation only used for the en → cs direction). Our method uses a genetic algorithm and MBR decoding to search for optimal translation under a given metric (in our case, a weighted combination of ChrF, BLEU, COMET22-DA, and COMET22-QE-DA). Our submissions are first in the constrained track and show competitive performance against top-tier unconstrained systems across various automatic metrics.
SKIM at WMT 2023 General Translation Task
Authors: Keito Kudo; Takumi Ito; Makoto Morishita; Jun Suzuki
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: The SKIM team’s submission used a standard procedure to build ensemble Transformer models, including base-model training, back-translation of base models for data augmentation, and retraining of several final models using back-translated training data. Each final model had its own architecture and configuration, including up to 10.5B parameters, and substituted self- and cross-sublayers in the decoder with a cross+self-attention sub-layer. We selected the best candidate from a large candidate pool, namely 70 translations generated from 13 distinct models for each sentence, using an MBR reranking method using COMET and COMET-QE. We also applied data augmentation and selection techniques to the training data of the Transformer models.
KYB General Machine Translation Systems for WMT23
Authors: Ben Li; Yoko Matsuzaki; Shivam Kalkar
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper describes our approach to constructing a neural machine translation system for the WMT 2023 general machine translation shared task. Our model is based on the Transformer architecture’s base settings. We optimize system performance through various strategies. Enhancing our model’s capabilities involves fine-tuning the pretrained model with an extended dataset. To further elevate translation quality, specialized pre- and post-processing techniques are deployed. Our central focus is on efficient model training, aiming for exceptional accuracy through the synergy of a compact model and curated data. We also performed ensembling augmented by N-best ranking, for both directions of English to Japanese and Japanese to English translation.
Yishu: Yishu at WMT2023 Translation Task
Authors: Luo Min; Yixin Tan; Qiulin Chen
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper introduces the Dtranx AI translation system, developed for the WMT 2023 Universal Translation Shared Task. Our team participated in two language directions: English to Chinese and Chinese to English. Our primary focus was on enhancing the effectiveness of the Chinese-to-English model through the implementation of bilingual models. Our approach involved various techniques such as data corpus filtering, model size scaling, sparse expert models (especially the Transformer model with adapters), large-scale back-translation, and language model reordering. According to automatic evaluation, our system secured the first place in the English-to-Chinese category and the second place in the Chinese-to-English category.
PROMT Systems for WMT23 Shared General Translation Task
Authors: Alexander Molchanov; Vladislav Kovalenko
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper describes the PROMT submissions for the WMT23 Shared General Translation Task. This year we participated in two directions of the Shared Translation Task: English to Russian and Russian to English. Our models are trained with the MarianNMT toolkit using the transformer-big configuration. We use BPE for text encoding, both models are unconstrained. We achieve competitive results according to automatic metrics in both directions.
AIST AIRC Submissions to the WMT23 Shared Task
Authors: Matiss Rikters; Makoto Miwa
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper describes the development process of NMT systems that were submitted to the WMT 2023 General Translation task by the team of AIST AIRC. We trained constrained track models for translation between English, German, and Japanese. Before training the final models, we first filtered the parallel and monolingual data, then performed iterative back-translation as well as parallel data distillation to be used for non-autoregressive model training. We experimented with training Transformer models, Mega models, and custom non-autoregressive sequence-to-sequence models with encoder and decoder weights initialised by a multilingual BERT base. Our primary submissions contain translations from ensembles of two Mega model checkpoints and our contrastive submissions are generated by our non-autoregressive models.
MUNI-NLP Submission for Czech-Ukrainian Translation Task at WMT23
Authors: Pavel Rychly; Yuliia Teslia
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: The system is trained on officialy provided data only. We have heavily filtered all the data to remove machine translated text, Russian text and other noise. We use the DeepNorm modification of the transformer architecture in the TorchScale library with 18 encoder layers and 6 decoder layers. The initial systems for backtranslation uses HFT tokenizer, the final system uses custom tokenizer derived from HFT.
Exploring Prompt Engineering with GPT Language Models for Document-Level Machine Translation: Insights and Findings
Authors: Yangjian Wu; Gang Hu
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper describes Lan-Bridge Translation systems for the WMT 2023 General Translation shared task. We participate in 2 directions: English to and from Chinese. With the emergence of large-scale models, various industries have undergone significant transformations, particularly in the realm of document-level machine translation. This has introduced a novel research paradigm that we have embraced in our participation in the WMT23 competition. Focusing on advancements in models such as GPT-3.5 and GPT-4, we have undertaken numerous prompt-based experiments. Our objective is to achieve optimal human evaluation results for document-level machine translation, resulting in our submission of the final outcomes in the general track.
Treating General MT Shared Task as a Multi-Domain Adaptation Problem: HW-TSC’s Submission to the WMT23 General MT Shared Task
Authors: Zhanglin Wu; Daimeng Wei; Zongyao Li; Zhengzhe Yu; Shaojun Li; Xiaoyu Chen; Hengchao Shang; Jiaxin Guo; Yuhao Xie; Lizhi Lei; Hao Yang; Yanfei Jiang
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper presents the submission of Huawei Translate Services Center (HW-TSC) to the WMT23 general machine translation (MT) shared task, in which we participate in Chinese↔English (zh↔en) language pair. We use Transformer architecture and obtain the best performance via a variant with larger parameter size. We perform fine-grained pre-processing and filtering on the provided large-scale bilingual and monolingual datasets. We mainly use model enhancement strategies, including Regularized Dropout, Bidirectional Training, Data Diversification, Forward Translation, Back Translation, Alternated Training, Curriculum Learning and Transductive Ensemble Learning. Our submissions obtain competitive results in the final evaluation.
UvA-MT’s Participation in the WMT 2023 General Translation Shared Task
Authors: Di Wu; Shaomu Tan; David Stap; Ali Araabi; Christof Monz
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper describes the UvA-MT’s submission to the WMT 2023 shared task on general machine translation. We participate in the constrained track in two directions: English ↔ Hebrew. In this competition, we show that by using one model to handle bidirectional tasks, as a minimal setting of Multilingual Machine Translation (MMT), it is possible to achieve comparable results with that of traditional bilingual translation for both directions. By including effective strategies, like back-translation, re-parameterized embedding table, and task-oriented fine-tuning, we obtained competitive final results in the automatic evaluation for both English → Hebrew and Hebrew → English directions.
Achieving State-of-the-Art Multilingual Translation Model with Minimal Data and Parameters
Authors: Hui Zeng
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This is LanguageX (ZengHuiMT)’s submission to WMT 2023 General Machine Translation task for 13 language directions. We initially employ an encoder-decoder model to train on all 13 competition translation directions as our baseline system. Subsequently, we adopt a decoder-only architecture and fine-tune a multilingual language model by partially sampling data from diverse multilingual datasets such as CC100 and WuDaoCorpora. This is further refined using carefully curated high-quality parallel corpora across multiple translation directions to enable the model to perform translation tasks. As per automated evaluation metrics, our model ranks first in the translation directions from English to Russian, English to German, and English to Ukrainian. It secures the second position in the directions from English to Czech, English to Hebrew, Hebrew to English, and Ukrainian to English, and ranks third in German to English, Japanese to English, and Russian to English among all participating teams. Our best-performing model, covering 13 translation directions, stands on par with GPT-4. Among all 13 translation directions, our multilingual model surpasses GPT-4 in bleu scores for 7 translation directions.
IOL Research Machine Translation Systems for WMT23 General Machine Translation Shared Task
Authors: Wenbo Zhang
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper describes the IOL Research team’s submission systems for the WMT23 general machine translation shared task. We participated in two language translation directions, including English-to-Chinese and Chinese-to-English. Our final primary submissions belong to constrained systems, which means for both translation directions we only use officially provided monolingual and bilingual data to train the translation systems. Our systems are based on Transformer architecture with pre-norm or deep-norm, which has been proven to be helpful for training deeper models. We employ methods such as back-translation, data diversification, domain fine-tuning and model ensemble to build our translation systems. An important aspect worth mentioning is our careful data cleaning process and the utilization of a substantial amount of monolingual data for data augmentation. Compared with the baseline system, our submissions have a large improvement in BLEU score.
GTCOM and DLUT’s Neural Machine Translation Systems for WMT23
Authors: Hao Zong
Venue: EMNLP 2023 (Proceedings of the Eighth Conference on Machine Translation)

Abstract: This paper presents the submission by Global Tone Communication Co., Ltd. and Dalian Univeristy of Technology for the WMT23 shared general Machine Translation (MT) task at the Conference on Empirical Methods in Natural Language Processing (EMNLP). Our participation spans 8 language pairs, including English-Ukrainian, Ukrainian-English, Czech-Ukrainian, English-Hebrew, Hebrew-English, English-Czech, German-English, and Japanese-English. Our systems are designed without any specific constraints or requirements, allowing us to explore a wider range of possibilities in machine translation. We prioritize backtranslation, utilize multilingual translation models, and employ fine-tuning strategies to enhance performance. Additionally, we propose a novel data generation method that leverages human annotation to generate high-quality training data, resulting in improved system performance. Specifically, we use a combination of human-generated and machine-generated data to fine-tune our models, leading to more accurate translations. The automatic evaluation results show that our system ranks first in terms of BLEU score in Ukrainian-English, Hebrew-English, English-Hebrew, and German-English.