XL-WA: a Gold Evaluation Benchmark for Word Alignment in 14 Language Pairs
Authors: Federico Martelli, Andrei Stefan Bejgu, Cesare Campagnano, Jaka Čibej, Rute Costa, Apolonija Gantar, Jelena Kallas, Svetla Peneva Koeva, Kristina Koppel, Simon Krek, Margit Langemets, Veronika Lipp, Sanni Nimb, Sussi Olsen, Bolette Sanford Pedersen, Valeria Quochi, Ana Salgado, László Simon, Carole Tiberius, Rafael-J Ureña-Ruiz, Roberto Navigli (Academia das Ciências de Lisboa, Babelscape, Bulgarian Academy of Sciences, Centro de Estudios de la Real Academia Española, HUN-REN Hungarian Research Centre for Linguistics, Institute of the Estonian Language, Instituut voor de Nederlandse Taal, CNR-ILC, Jožef Stefan Institute, NOVA CLUNL, Sapienza Università di Roma, Society for Danish Language and Literature, University of Copenhagen, University of Ljubljana, 2023)
Topic: Machine Translation

Abstract
Word alignment plays a crucial role in several NLP tasks, such as lexicon injection and cross-lingual label projection. The evaluation of word alignment systems relies heavily on manually-curated datasets, which are not always available, especially in mid- and low-resource languages. In order to address this limitation, we propose XL-WA, a novel entirely manually-curated evaluation benchmark for word alignment covering 14 language pairs. We illustrate the creation process of our benchmark and compare statistical and neural approaches to word alignment in both language-specific and zero-shot settings, thus investigating the ability of state-of-the-art models to generalize on unseen language pairs. We release our new benchmark at: LINK.

Introduction
Word alignment is the computational task of identifying translation correspondences at word and multi-word levels between parallel sentences. Historically, word alignment played a crucial role in Statistical Machine Translation (SMT). However, while SMT has been replaced by end-to-end neural architectures that attain considerably higher performances, word alignment – also thanks to novel neural approaches – still plays a crucial role in many other NLP tasks, such as lexicon injection and, most importantly, cross-lingual annotation projection. For instance, Procopio et al. recently proposed a state-of-the-art approach to cross-lingual label projection based on word alignment, which allows high-quality sense-tagged datasets to be produced automatically. Furthermore, word alignment has also been leveraged effectively to create silver datasets, not only for Word Sense Disambiguation (WSD) but also for other semantic tasks, such as Semantic Role Labeling (SRL), thereby addressing the knowledge acquisition bottleneck, especially when dealing with mid- and low-resource languages. While, on the one hand, current architectures for word alignment are achieving increasingly better performance, on the other hand, the lack of high-quality manual data in multiple languages significantly limits their potential and scalability. With a view to addressing the aforementioned drawbacks, our contributions are as follows: 1. We propose a fully manually-annotated evaluation benchmark for word alignment with a total of 14 language pairs, each composed of English and one of the following languages: Arabic, Bulgarian, Chinese, Danish, Dutch, Estonian, Hungarian, Italian, Korean, Portuguese, Russian, Slovenian, Spanish, and Swedish. 2. We experiment with statistical and neural approaches to word alignment and evaluate them against our newly created benchmark. 3. We demonstrate that the concatenation of our novel datasets can be exploited effectively to train a neural approach that generalizes on unseen languages in a zero-shot setting, thereby addressing the lack of training data in low-resource languages.

Conclusion
In this work, we introduce XL-WA, a novel evaluation benchmark for word alignment in 14 language pairs. We detail the creation process for our novel evaluation suite, as well as our experimental setup in which we compare statistical and neural approaches to word alignment. We investigate the behavior of models in zero-shot scenarios and show that the concatenation of our datasets can be used effectively to align languages unseen during training, thus tackling the paucity or limited availability of data for word alignment in low-resource languages. We release our new benchmark at: LINK. As future work, we intend to investigate the impact of language-specific peculiarities on the overall performance of neural models for word alignment. Furthermore, we plan to increase the language coverage of XL-WA and, importantly, investigate the role played by additional low-resource languages in zero-shot settings. Finally, we aim to explore novel neural approaches to word alignment which can be employed in the field of cross-lingual label projection in order to create multilingual silver training datasets for several Natural Language Understanding tasks, such as WSD, SRL and Semantic Parsing.