There and Back Again: Cross-Lingual Transfer Learning for Event Detection
Authors: Tommaso Caselli, Ahmet Üstün (University of Groningen, 2019)
Topic: Language Models

Abstract
In this contribution, we investigate the generalization abilities of a pre-trained multilingual Language Model, namely Multilingual BERT, in different transfer learning scenarios for event detection and classification for Italian and English. Our results show that zero-shot models have satisfying, although not optimal, performances in both languages (average F1 higher than 60 for event detection vs. average F1 ranging between 40 and 50 for event classification). We also show that adding extra fine-tuning data of the evaluation language is not only beneficial but results in better models when compared to the corresponding non-zero-shot transfer ones, achieving highly competitive results when compared to state-of-the-art systems.

Introduction
Recently pre-trained word representations encoded in Language Models (LM) have gained lot of popularity in Natural Language Processing (NLP) thanks to their ability to encode high-level syntactic-semantic language features and produce state-of-the-art results in various tasks, such as Named Entity Recognition (Peters et al., 2018), Machine Translation (Johnson et al., 2017; Ramachandran et al., 2017), Text Classification (Eriguchi et al., 2018; Chronopoulou et al., 2019), among others. These models are pre-trained on large amounts of unannotated text and then fine-tuned using the induced LM structure to generalize over specific training data. Given their success in monolingual environments, especially for English, there has been a growing interest in the development of cross-lingual as well as multilingual representations (Vulić and Moens, 2015; Ammar et al., 2016; Conneau et al., 2018; Artetxe et al., 2018) to investigate different cross-lingual transfer learning scenarios, including zero-shot transfer, i.e. the direct application of a model fine-tuned using data in one language to a different test language. Following the approach in Pires et al. (2019), in this paper we investigate the generalisation abilities of Multilingual BERT (Devlin et al., 2019) on English (EN) and Italian (IT). Multilingual BERT is particularly well suited for this task because it easily allows the implementation of cross-lingual transfer learning, including zero-shot transfer. We use event detection as our downstream task, a highly complex semantic task with a well-established tradition in NLP (Ahn, 2006; Ji and Grishman, 2008; Ritter et al., 2012; Nguyen and Grishman, 2015; Huang et al., 2018). The goal of the task is to identify event mentions, i.e. linguistic expressions describing “things” that happen or hold as true in the world, and subsequently classify them according to a (pre-defined) taxonomy. The complexity of the task relies on its high dependence on the context of occurrence of the expressions that may trigger an event mention. Indeed, the eventiveness of an expression is prone to ambiguity because there exists a continuum between eventive and non-eventive readings in the space of event semantics (Araki et al., 2018). Such intrinsic ambiguity of event expressions challenges the generalisation abilities of stochastic models and allows investigating advantages and limits of transfer learning approaches when semantics has a pivotal role in the resolution of a problem/task. We explore different multi-lingual and cross-lingual aspects of transfer learning with respect to event detection through a series of experiments, focusing on the following research questions: RQ1 How well do Multilingual BERT fine-tuned models generalise in zero-shot transfer learning scenarios on both languages? RQ2 Do we obtain more robust models by fine-tuning zero-shot models with additional (training) data of the evaluation language? Our results show that Multilingual BERT obtains satisfying performances in zero-shot scenarios for the identification of event triggers (average F1 63.53 on Italian and 66.79 on English), while this is not the case for event classification (average F1 42.86 on Italian and 51.26 on English). We also show that extra fine-tuning the zero-shot models with data of the evaluation language is not just beneficial, but it actually gives better results than models fine-tuned on the corresponding test language only (i.e. fine-tuning and test in the same language), and achieves competitive results with state-of-the-art systems developed using dedicated architectures. Our code is available (https://github.com/ahmetustun/BertForEvent).

Conclusion
In this contribution we investigated the generalisation abilities of Multilingual BERT on Italian and English using event detection as a downstream task. The results show that Multilingual BERT seems to handle cross-lingual generalisation between Italian and English in a satisfying way, although with some limitations. Limitations in this case come from two sources: annotation differences in the two languages and, partially, the shared multilingual vocabulary. Zero-shot systems appear to be particularly sensitive to the fine-tuning data, and, in these experiments, they provide empirical evidence of the impact of different annotation decisions for events in English and Italian. We have shown that extra fine-tuning with data of the evaluation language not only is beneficial but it may lead to better systems, suggesting that the multilingual model may be combining information from the two languages, and thus obtaining competitive results with respect to task-specific architectures. This opens up to new strategies for the development of systems by using interoperable annotated data in different languages to improve performances and possibly obtain more robust and portable models across different data distributions.