Venue: NAACL 2022

# MINION: a Large-Scale and Diverse Dataset

for Multilingual Event Detection

Amir Pouran Ben Veyseh\({}^{1}\), Minh Van Nguyen\({}^{1}\),

**Franck Dernoncourt\({}^{2}\), and Thien Huu Nguyen\({}^{1}\)**

\({}^{1}\) Dept. of Computer and Information Science, University of Oregon, Eugene, OR, USA

\({}^{2}\) Adobe Research, Seattle, WA, USA {apouranb,minhnv,thien}@cs.uoregon.edu, dernonco@adobe.com

###### Abstract

Event Detection (ED) is the task of identifying and classifying trigger words of event mentions in text. Despite considerable research efforts in recent years for English text, the task of ED in other languages has been significantly less explored. Switching to non-English languages, important research questions for ED include how well existing ED models perform on different languages, how challenging ED is in other languages, and how well ED knowledge and annotation can be transferred across languages. To answer those questions, it is crucial to obtain multilingual ED datasets that provide consistent event annotation for multiple languages. There exist some multilingual ED datasets; however, they tend to cover a handful of languages and mainly focus on popular ones. Many languages are not covered in existing multilingual ED datasets. In addition, the current datasets are often small and not accessible to the public. To overcome those shortcomings, we introduce a new large-scale multilingual dataset for ED (called MINION) that consistently annotates events for 8 different languages; 5 of them have not been supported by existing multilingual datasets. We also perform extensive experiments and analysis to demonstrate the challenges and transferability of ED across languages in MINION that in all call for more research effort in this area. We will release the dataset to promote future research on multilingual ED.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Event Detection (ED) is one of the critical steps for an Event Extraction system in Information Extraction (IE) that aims is to recognize mentions of events in text, i.e., change of state of real world entities. Specifically, an ED system identifies the word(s) that most clearly refer to the occurrence of an event, i.e., event trigger, and also detects the type of event that is evoked by the event trigger. For instance, in the sentence "_The city was reportedly struck by F16 missiles._", the word "_struck_" is the trigger for an _ATTACK_ event. An ED model can be incorporated into other IE pipelines to facilitate the extraction of information related to events and entities, thereby supporting various downstream applications such as knowledge base construction, question answering and text summarization.

Due to its importance, ED has been extensively studied in the IE and NLP community over the past decade. Existing methods for ED extend from feature-based models Ahn (2006); Liao and Grishman (2010); Miwa et al. (2014a), to advanced deep learning methods Nguyen and Grishman (2015); Chen et al. (2015); Sha et al. (2018); Wang et al. (2019); Yang et al. (2019); Cui et al. (2020); Lai et al. (2020); Pouran Ben Veyseh et al. (2021). As such, the creation of large annotated datasets for ED, e.g., ACE 2005 Walker et al. (2006), has been critical to progress measurement and growing development of ED research. However, a majority of current datasets for ED only provide annotation for texts in a single language (i.e., monolingual datasets). For instance, the recent challenging datasets for ED, e.g., MAVEN Wang et al. (2020), RAMS Ebner et al. (2020), or CySecED Man et al. (2020), are all proposed for English documents only. In addition, there are a few existing datasets that include ED annotation for multiple languages (multilingual datasets), e.g., ACE 2005 Walker et al. (2006), TAC KBP Mitamura et al. (2016, 2017), and TempEval-2 Verhagen et al. (2010). However, those multilingual datasets only cover a handful of languages (i.e., 3 languages in ACE 2005 and TAC KBP, and 6 languages in TempEval-2), mainly focusing on popular languages such as English, Chinese, Arabic, and Spanish, and leaving many other languages unexplored for ED. For instance, Turkish and Polish are not covered in existing multilingual datasets for ED. We also note that existing ED datasets tend to employ different annotation schema and guidelines that prevent the combina

tion of current datasets to create a larger one. In all, the limited coverage of languages and annotation discrepancy in current monolingual/multilingual ED datasets hinder comprehensive studies for the challenges of ED in diverse languages. It also limits thorough evaluations for multilingual generalization of ED models. Finally, we note that the major multilingual datasets for ED are not publicly accessible due to the licence of involving documents, e.g., ACE 2005 and TAC KBP, thus further impeding research effort in this area.

To address such issues, our goal is to introduce a new Multilingual Event Detection dataset (called MINION) to support multilingual research for ED. In particular, we provide a large-scale dataset that manually annotates event triggers for 8 typologically different languages, i.e., English, Spanish, Portuguese, Polish, Turkish, Hindi, Japanese and Korean. Among them, the five languages Portuguese, Polish, Turkish, Hindi, and Japanese are not covered in existing popular datasets for multilingual ED (i.e., ACE 2005, TAC KBP, and TempEval-2). To facilitate public release and sharing of the dataset, we employ the event articles from Wikipedia for annotation in 8 languages. In addition, to improve quality of the data, we inherit the annotation schema and guideline in ACE 2005, the well-designed and widely-used dataset for ED research. In total, our MINION dataset involves more than 50K annotated event triggers, which is much larger than those in existing multilingual ED datasets (i.e., less than 11K and 27K in ACE 2005 and TempEval-2 respectively). We expect that the significantly larger size with more diverse set of languages and public texts in MINION can contribute to accelerate and extend research in ED to a larger population.

Given the proposed dataset, we conduct thorough analysis on MINION using the state-of-the-art (SOTA) models for ED. In particular, we first study the challenges of ED in different languages using monolingual evaluations where ED models are trained and tested in the same languages. Our experiments suggest that the performance of existing ED models is not yet satisfactory in multiple languages and the model performance on non-English languages is in general poorer than those for English. We also show that current pre-trained language models for specific languages (i.e., monolingual models) are less effective for ED models than multilingual pre-trained language models, e.g., mBERT Devlin et al. (2019). In all, our findings highlight greater challenges of ED for non-English languages that should be further pursued in future research.

In addition, our MINION dataset also facilitate zero-shot cross-lingual transfer learning experiments that serve to reveal the transferability of ED knowledge and annotation across languages. In these experiments, ED models are trained on English data (the source language), but tested in other target languages. Our results in this setting demonstrate a wide range of cross-lingual performance for different target languages in MINION that introduces a diverse set of languages and data for ED research. Finally, we report extensive analysis on MINION to provide further data insights for future ED research, including challenges of data annotation, language differences, and cross-dataset evaluation. We will release MINION to foster future research for multilingual ED.

## 2 Data Annotation

Our dataset MINION follows the same definition of events as the annotation guideline in ACE 2005 Walker et al. (2006). Specifically, an event is defined as an occurrence that results in the change of state of a real world entity. Moreover, an event mention is evoked by an event trigger which most clearly describes the occurrence of the event. While event triggers are mostly single words, we also allow multi-word event triggers to better accommodate ED annotation in multiple languages. For instance, the phrasal verb "_tayin etmek_" with two words in Turkish, meaning "_appoint_", is necessary to express the event type _Start-Position_.

We also inherit the annotation schema/ontology (i.e., to define event types for annotation) and guideline in ACE 2005 to benefit from its well-designed documentation and be consistent with most of prior ED research. However, to improve the quality of the annotated data, we prune some event sub-types from the original ACE 2005 ontology in our dataset. In particular, event sub-types that have very similar meanings in some language are not included in our final ontology. This promotes the distinction between event labels and avoids confusion for annotators to provide high-quality data in different languages. For instance, the event sub-types _Convict_ and _Sentence_ are very similar in Turkish (i.e., both _Convic_ and _Sentence_ can be translated as _Mahkum etmek_ in Turkish), thus being removed 

in our ontology. In addition, we also exclude event sub-types in ACE 2005 that are not frequent in our collected data from Wikipedia (more details on data collection later), e.g., _Nominate_ and _DeclareBankruptcy_. Finally, 16 event sub-types (for 8 event types) are preserved in the final event schema for our dataset. We provide detailed explanation and sample sentences for the event types in our dataset in the Appendix A.

### Candidate Selection

As mentioned in the introduction, we aim to annotate ED data for 8 languages, i.e., English, Spanish, Portuguese, Polish, Turkish, Hindi, Japanese and Korean. These languages are selected due to their diversity in term of typology and novelty w.r.t. to existing multilingual ED datasets that can be helpful for multilingual model development and generalization evaluation. To collect text data for annotation in each language, we employ the articles of the language-specific editions of Wikipedia. Specifically, for each language, we obtain its latest dump of Wikipedia articles1, then process the dump with the parser _WikiExtractor_[4] to extract textual and meta data for articles. To increase the likelihood of encountering event mentions for effective annotation, we utilize the articles that are classified under one of the sub-categories of the _Event_ category in Wikipedia. In particular, we focus on six sub-categories _Economy_, _Politics_, _Technology_, _Crimes_, _Nature_, and _Military_ due to their relevance to the event types in our ontology. Note that we map these (sub)categories in English to the corresponding (sub)categories in other languages using the provided links in Wikipedia. Afterward, to split the texts into sentences and tokens, we leverage the multilingual toolkit Transit [37] that has demonstrated state-of-the-art performance for such tasks in our languages.

Footnote 1: Dumps were downloaded in May 2021.

Given a Wikipedia article, an approach for ED annotation is to ask the annotators to annotate the entire document for event triggers at once. However, as Wikipedia articles tend to be long, this approach might be overwhelming for annotators, thus potentially limiting the annotation quality. To this end, motivated by the annotation with 5-sentence windows in the RAMS dataset [9], we split each article into segments of 5 sentences that will be annotated separately by annotators. In this way, annotators only need to process a shorter context at a time to improve the attention and accuracy of annotated data. This annotation approach is also supported by a large amount of prior ED research where a majority of previous ED models have employed context information in single sentences to deliver high extraction performance for the event types in ACE 2005 [37, 38, 39, 38, 37, 36], including models for multiple languages [11, 35, 36].

### Annotation Process

To annotate the produced article segments, we hire annotators from upwork.com, a crowd-sourcing platform with freelancer annotators across the globe. In particular, our annotator candidate pool for each language of interest involves native speakers of the language who also have experience on related data annotation projects (e.g., for named entity recognition), an approval rate higher than 95%, and fluency in English. These information is provided by annotator profiles in Upwork. In the next step, the candidates are trained for ED annotation using the English annotation guideline and examples for the designed event schema in our dataset (i.e., inherited from ACE 2005). Finally, we ask the candidates to take an annotation test designed for ED in English and only candidates with passing results are officially selected for the annotators of our multilingual ED dataset. Overall, we recruit several annotators for each language of interest as shown in Table 2. To prepare for the actual annotation, the annotators for each language will work together to produce a translation of the English annotation guideline/examples where language-specific annotation rules are discussed and included in the translated guideline to form common annotation perception for the language. The translated guideline and examples are also verified by our language experts to avoid any potential conflicts and issues.

Finally, given the language-specific guidelines, the annotators for each language will independently annotate a chunk of article segments for that language. The breakdown numbers of annotated text segments for each language and Wikipedia subcategory in our MINION dataset are shown in Table 3. As such, 20% of the annotated text segments for each language is selected for co-annotation by the annotators to measure inter-annotator agree 

ment (IAA) scores while the remaining 80% is distributed to annotators for separate annotation. Table 2 reports the Krippendorff's alpha [15] with MASI distance metric [37] for the IAA scores of each language in our dataset. After independent annotation, the annotators will resolve the conflict cases to produce the final version of our MINION dataset. Overall, our dataset demonstrates high agreement scores for all the 8 languages, thus providing a high-quality dataset for multilingual ED.

### Data Analysis

The main statistics for our MINION dataset is provided in Table 3. This table shows that for a majority of languages, there are multiple event triggers in a text segment, thereby introducing a challenge for ED models due to the overlap of event context. In addition, the table shows that text segments in some languages are more replete with event mentions than those for other languages. Specifically, comparing Polish and English text segments, the density of event mentions in Polish is almost two times more than that for English. Finally, Figure 1 shows the distributions of 8 event types for the 8 languages in our dataset. As can be seen, the languages in our dataset tend to involve different levels of discrepancy regarding the distributions over event types. As such, the type density and distribution divergence between languages suggest other challenges that robust ED models should handle to perform well across languages in MINION.

### Annotation Challenges

Despite the high inter-annotator agreement scores, there are some conflicts between our annotators during the annotation process due to the ambiguity of event triggers, especially in the multilingual setting. This section highlights some of the key ambiguities/conflicts that we encounter during our analysis of annotation results from the annotators. Note that all of these conflicts have been resolved by the annotators in the final version of our dataset.

**Language-Specific Challenges**: Despite common notion of events in different languages, each language might has its own exceptions regarding how an event trigger should be annotated, causing confusions/conflict for our annotators in the annotation process. One exception concerns the necessity to include event arguments in the annotation of an event trigger in some language. For example, in the Polish sentence "_Samolot sie rozbit_" (translated as "_The plane crashed itself_"), some annotators believe that the meaning of the verb "_rozbit_" (i.e., crashed) is incomplete if its argument word "_site_" (i.e., itself) is not associated. As such, annotating both the verb and its argument (i.e., "_sie rozbit_") is necessary to express an event in this case. However, other annotators suggest that only annotating the word "_rozbit_" is sufficient. Our annotators have decided to annotate event triggers along with necessary arguments to achieve their complete meanings in such cases.

**Background Knowledge**: Background knowledge is sometime important to correctly recognize an event trigger in input text. In such cases, the annotators might have conflicting event annotation decisions for a word as their levels of background knowledge are different. For instance, in the sentence "_The match was canceled in the memory of victims of Katyn crime_", some annotators annotate the word "_crime_" as a _Die_ event trigger as they

\begin{table}
\begin{tabular}{l|c c c c c c c c} Category & English & Spanish & Portuguese & Polish & Turkish & Hindi & Japanese & Korean \\ \hline Economy & 1,095 & 112 & 168 & 315 & 297 & 189 & 199 & 250 \\ Politics & 3,202 & 308 & 772 & 1,270 & 1,233 & 349 & 232 & 248 \\ Technology & 2,171 & 189 & 400 & 712 & 815 & 295 & 312 & 249 \\ Crimes & 893 & 78 & 220 & 152 & 118 & 95 & 80 & 73 \\ Nature & 1,195 & 398 & 705 & 455 & 398 & 245 & 299 & 185 \\ Military & 4,444 & 415 & 1,003 & 1,575 & 1,619 & 326 & 378 & 495 \\ \hline Total & 13,000 & 1,500 & 3,268 & 4,479 & 4,480 & 1,499 & 1,500 & 1,500 \\ \end{tabular}
\end{table}
Table 1: Numbers of annotated segments in each Wikipedia subcategory for the 8 languages.

\begin{table}
\begin{tabular}{l|r r} Language & \#Annotator & IAA \\ \hline English & 10 & 0.834 \\ Spanish & 10 & 0.812 \\ Portuguese & 5 & 0.803 \\ Polish & 8 & 0.799 \\ Turkish & 10 & 0.813 \\ Hindi & 6 & 0.803 \\ Japanese & 5 & 0.789 \\ Korean & 6 & 0.810 \\ \end{tabular}
\end{table}
Table 2: Agreement scores for 8 languages in MINION.

 

know that "_crime_" is referring to a mass execution event. However, some annotators do not consider "_crime_" as an event trigger as they are not aware of the execution event. Eventually, we have decided to annotate the text segments based on only the presented information in the input texts to resolve conflicts and avoid inconsistency.

## 3 Experiments

This section aims to study the challenges of ED for 8 languages in our MINION dataset. As such, we evaluate the performance of the state-of-the-art (SOTA) ED models in the monolingual situations where models are trained and tested on the annotated data of the same language. To prepare for the experiments, we randomly split the annotated data for each language in MINION into separate training/development/test sets with the ratio of 80/10/10 (respectively). As MINION allows multiword event triggers to accommodate language specialities in multiple languages, we model the ED task in our dataset as a sequence labeling problem. Concretely, given an input text segment \(D=[w_{1},w_{2},\ldots,w_{n}]\) with \(n\) words, ED models need to predict the label sequence \(Y=[y_{1},y_{2},\ldots,y_{n}]\) where \(y_{i}\) indicates the label for the word \(w_{i}\in D\) using the BIO tagging schema.

To this end, following prior work on multilingual ED Wang et al. (2020) and cross-lingual ED M'hamdi et al. (2019), we examine the following representative SOTA models for sequence-labeling ED: (1) **Transformer**: A pre-trained transformer-based language model (PTLM), e.g., mBERT Devlin et al. (2019), is augmented with a feed-forward network to predict a label for each word in the input text; (2) **Transformer+CRF**: This model also employs an PTLM as the **Transformer** model; however, a Conditional Random Field (CRF) layer is additionally introduced as the final layer to predict the label sequence \(Y\); (3) **Transformer+BiLSTM**: This model extends the **Transformer** model by injecting a bidirectional Long Short-Term Memory network (BiLSTM) between the PTLM and the feed-forward network to further abstract the representation vectors; and (4) **Transformer+BiLSTM+CRF**: This model is similar to the **Transformer+BiLSTM** model with an exception that a CRF layer is employed in the end for label sequence prediction. As such, to implement the models, we explore two SOTA multilingual PTLMs models, i.e., mBERT Devlin et al. (2019) and XLMR Conneau et al. (2020) (their base versions) for text encoding. In the model notation, we will replace the prefix "Transformer" with "mBERT" or "XLMR" depending on the actual PTM to use (e.g., mBERT, mBERT+CRF, mBERT+BiLSTM). Following prior work Wang et al. (2020); M'hamdi et al. (2019), in the experiments, we evaluate the models using precision, recall and F1 scores for correctly predicting event trigger boundaries and types in text.

Our fine-tuning process suggests similar values of hyper-parameters for the models across languages in MINION. In particular, for English, we use one layer for BiLSTM mod

Figure 1: Distributions of event types in each language



ules with 300 dimensions for the hidden states (for **Transformer+BiLSTM** and **Transformer+BiLSTM+CRF**). For feed-forward networks, we employ 2 layers with 200 dimensions for the hidden vectors. The learning rate is set to \(1e\)-4 for the Adam optimizer and the batch size of 8 is employed during training.

**Monolingual Performance**: The performance of the four ED models on the test data of each language are presented in Tables 4 (for mBERT) and 5 (for XLMR). There are several observations from these tables. First, the best average F1 score of the models over different languages is 72.31% (achieved by the XLMR model). This performance is still considerably lower than a perfect model, thus suggesting significant challenges of ED in multiple languages and calling for more research effort in this area. Second, the performance of the models for non-English language is significantly worse than the English counterpart. This difference thus further highlights the necessity of more research on ED for non-English languages. Finally, the superior performance of XLMR over other models in almost all languages indicates better effectiveness of the multilingual PTLM model XLMR for ED in different languages (compared to mBERT). It also implies that traditional BiLSTM and CRF layers for sequence labeling are less necessary for multilingual ED when a PTLM is employed for text encoding. As such, in the following experiments, we will employ **Transformer** as the main ED model for further analysis.

**Monolingual PTLMs**: To better understand the benefits of multilingual PTLMs (i.e., mBERT and XLMR) for multilingual ED, we further evaluate the performance the **Transformer** model when monolingual language-specific PTLMs are leveraged to encode input texts (i.e., replacing mBERT and XLMR). Accordingly, for monolingual language-specific PTLMs, we consider both BERT-based and RoBERTa-based models for comprehensiveness. Tables 6 (for BERT) and 7 (for RoBERTa) report the monolingual performance of **Transformer** when monolingual language-specific PTLMs are employed. Note that we only show ED performance for languages where monolingual PTLMs are publicly available. As can be seen, compared to multilingual PTLMs, monolingual PTLMs (based on BERT or RoBERTa) improve the performance of **Transformer** for English. However, for other languages, monolingual PTLMs are on-par (for BERT-based models) or significantly worse (for RoBERTa-based models) than multilingual PTLMs for ED, thus demonstrating the general

\begin{table}
\begin{tabular}{l|c c c|c c c|c c c|c c c} Language & \multicolumn{3}{c|}{mBERT} & \multicolumn{3}{c|}{mBERT+CRF} & \multicolumn{3}{c|}{mBERT+BiLSTM} & \multicolumn{3}{c}{mBERT+BiLSTM+CRF} \\ \cline{2-19}  & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 \\ \hline English & 77.37 & 79.69 & 78.51 & 75.46 & 78.81 & 77.10 & 76.63 & 79.60 & 78.09 & 80.23 & 79.39 & 79.81 \\ Spanish & 71.87 & 65.56 & 68.57 & 69.77 & 64.77 & 67.18 & 65.23 & 65.35 & 65.29 & 67.39 & 66.06 & 66.72 \\ Portuguese & 80.33 & 73.50 & 76.76 & 78.75 & 72.36 & 75.42 & 76.49 & 73.52 & 74.98 & 77.46 & 73.84 & 75.61 \\ Polish & 70.42 & 68.48 & 69.43 & 69.68 & 67.03 & 68.33 & 70.25 & 65.87 & 67.99 & 72.43 & 65.38 & 68.73 \\ Turkish & 63.2 & 64.99 & 64.08 & 62.95 & 63.29 & 63.12 & 63.55 & 64.82 & 64.18 & 64.21 & 67.70 & 65.91 \\ Hindi & 73.15 & 72.19 & 72.67 & 71.83 & 73.18 & 72.50 & 70.18 & 70.08 & 70.13 & 71.38 & 72.22 & 71.80 \\ Japanese & 71.83 & 60.00 & 65.28 & 70.80 & 59.41 & 64.61 & 71.43 & 56.47 & 63.08 & 72.67 & 58.84 & 65.03 \\ Korean & 78.48 & 75.15 & 76.78 & 76.90 & 74.94 & 75.91 & 77.83 & 75.60 & 76.70 & 78.27 & 77.99 & 78.13 \\ \hline Avg. & 73.33 & 69.95 & 71.51 & 72.02 & 69.22 & 70.52 & 71.45 & 68.91 & 70.06 & 73.01 & 70.18 & 71.47 \\ \end{tabular}
\end{table}
Table 4: Performance of the ED models in the monolingual setting using mBERT on MINION.

\begin{table}
\begin{tabular}{l|c c c|c c c|c c c|c c c} Language & \multicolumn{3}{c|}{XLMR} & \multicolumn{3}{c|}{XLMR+CRF} & \multicolumn{3}{c}{XLMR+BiLSTM} & \multicolumn{3}{c}{XLMR+BiLSTM+CRF} \\ \cline{2-19}  & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 \\ \hline English & 81.45 & 77.79 & 79.58 & 80.15 & 75.79 & 77.91 & 78.46 & 74.84 & 76.61 & 79.26 & 76.18 & 77.69 \\ Spanish & 73.74 & 67.88 & 70.69 & 72.27 & 66.69 & 69.37 & 70.28 & 65.86 & 68.00 & 71.34 & 67.03 & 69.12 \\ Portuguese & 72.50 & 76.72 & 74.25 & 70.28 & 76.35 & 73.19 & 69.07 & 80.22 & 74.23 & 70.44 & 80.46 & 75.12 \\ Polish & 70.76 & 69.61 & 70.18 & 71.91 & 64.81 & 68.18 & 70.37 & 65.24 & 67.71 & 72.30 & 66.76 & 69.42 \\ Turkish & 63.66 & 66.71 & 65.15 & 62.18 & 65.99 & 64.03 & 64.30 & 60.15 & 62.16 & 65.12 & 62.07 & 63.56 \\ Hindi & 77.04 & 68.87 & 72.72 & 76.54 & 66.74 & 71.31 & 76.69 & 65.76 & 70.81 & 75.39 & 68.60 & 71.84 \\ Japanese & 72.30 & 63.69 & 67.72 & 71.10 & 61.58 & 66.00 & 70.89 & 63.92 & 67.23 & 72.64 & 60.77 & 66.18 \\ Korean & 74.32 & 82.42 & 78.16 & 73.16 & 81.35 & 77.04 & 72.69 & 78.31 & 75.40 & 73.00 & 79.34 & 76.04 \\ \hline Avg. & 73.22 & 71.71 & 72.31 & 72.20 & 69.91 & 70.88 & 71.59 & 69.29 & 70.27 & 72.44 & 70.15 & 71.12 \\ \end{tabular}
\end{table}
Table 5: Performance of the ED models in the monolingual setting using XLMR on MINION.

 

advantage of multilingual PTLMs for ED. In addition, it is suggestive that future work can explore methods to improve monolingual language-specific PTLMs for ED in different languages.

**Cross-lingual Performance**: To understand the transferability of ED knowledge and annotation across languages, we explore the cross-lingual evaluation setting where models are trained on English data (the source language) and directly evaluated on test data of other target languages in MINION. As such, we report the cross-lingual performance of **Transformer** with both mBERT and XLMR as the PTLMs in Table 8. Note that we inherit the same hyper-parameters selected for **Transformer** in the fine-tuning process of monolingual experiments for consistency.

Compared to the monolingual performance counterparts of mBERT and XLMR in Tables 4 and 5, it is clear that the performance of **Transformer** in non-English languages decreases significantly in the cross-lingual evaluation, i.e., the average performance loss due to cross-lingual evaluation is 15.2% for both mBERT and XLMR. We also observe a wide range of cross-lingual performance for the target languages in Table 8, thus suggesting the diverse nature of the data and languages in MINION to support robust model development for ED. Among the target languages, Portuguese exhibits the smallest performance difference between monolingual and cross-lingual settings while the largest performance loss with cross-lingual transfer occurs in Japanese, Turkish, Korean, and Hindi. One possible reason for such performance loss is due to the language structure difference where Japanese, Turkish, Korean, and Hindi follow the Subject-Object-Verb word order while English and other languages in our dataset utilize the Subject-Verb-Object order. Another reason can be linked to different patterns/distributions of event triggers in different languages. For instance, some languages tend to mention the events using verbs (e.g., in English 78% of the triggers are verb) while other languages might use more diverse parts of speech to express event trigger (e.g., in Japanese only 63% of triggers are verbs). Also, Section 4 provide an additional explanation regarding the diversity of event triggers in different languages. In all, the cross-lingual performance in our MINION dataset demonstrates the challenges of transferring ED knowledge across languages that can be further studied in future work.

## 4 Analysis

This section provides additional analysis to better understand the multilingual ED task in MINION.

**Cross-dataset Evaluation**: As the event ontology in MINION is inherited and pruned from the ACE 2005 dataset, it is helpful to learn how the annotated events in MINION is different from those in ACE 2005. To this end, we propose to evaluate model performance on the cross-dataset setting: models are trained on the English data of ACE 2005 and evaluated on test data of different languages in MINION. In particular, we utilize the standard data split from prior work Nguyen and Grishman (2015); Chen et al. (2015); Wang et al. (2019) to obtain English training and development data in ACE 2005. Also, we filter the ACE 2005 data so only triggers of event sub-types in our MINION

\begin{table}
\begin{tabular}{l|c c c} Language & P & R & F1 \\ \hline English Devlin et al. (2019) & 78.12 & 81.61 & 79.83 \\ Spanish Canéte et al. (2020) & 72.73 & 62.25 & 67.08 \\ Portuguese Souza et al. (2020) & 81.82 & 72.00 & 76.60 \\ Polish Kleczek (2021) & 71.79 & 65.00 & 68.23 \\ Turkish MDZ (2021) & 67.75 & 60.57 & 63.96 \\ \end{tabular}
\end{table}
Table 6: Test data performance of **Transformer** in the monolingual setting using available language-specific monolingual setting using available language-specific BERT models on MINION. The citations indicate the sources of the language-specific BERT.

\begin{table}
\begin{tabular}{l|c c c c} Language & \multicolumn{2}{c}{mBERT} & \multicolumn{2}{c}{XLMR} \\ \cline{2-5}  & P & R & F1 & P & R & F1 \\ \hline English & 77.37 & 79.69 & 78.51 & 81.45 & 77.79 & 79.58 \\ \hline Spanish & 74.32 & 54.14 & 62.64 & 78.27 & 52.48 & 62.83 \\ Portuguese & 70.79 & 71.50 & 71.14 & 73.36 & 72.18 & 72.77 \\ Polish & 73.83 & 49.84 & 59.51 & 79.55 & 48.33 & 60.13 \\ Turkish & 69.25 & 35.14 & 46.62 & 66.84 & 36.49 & 47.21 \\ Hindi & 66.10 & 51.66 & 57.99 & 64.74 & 52.84 & 58.19 \\ Japanese & 52.44 & 25.29 & 34.13 & 55.39 & 25.71 & 35.12 \\ Korean & 67.24 & 42.27 & 55.52 & 80.69 & 43.80 & 56.78 \\ \end{tabular}
\end{table}
Table 8: Cross-lingual performance of **Transformer** that is trained on English training data and evaluated on test data of other languages in MINION.

 

dataset are retained for a compatibility between two datasets. Due to its superior performance in previous experiments, we employ the **Transformer** model with XLMR in this experiment. The hyperparameters for the model is fine-tuned on the development data of ACE 2005. Table 9 shows the model performance in the cross-dataset evaluation. Compared to the corresponding cross-lingual performance of MINION in Table 8, it is clear that the performance on MINION is significantly worse when the model is trained on ACE 2005 data. As such, a possible explanation for this performance loss includes domain difference between ACE 2005 and MINION, i.e., MINION involve Wikipedia articles while ACE 2005 is based on news articles, conversational telephone speeches, and others. In addition, as the size of English training data in MINION (i.e., over 14K triggers) is significantly larger than those for ACE 2005 (i.e., less than 6K triggers), the training data in MINION might cover more event patterns to produce better performance for ED models. Future work can explore this cross-dataset evaluation setting to build more robust models for ED.

**Trigger Diversity in Different Languages**: To understand how events are expressed in different languages, we explore the ratio of unique trigger words over the total number of event triggers for an event sub-type (called unique ratio). Figure 2 shows the averages of unique ratios over event sub-types for different languages in our MINION dataset. As such the diagram shows that English is relatively simpler than other languages in ED as its diversity of event triggers for event types is the least among all the considered languages. Korean, Turkish, and Japanese are the languages that exhibit the largest diversities of event triggers. This further helps to explain the worst cross-lingual performance of models from English to Korean, Turkish, and Japanese in Table 8.

**Challenging the Supremacy of English for Event Detection**: English has been the major language for ED research. In particular, in cross-lingual transfer learning for ED, English has often been considered as a high-resource source language to train ED models to apply to other target languages [37, 3]. In this experiment, we argue that English is not necessary the optimal source language for cross-lingual transfer learning of ED. In particular, using **Transformer** with XLMR as the base model, we train the model on the training data of both English and Spanish; the resulting models are evaluated on the test data of the other languages in MINION. To ensure a fair comparison, we use the same size of training data for English and Spanish, i.e., 3,000 annotated text segments randomly sampled in MINION. Table 10 presents the cross-lingual performance of the models. The table demonstrates that using Spanish as the source language can achieve better performance than English for all the target languages in MINION. As such, our findings suggest that choosing appropriate source languages for cross-lingual transfer learning of ED is important and can be further explored in future work.

## 5 Related Work

Early attempts for ED have employed feature-based models [16, 17, 18, 19, 20, 21, 22] while deep learning has recently been proven to be a better approach for ED [23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 222, 213, 214, 215, 216, 217, 218, 219, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 282, 284, 286, 288, 289, 291, 285, 287, 288, 289, 292, 293, 294, 295, 296, 297, 298, 299, 299, 300, 31, 320, 332, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 61, 62, 63, 64, 65, 66, 67, 68, 69, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 19, 19, 19, 19, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 29, 31, 32, 33, 34, 35, 36, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 51, 52, 54, 53, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 72, 74, 75, 76, 78, 79, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 19, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 39, 40, 41, 42, 43, 44, 45, 46, 48, 49, 50, 51, 52, 53, 54, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 73, 74, 75, 76, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 19, 19, 20, 21, 23, 24, 25, 26, 27, 28, 29, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 45, 46, 47, 48, 49, 52, 54, 55, 56, 57, 58, 59, 61, 62, 63, 64, 65, 66, 67, 68, 69, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 19, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 39, 40, 41, 42, 43, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 56, 57, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 74, 75, 76, 78, 79, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 19, 19, 21, 22, 23, 24, 25, 26, 27, 28, 29, 33, 34, 35, 36, 37, 39, 40, 41, 42, 43, 45, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 59, 

Pouran Ben Veyseh et al., 2021a). There have also been recent efforts on creating new datasets for ED for different domains, including biomedical texts Kim et al. (2009), literary texts Sims et al. (2019), cybersecurity texts Satyapanich et al. (2020); Man et al. (2020), Wikipedia texts Wang et al. (2020), fine-grained event types Le and Nguyen (2021), and historical texts Lai et al. (2021b). However, such prior works and datasets for ED are mainly devoted to English, ignoring challenges in many non-English languages. Non-English datasets for ED also exist Kobylinski and Wasiluk (2019); Sahoo et al. (2020); however, these datasets are only annotated for one language with divergent ontology and annotation guidelines, thus unable to support comprehensive studies and transferability research for ED on multiple languages.

Existing ED datasets that cover multiple languages involve ACE 2005 Walker et al. (2006), TAC KBP Mitamura et al. (2016, 2017), and TempEval-2 Verhagen et al. (2010). Among such datasets, ACE 2005 is the most popular dataset used in prior multilingual/cross-lingual ED research Chen and Ji (2009); M'hamdi et al. (2019); Ahmad et al. (2021); Nguyen et al. (2021c); Nguyen and Nguyen (2021). However, such multilingual datasets suffer from the issues of small data size, limited language coverage with greater focus on popular languages, and inaccessibility to the public as discussed in the introduction. Finally, we also note some prior works that claim event detection datasets for non-English datasets Im et al. (2009); Kucuk and Yazici (2011); Lejeune et al. (2015). However, such datasets are not comparable to our dataset as their event detection task is indeed a sentence classification problem where established definition of events with event triggers are not followed and annotated.

## 6 Conclusion

We introduce a new dataset for ED in 8 typologically different languages. The dataset is significantly larger and covers more and newer languages than prior resources. Specifically, 31,226 text segments from language-specific articles of Wikipedia are manually annotated in the dataset. Our experiments and analysis demonstrate the high quality of the dataset and the multilingual challenges of ED, providing ample room for future research in this direction. In the future, we will extend the dataset to include event argument annotations.

## Acknowledgement

This research has been supported by the Army Research Office (ARO) grant W911NF-21-1-0112 and the NSF grant CNS-1747798 to the IUCRC Center for Big Learning. This research is also based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the Better Extraction from Text Towards Enhanced Retrieval (BETTER) Program. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ARO, ODNI, IARPA, the Department of Defense, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This document does not contain technology or technical data controlled under either the U.S. International Traffic in Arms Regulations or the U.S. Export Administration Regulations. We thank the anonymous reviewers and Tracy King for their helpful feedback.

## References

* Ahmad et al. (2021) Wasi Uddin Ahmad, Nanyun Peng, and Kai-Wei Chang. 2021. Gate: Graph attention transformer encoder for cross-lingual relation and event extraction. In _Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI)_.
* Ahn (2006) David Ahn. 2006. The stages of event extraction. In _Proceedings of the Workshop on Annotating and Reasoning about Time and Events_.
* Attardi (2015) Giusepppe Attardi. 2015. Wikiextractor. [https://github.com/attardi/wikiextractor](https://github.com/attardi/wikiextractor).

Figure 2: The average ratios of unique event triggers over event sub-types for each language in MINION.

 

Jose Canete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Perez. 2020. Spanish pre-trained bert model and evaluation data. In _PMLADC at International Conference on Learning Representations (ICLR) 2020_.
* Chen et al. (2015) Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Chen and Ji (2009) Zheng Chen and Heng Ji. 2009. Can one language bootstrap the other: A case study on event extraction. In _Proceedings of the NAACL-HLT 2009 Workshop on Semi-supervised Learning for Natural Language Processing_.
* C. Clarkin-PL CLARIN-PL (2021) CLARIN-PL. 2021. Polish roberta.
* Conneau et al. (2020) Alexis Conneau, Kartiaky Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
* Cui et al. (2020) Shiyao Cui, Bowen Yu, Tingwen Liu, Zhenyu Zhang, Xuebin Wang, and Jinqiao Shi. 2020. Edge-enhanced graph convolution networks for event detection with syntactic relation. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 2329-2339, Online. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ebner et al. (2020) Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence argument linking. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
* Hong et al. (2011) Yu Hong, Jianfeng Zhang, Bin Ma, Jianmin Yao, Guodong Zhou, and Qiaoming Zhu. 2011. Using cross-entity inference to improve event extraction. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Im et al. (2009) Seohyun Im, Hyunjo You, Hayun Jang, Seungho Nam, and Hypoil Shin. 2009. KTimeML: Specification of temporal and event expressions in Korean text. In _Proceedings of the 7th Workshop on Asian Language Resources (ALR7)_.
* Ji and Grishman (2008) Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Kim et al. (2009) Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Jun'ichi Tsuji. 2009. Overview of BioNLP'09 shared task on event extraction. In _Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task_.
* Kobylinski and Wasiluk (2019) Lukasz Kobylinski and Michal Wasiluk. 2019. Deep learning in event detection in Polish. In _Proceedings of the 10th Global Wordnet Conference_, pages 216-221, Wroclaw, Poland. Global Wordnet Association.
* Krippendorff (2011) Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
* Kucuk and Yazici (2011) Dilek Kucuk and Adnan Yazici. 2011. Exploiting information extraction techniques for automatic semantic video indexing with an application to turtish news videos. In _Knowl. Based Syst._
* Kleczek (2021) Darek Kleczek. 2021. Polish bert.
* Lai et al. (2021a) Viet Lai, Franck Dernoncourt, and Thien Huu Nguyen. 2021a. Learning prototype representations across few-shot tasks for event detection. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5270-5277, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Lai et al. (2021b) Viet Lai, Minh Van Nguyen, Heidi Kaufman, and Thien Huu Nguyen. 2021b. Event extraction from historical texts: A new dataset for black rebellions. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 2390-2400, Online. Association for Computational Linguistics.
* Lai et al. (2020) Viet Dac Lai, Tian Ngo Nguyen, and Thien Huu Nguyen. 2020. Event detection: Gate diversity and syntactic importance scores for graph convolution neural networks. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5405-5411, Online. Association for Computational Linguistics.
* Le and Nguyen (2021) Duong Le and Thien Huu Nguyen. 2021. Fine-grained event trigger detection. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2745-2752, Online. Association for Computational Linguistics.
* Lejeune et al. (2015) Gael Lejeune, Romain Brixtel, Antoine Doucet, and Nadine Lucas. 2015. Multilingual event extraction for epidemic detection. In _Artif. Intell. Medicine_.
* Li et al. (2013) Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_ 

Shasha Liao and Ralph Grishman. 2010. Filtered ranking for bootstrapping in event extraction. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010).
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
* Man et al. (2020) Duc Trong Hieu Man, Duc Trong Le, Amir Pouran Ben Veyseh, Thuat Nguyen, and Thien Huu Nguyen. 2020. Introducing a new dataset for event detection in cybersecurity texts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
* Lam (2021) Digital Library team MDZ. 2021. Turkish bert.
* M'hamdi et al. (2019) Meyrem M'hamdi, Marjorie Freedman, and Jonathan May. 2019. Contextualized cross-lingual event trigger extraction with minimal resources. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL).
* Mitamura et al. (2016) Teruko Mitamura, Zhengzhong Liu, and Eduard H. Hovy. 2016. Overview of TAC-KBP 2016 event nugget track. In Proceedings of the Text Analysis Conference (TAC).
* Mitamura et al. (2017) Teruko Mitamura, Zhengzhong Liu, and Eduard H. Hovy. 2017. Events detection, coreference and sequencing: What's next? overview of the TAC KBP 2017 event track. In Proceedings of the Text Analysis Conference (TAC).
* Miwa et al. (2014a) Makoto Miwa, Paul Thompson, Ioannis Korkontzelos, and Sophia Ananiadou. 2014a. Comparable study of event extraction in newswire and biomedical domains. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers.
* Miwa et al. (2014b) Makoto Miwa, Paul Thompson, Ioannis Korkontzelos, and Sophia Ananiadou. 2014b. Comparable study of event extraction in newswire and biomedical domains. In Proceedings of the International Conference on Computational Linguistics (COLING).
* MMG MMG (2021) MMG MMG. 2021. Spanish roberta.
* Tung et al. (2021) Nghia Ngo Trung, Duy Phung, and Thien Huu Nguyen. 2021. Unsupervised domain adaptation for event detection using domain-specific adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online. Association for Computational Linguistics.
* Van Nguyen et al. (2021a) Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. 2021a. Transkit: A light-weight transformer-based toolkit for multilingual natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations.
* van Nguyen and Nguyen (2021) Minh Van Nguyen and Thien Huu Nguyen. 2021. Improving cross-lingual transfer for event argument extraction with language-universal sentence structures. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
* Nguyen et al. (2021b) Minh Van Nguyen, Twan Ngo Nguyen, Bonan Min, and Thien Huu Nguyen. 2021b. Crosslingual transfer learning for relation and event extraction via word category and class alignments. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5414-5426, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Nguyen et al. (2021c) Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
* Nguyen and Grishman (2015) Thien Huu Nguyen and Ralph Grishman. 2015. Event detection and domain adaptation with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers).
* Nguyen and Grishman (2018) Thien Huu Nguyen and Ralph Grishman. 2018. Graph convolutional networks with argument-aware pooling for event detection. In Proceedings of the AAAI Conference on Artificial Intelligence.
* Parmar (2021) Suraj Parmar. 2021. Hindi roberta.
* Passonneau (2006) Rebecca Passonneau. 2006. Measuring agreement on set-valued items (masi) for semantic and pragmatic annotation. In The International Conference on Language Resources and Evaluation (LREC).
* Patwardhan and Riloff (2009) Siddharth Patwardhan and Ellen Riloff. 2009. A unified model of phrasal and sentential evidence for information extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
* Devyesh et al. (2021a) Amir Pouran Ben Veyseh, Viet Lai, Franck Dernoncourt, and Thien Huu Nguyen. 2021a. Unleash GPT-2 power for event detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online. Association for Computational Linguistics 

Amir Pouran Ben Veyseh, Minh Van Nguyen, Ngha Ngo Trung, Bonan Min, and Thien Huu Nguyen. 2021b. Modeling document-level context for event detection via important context selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Cited by: SS4.
* S. Kumar Sahoo, S. Saha, A. Ekbal, and P. Bhattacharyya (2020)A platform for event extraction in Hindi. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, pp. 2241-2250. External Links: Link, Document Cited by: SS3.
* T. Satyapanich, F. Ferraro, and Tim Finin (2020)Casie: extracting cybersecurity event information from text. In Proceedings of the AAAI conference on artificial intelligence, Cited by: SS4.
* L. Sha, F. Qian, B. Chang, and Z. Sui (2018)Jointly extracting event triggers and arguments by dependency-bridge rnn and tensor-based argument interaction. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI), Cited by: SS4.
* M. Sims, J. Ho Park, and D. Bamman (2019)Literary event detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3623-3634. External Links: Link, Document Cited by: SS4.
* F. Souza, R. Nogueira, and R. Lotufo (2020)BERTimbau: pretrained BERT models for Brazilian Portuguese. In 9th Brazilian Conference on Intelligent Systems, BRACIS, Rio Grande do Sul, Brazil, October 20-23, Cited by: SS4.
* M. Verhagen, R. Sauri, T. Caselli, and J. Pustejovsky (2010)SemEval-2010 task 13: tempeval-2. In Proceedings of the 5th International Workshop on Semantic Evaluation, Cited by: SS4.
* C. Walker, S. Strassel, J. Medero, and K. Maeda (2006)ACE 2005 multilingual training corpus. In Technical report, Linguistic Data Consortium, Cited by: SS4.
* X. Wang, X. Han, Z. Liu, M. Sun, and P. Li (2019)Adversarial training for weakly supervised event detection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 998-1008. External Links: Link, Document Cited by: SS4.
* X. Wang, Z. Wang, X. Han, W. Jiang, R. Han, Z. Liu, J. Li, P. Li, Y. Lin, and J. Zhou (2020)Maven: a massive general domain event detection dataset. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: SS4.
* W. Wongso (2021)Japanese roberta. Cited by: SS4.
* B. Yang and T. M. Mitchell (2016)Joint extraction of events and entities within a document context. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Cited by: SS4.
* S. Yang, D. Feng, L. Qiao, Z. Kan, and D. Li (2019)Exploring pre-trained language models for event extraction and generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 5284-5294. External Links: Link, Document Cited by: SS4.

 

Event Types in MINION

There are 16 event types annotated in the proposed MINION dataset. Table 11 shows the event types along with their description and examples. We inherit event type definition and examples from ACE annotation guideline2(Walker et al., 2006).

 



\begin{table}
\begin{tabular}{p{42.7pt}|p{310.8pt}|p{56.9pt}|p{142.3pt}} \hline ID & Type\_SetType & Description & Example triggers are highlighted) \\ \hline
1 & Life\_Be-Bora & A BE-BORN Event occurs whenever a PERSON Entity is given birth to, Please note that we do not include the birth of other things or ideas. & Juste Due was born in Casper, Wyoming on March 18, 1964. \\ \hline
2 & Life\_Mary & MARRY Events are official Events, where two people are married under the legal GATA. & Juste Doc and John Smith were **married** on June 9, 1998. \\ \hline
3 & Life\_Dwece & A DYNACE Event occurs whenever two people are officially observed under the legal destination of divorce. We do not include separations or church amendments. & The couple **diovered** four years later. \\ \hline
4 & Life\_Iwhere & An INDURE Event occres whenever a PERSON Entity experiences physical harm: INDURE Events can be accidental, intentional or self-inflicted. & Fora softer were **vounded** in the attack. \\ \hline
5 & Life\_De & A DIE Even occurs whenever the title of a PIRSON fairly ends. DIF Events can be accidental, intentional or self-inflicted & Forecast groups have threatened to **kill** foreign hostages. \\ \hline
6 & Movement\_Transport & A TRANSPERON receiver as ARTIENT (WEAPON or VEHULED or a PERSON is moved from ten PLACE (GPE, FACILITY, LOCATION) in another. & But there is no a second kind retapped over Russia’s declaration that it will **still** weapons to link, contrary to the earlier male agreement. \\ \hline
7 & Transaction\_Transformer & TRANSFERON/WRESHUP Errors refer to the giving, receiving, borrowing, or looking money when it is not the context of purchasing something. The content considers an (1) people getting money to information (and getting nothing mobile return; and (2) organizations leading money) to people or other ones. & There is also a second kind retapped over Russia’s declaration that it will **still** weapons to link, contrary to the earlier male agreement. \\ \hline
8 & Transaction\_Transformer & TRANSFERON/WRESHUP Errors refer to the buying, willing, losing, browsing, going, or receiving of antifacts or organizations. & This new **perschued** two nuclear submutments from Russia. \\ \hline
8 & Transaction\_Transformer & TRANSFERON/WRESHUP Errors refer to the giving, receiving, borrowing, or looking money when it is not the context of purchasing something. The content considers an (1) people getting money to information (and getting nothing mobile return; and (2) organizations leading money) to people or other ones. & The charity was suspected of **giving** money may at Al Quads. \\ \hline
9 & Conflict\_Attack & A ATACK Event is defined as a violent (physical act causing harm or damage; ATTACK scenario includes any health event as covered by the INPUT®