SlowGuess/NAACL_2022_OCR huggingface dataset:
Amir Pouran Ben Veyseh<sup>1</sup>, Minh Van Nguyen<sup>1</sup>, Franck Dernoncourt<sup>2</sup>, and Thien Huu Nguyen<sup>1</sup>
{apouranb,minhny,Thien}@cs.uoregon.edu,
dernonco@adobe.com
Event Detection (ED) is the task of identifying and classifying trigger words of event mentions in text. Despite considerable research efforts in recent years for English text, the task of ED in other languages has been significantly less explored. Switching to non-English languages, important research questions for ED include how well existing ED models perform on different languages, how challenging ED is in other languages, and how well ED knowledge and annotation can be transferred across languages. To answer those questions, it is crucial to obtain multilingual ED datasets that provide consistent event annotation for multiple languages. There exist some multilingual ED datasets; however, they tend to cover a handful of languages and mainly focus on popular ones. Many languages are not covered in existing multilingual ED datasets. In addition, the current datasets are often small and not accessible to the public. To overcome those shortcomings, we introduce a new large-scale multilingual dataset for ED (called MINION) that consistently annotates events for 8 different languages; 5 of them have not been supported by existing multilingual datasets. We also perform extensive experiments and analysis to demonstrate the challenges and transferability of ED across languages in MINION that in all call for more research effort in this area. We will release the dataset to promote future research on multilingual ED.
Event Detection (ED) is one of the critical steps for an Event Extraction system in Information Extraction (IE) that aims to recognize mentions of events in text, i.e., change of state of real world entities. Specifically, an ED system identifies the word(s) that most clearly refer to the occurrence of an event, i.e., event trigger, and also detects the type of event that is evoked by the event trigger. For instance, in the sentence "The city was reportedly
struck by F16 missiles.", the word "struck" is the trigger for an ATTACK event. An ED model can be incorporated into other IE pipelines to facilitate the extraction of information related to events and entities, thereby supporting various downstream applications such as knowledge base construction, question answering and text summarization.
Due to its importance, ED has been extensively studied in the IE and NLP community over the past decade. Existing methods for ED extend from feature-based models (Ahn, 2006; Liao and Grishman, 2010; Miwa et al., 2014a), to advanced deep learning methods (Nguyen and Grishman, 2015; Chen et al., 2015; Sha et al., 2018; Wang et al., 2019; Yang et al., 2019; Cui et al., 2020; Lai et al., 2020; Pouran Ben Veyseh et al., 2021b). As such, the creation of large annotated datasets for ED, e.g., ACE 2005 (Walker et al., 2006), has been critical to progress measurement and growing development of ED research. However, a majority of current datasets for ED only provide annotation for texts in a single language (i.e., monolingual datasets). For instance, the recent challenging datasets for ED, e.g., MAVEN (Wang et al., 2020), RAMS (Ebner et al., 2020), or CySecED (Man et al., 2020), are all proposed for English documents only. In addition, there are a few existing datasets that include ED annotation for multiple languages (multilingual datasets), e.g., ACE 2005 (Walker et al., 2006), TAC KBP (Mitamura et al., 2016, 2017), and TempEval-2 (Verhagen et al., 2010). However, those multilingual datasets only cover a handful of languages (i.e., 3 languages in ACE 2005 and TAC KBP, and 6 languages in TempEval-2), mainly focusing on popular languages such as English, Chinese, Arabic, and Spanish, and leaving many other languages unexplored for ED. For instance, Turkish and Polish are not covered in existing multilingual datasets for ED. We also note that existing ED datasets tend to employ different annotation schema and guidelines that prevent the combina
Human Language Technologies, pages 2286 - 2299
tion of current datasets to create a larger one. In all, the limited coverage of languages and annotation discrepancy in current monolingual/multilingual ED datasets hinder comprehensive studies for the challenges of ED in diverse languages. It also limits thorough evaluations for multilingual generalization of ED models. Finally, we note that the major multilingual datasets for ED are not publicly accessible due to the licence of involving documents, e.g., ACE 2005 and TAC KBP, thus further impeding research effort in this area.
To address such issues, our goal is to introduce a new Multilingual Event Detection dataset (called MINION) to support multilingual research for ED. In particular, we provide a large-scale dataset that manually annotates event triggers for 8 typologically different languages, i.e., English, Spanish, Portuguese, Polish, Turkish, Hindi, Japanese and Korean. Among them, the five languages Portuguese, Polish, Turkish, Hindi, and Japanese are not covered in existing popular datasets for multilingual ED (i.e., ACE 2005, TAC KBP, and TempEval-2). To facilitate public release and sharing of the dataset, we employ the event articles from Wikipedia for annotation in 8 languages. In addition, to improve quality of the data, we inherit the annotation schema and guideline in ACE 2005, the well-designed and widely-used dataset for ED research. In total, our MINION dataset involves more than 50K annotated event triggers, which is much larger than those in existing multilingual ED datasets (i.e., less than 11K and 27K in ACE 2005 and TempEval-2 respectively). We expect that the significantly larger size with more diverse set of languages and public texts in MINION can contribute to accelerate and extend research in ED to a larger population.
Given the proposed dataset, we conduct thorough analysis on MINION using the state-of-the-art (SOTA) models for ED. In particular, we first study the challenges of ED in different languages using monolingual evaluations where ED models are trained and tested in the same languages. Our experiments suggest that the performance of existing ED models is not yet satisfactory in multiple languages and the model performance on non-English languages is in general poorer than those for English. We also show that current pre-trained language models for specific languages (i.e., monolingual models) are less effective for ED models than multilingual pre-trained language models, e.g.,
mBERT (Devlin et al., 2019). In all, our findings highlight greater challenges of ED for non-English languages that should be further pursued in future research.
In addition, our MINION dataset also facilitate zero-shot cross-lingual transfer learning experiments that serve to reveal the transferability of ED knowledge and annotation across languages. In these experiments, ED models are trained on English data (the source language), but tested in other target languages. Our results in this setting demonstrate a wide range of cross-lingual performance for different target languages in MINION that introduces a diverse set of languages and data for ED research. Finally, we report extensive analysis on MINION to provide further data insights for future ED research, including challenges of data annotation, language differences, and cross-dataset evaluation. We will release MINION to foster future research for multilingual ED.
Our dataset MINION follows the same definition of events as the annotation guideline in ACE 2005 (Walker et al., 2006). Specifically, an event is defined as an occurrence that results in the change of state of a real world entity. Moreover, an event mention is evoked by an event trigger which most clearly describes the occurrence of the event. While event triggers are mostly single words, we also allow multi-word event triggers to better accommodate ED annotation in multiple languages. For instance, the phrasal verb "tayin etmek" with two words in Turkish, meaning "appoint", is necessary to express the event type Start-Position.
We also inherit the annotation schema/ontology (i.e., to define event types for annotation) and guideline in ACE 2005 to benefit from its well-designed documentation and be consistent with most of prior ED research. However, to improve the quality of the annotated data, we prune some event sub-types from the original ACE 2005 ontology in our dataset. In particular, event sub-types that have very similar meanings in some language are not included in our final ontology. This promotes the distinction between event labels and avoids confusion for annotators to provide high-quality data in different languages. For instance, the event sub-types Convict and Sentence are very similar in Turkish (i.e., both Convict and Sentence can be translated as Mahkum etmek in Turkish), thus being removed
in our ontology. In addition, we also exclude event sub-types in ACE 2005 that are not frequent in our collected data from Wikipedia (more details on data collection later), e.g., Nominate and Declare-Bankruptcy. Finally, 16 event sub-types (for 8 event types) are preserved in the final event schema for our dataset. We provide detailed explanation and sample sentences for the event types in our dataset in the Appendix A.
2.1 Candidate Selection
As mentioned in the introduction, we aim to annotate ED data for 8 languages, i.e., English, Spanish, Portuguese, Polish, Turkish, Hindi, Japanese and Korean. These languages are selected due to their diversity in term of typology and novelty w.r.t. to existing multilingual ED datasets that can be helpful for multilingual model development and generalization evaluation. To collect text data for annotation in each language, we employ the articles of the language-specific editions of Wikipedia. Specifically, for each language, we obtain its latest dump of Wikipedia articles<sup>1</sup>, then process the dump with the parser WikiExtractor (Attardi, 2015) to extract textual and meta data for articles. To increase the likelihood of encountering event mentions for effective annotation, we utilize the articles that are classified under one of the sub-categories of the Event category in Wikipedia. In particular, we focus on six sub-categories Economy, Politics, Technology, Crimes, Nature, and Military due to their relevance to the event types in our ontology. Note that we map these (sub)categories in English to the corresponding (sub)c categories in other languages using the provided links in Wikipedia. Afterward, to split the texts into sentences and tokens, we leverage the multilingual toolkit Trankit (Nguyen et al., 2021a) that has demonstrated state-of-the-art performance for such tasks in our languages.
Given a Wikipedia article, an approach for ED annotation is to ask the annotators to annotate the entire document for event triggers at once. However, as Wikipedia articles tend to be long, this approach might be overwhelming for annotators, thus potentially limiting the annotation quality. To this end, motivated by the annotation with 5-sentence windows in the RAMS dataset (Ebner et al., 2020), we split each article into segments of 5 sentences that will be annotated separately by annotators. In this way, annotators only need to process a shorter
context at a time to improve the attention and accuracy of annotated data. This annotation approach is also supported by a large amount of prior ED research where a majority of previous ED models have employed context information in single sentences to deliver high extraction performance for the event types in ACE 2005 (Nguyen and Grishman, 2015, 2018; Wang et al., 2019; Yang et al., 2019; Cui et al., 2020), including models for multiple languages (M'hamdi et al., 2019; Ahmad et al., 2021; Nguyen et al., 2021b).
2.2 Annotation Process
To annotate the produced article segments, we hire annotators from upwork.com, a crowd-sourcing platform with freelancer annotators across the globe. In particular, our annotator candidate pool for each language of interest involves native speakers of the language who also have experience on related data annotation projects (e.g., for named entity recognition), an approval rate higher than  $95\%$ , and fluency in English. These information is provided by annotator profiles in Upwork. In the next step, the candidates are trained for ED annotation using the English annotation guideline and examples for the designed event schema in our dataset (i.e., inherited from ACE 2005). Finally, we ask the candidates to take an annotation test designed for ED in English and only candidates with passing results are officially selected for the annotators of our multilingual ED dataset. Overall, we recruit several annotators for each language of interest as shown in Table 2. To prepare for the actual annotation, the annotators for each language will work together to produce a translation of the English annotation guideline/examples where language-specific annotation rules are discussed and included in the translated guideline to form common annotation perception for the language. The translated guideline and examples are also verified by our language experts to avoid any potential conflicts and issues.
Finally, given the language-specific guidelines, the annotators for each language will independently annotate a chunk of article segments for that language. The breakdown numbers of annotated text segments for each language and Wikipedia subcategory in our MINION dataset are shown in Table 3. As such,  $20\%$  of the annotated text segments for each language is selected for co-annotation by the annotators to measure inter-annotator agree
${}^{1}$  Dumps were downloaded in May 2021.
Category	English	Spanish	Portuguese	Polish	Turkish	Hindi	Japanese	Korean
Economy	1,095	112	168	315	297	189	199	250
Politics	3,202	308	772	1,270	1,233	349	232	248
Technology	2,171	189	400	712	815	295	312	249
Crimes	893	78	220	152	118	95	80	73
Nature	1,195	398	705	455	398	245	299	185
Military	4,444	415	1,003	1,575	1,619	326	378	495
Total	13,000	1,500	3,268	4,479	4,480	1,499	1,500	1,500
Table 1: Numbers of annotated segments in each Wikipedia subcategory for the 8 languages.
Language	#Annotator	IAA
English	10	0.834
Spanish	10	0.812
Portuguese	5	0.803
Polish	8	0.799
Turkish	10	0.813
Hindi	6	0.803
Japanese	5	0.789
Korean	6	0.810
Table 2: Agreement scores for 8 languages in MINION.
ment (IAA) scores while the remaining  $80\%$  is distributed to annotators for separate annotation. Table 2 reports the Krippendorff's alpha (Krippendorff, 2011) with MASI distance metric (Passonneau, 2006) for the IAA scores of each language in our dataset. After independent annotation, the annotators will resolve the conflict cases to produce the final version of our MINION dataset. Overall, our dataset demonstrates high agreement scores for all the 8 languages, thus providing a high-quality dataset for multilingual ED.
The main statistics for our MINION dataset is provided in Table 3. This table shows that for a majority of languages, there are multiple event triggers in a text segment, thereby introducing a challenge for ED models due to the overlap of event context. In addition, the table shows that text segments in some languages are more replete with event mentions than those for other languages. Specifically, comparing Polish and English text segments, the density of event mentions in Polish is almost two times more than that for English. Finally, Figure 1 shows the distributions of 8 event types for the 8 languages in our dataset. As can be seen, the languages in our dataset tend to involve different levels of discrepancy regarding the distributions over event types. As such, the type density and distribution divergence between languages suggest
other challenges that robust ED models should handle to perform well across languages in MINION.
2.4 Annotation Challenges
Despite the high inter-annotator agreement scores, there are some conflicts between our annotators during the annotation process due to the ambiguity of event triggers, especially in the multilingual setting. This section highlights some of the key ambiguities/conflicts that we encounter during our analysis of annotation results from the annotators. Note that all of these conflicts have been resolved by the annotators in the final version of our dataset.
Language-Specific Challenges: Despite common notion of events in different languages, each language might has its own exceptions regarding how an event trigger should be annotated, causing confusions/conflict for our annotators in the annotation process. One exception concerns the necessity to include event arguments in the annotation of an event trigger in some language. For example, in the Polish sentence "Samolot sie rozbil" (translated as "The plane crashed itself"), some annotators believe that the meaning of the verb "rozbil" (i.e., crashed) is incomplete if its argument word "sie" (i.e., itself) is not associated. As such, annotating both the verb and its argument (i.e., "sie rozbil") is necessary to express an event in this case. However, other annotators suggest that only annotating the word "rozbil" is sufficient. Our annotators have decided to annotate event triggers along with necessary arguments to achieve their complete meanings in such cases.
Background Knowledge: Background knowledge is sometime important to correctly recognize an event trigger in input text. In such cases, the annotators might have conflicting event annotation decisions for a word as their levels of background knowledge are different. For instance, in the sentence "The match was canceled in the memory of victims of Katyn crime", some annotators annotate the word "crime" as a Die event trigger as they
Language	#Seg.	Avg. Length	#Triggers	Avg. #Trigger/Seg.	Most Frequent Types	Challenging Type	Language Family
English	13,000	123	17,644	1.35	Life, Conflict, Movement	Personnel	Germanic
Spanish	3,268	112	6,063	1.85	Personnel, Life, Conflict	Conflict	Italic
Portuguese	1,500	102	1,875	1.25	Life, Movement, Conflict	Personnel	Italic
Polish	4,479	108	11,891	2.65	Life, Personnel, Conflict	Transaction	Balto-Slavic
Turkish	4,480	117	8,394	1.87	Life, Conflict, Personnel	Personnel	Turkic
Hindi	1,499	98	1,811	1.20	Life, Movement, Conflict	Conflict	Indo-Iranian
Japanese	1,500	99	1,730	1.15	Personnel, Life, Conflict	Personnel	Japonic
Korean	1,500	103	1,526	1.01	Personnel, Life, Conflict	Personnel	Koreanic
Total	31,226	-	50,934	-	-	-	-
Table 3: Statistics of the MINION dataset. Seg. represent text segments. All annotated segments consist of 5 sentences and their lengths (Avg. Length) are computed in terms of number of tokens. "Challenging Type" indicates the type whose event trigger annotation involves the largest disagreement between annotators in each language.
Figure 1: Distributions of event types in each language
know that “crime” is referring to a mass execution event. However, some annotators do not consider “crime” as an event trigger as they are not aware of the execution event. Eventually, we have decided to annotate the text segments based on only the presented information in the input texts to resolve conflicts and avoid inconsistency.
This section aims to study the challenges of ED for 8 languages in our MINION dataset. As such, we evaluate the performance of the state-of-the-art (SOTA) ED models in the monolingual situations where models are trained and tested on the annotated data of the same language. To prepare for the experiments, we randomly split the annotated data for each language in MINION into separate training/development/test sets with the ratio of 80/10/10 (respectively). As MINION allows multi-word event triggers to accommodate language specialities in multiple languages, we model the ED task in our dataset as a sequence labeling problem. Concretely, given an input text segment  $D = [w_{1}, w_{2}, \ldots, w_{n}]$  with  $n$  words, ED models need to predict the label sequence  $Y = [y_{1}, y_{2}, \ldots, y_{n}]$  where  $y_{i}$  indicates the label for the word  $w_{i} \in D$
using the BIO tagging schema.
To this end, following prior work on multilingual ED (Wang et al., 2020) and cross-lingual ED (M'hamdi et al., 2019), we examine the following representative SOTA models for sequence-labeling ED: (1) Transformer: A pre-trained transformer-based language model (PTLM), e.g., mBERT (Devlin et al., 2019), is augmented with a feed-forward network to predict a label for each word in the input text; (2) Transformer+CRF: This model also employs an PTLM as the Transformer model; however, a Conditional Random Field (CRF) layer is additionally introduced as the final layer to predict the label sequence  $Y$ ; (3) Transformer+BiLSTM: This model extends the Transformer model by injecting a bidirectional Long Short-Term Memory network (BiLSTM) between the PTLM and the feed-forward network to further abstract the representation vectors; and (4) Transformer+BiLSTM+CRF: This model is similar to the Transformer+BiLSTM model with an exception that a CRF layer is employed in the end for label sequence prediction. As such, to implement the models, we explore two SOTA multilingual PTLMs models, i.e., mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020) (their base versions) for text encoding. In the model notation, we will replace the prefix "Transformer" with "mBERT" or "XLMR" depending on the actual PTLM to use (e.g., mBERT, mBERT+CRF, mBERT+BiLSTM). Following prior work (Wang et al., 2020; M'hamdi et al., 2019), in the experiments, we evaluate the models using precision, recall and F1 scores for correctly predicting event trigger boundaries and types in text.
Our fine-tuning process suggests similar values of hyper-parameters for the models across languages in MINION. In particular, for English, we use one layer for BiLSTM mod
Language	mBERT	mBERT+CRF	mBERT+BiLSTM	mBERT+BiLSTM+CRF
P	R	F1	P	R	F1	P	R	F1	P	R	F1
English	77.37	79.69	78.51	75.46	78.81	77.10	76.63	79.60	78.09	80.23	79.39	79.81
Spanish	71.87	65.56	68.57	69.77	64.77	67.18	65.23	65.35	65.29	67.39	66.06	66.72
Portuguese	80.33	73.50	76.76	78.75	72.36	75.42	76.49	73.52	74.98	77.46	73.84	75.61
Polish	70.42	68.48	69.43	69.68	67.03	68.33	70.25	65.87	67.99	72.43	65.38	68.73
Turkish	63.2	64.99	64.08	62.95	63.29	63.12	63.55	64.82	64.18	64.21	67.70	65.91
Hindi	73.15	72.19	72.67	71.83	73.18	72.50	70.18	70.08	70.13	71.38	72.22	71.80
Japanese	71.83	60.00	65.28	70.80	59.41	64.61	71.43	56.47	63.08	72.67	58.84	65.03
Korean	78.48	75.15	76.78	76.90	74.94	75.91	77.83	75.60	76.70	78.27	77.99	78.13
Avg.	73.33	69.95	71.51	72.02	69.22	70.52	71.45	68.91	70.06	73.01	70.18	71.47
Table 4: Performance of the ED models in the monolingual setting using mBERT on MINION.
Language	XLMR	XLMR+CRF	XLMR+BiLSTM	XLMR+BiLSTM+CRF
P	R	F1	P	R	F1	P	R	F1	P	R	F1
English	81.45	77.79	79.58	80.15	75.79	77.91	78.46	74.84	76.61	79.26	76.18	77.69
Spanish	73.74	67.88	70.69	72.27	66.69	69.37	70.28	65.86	68.00	71.34	67.03	69.12
Portuguese	72.50	76.72	74.25	70.28	76.35	73.19	69.07	80.22	74.23	70.44	80.46	75.12
Polish	70.76	69.61	70.18	71.91	64.81	68.18	70.37	65.24	67.71	72.30	66.76	69.42
Turkish	63.66	66.71	65.15	62.18	65.99	64.03	64.30	60.15	62.16	65.12	62.07	63.56
Hindi	77.04	68.87	72.72	76.54	66.74	71.31	76.69	65.76	70.81	75.39	68.60	71.84
Japanese	72.30	63.69	67.72	71.10	61.58	66.00	70.89	63.92	67.23	72.64	60.77	66.18
Korean	74.32	82.42	78.16	73.16	81.35	77.04	72.69	78.31	75.40	73.00	79.34	76.04
Avg.	73.22	71.71	72.31	72.20	69.91	70.88	71.59	69.29	70.27	72.44	70.15	71.12
Table 5: Performance of the ED models in the monolingual setting using XLMR on MINION.
ules with 300 dimensions for the hidden states (for Transformer+BiLSTM and Transformer+BiLSTM+CRF). For feed-forward networks, we employ 2 layers with 200 dimensions for the hidden vectors. The learning rate is set to  $1e - 4$  for the Adam optimizer and the batch size of 8 is employed during training.