SlowGuess/NAACL_2022_OCR huggingface dataset:
Monolingual Performance: The performance of the four ED models on the test data of each language are presented in Tables 4 (for mBERT) and 5 (for XLMR). There are several observations from these tables. First, the best average F1 score of the models over different languages is  $72.31\%$  (achieved by the XLMR model). This performance is still considerably lower than a perfect model, thus suggesting significant challenges of ED in multiple languages and calling for more research effort in this area. Second, the performance of the models for non-English language is significantly worse than the English counterpart. This difference thus further highlights the necessity of more research on ED for non-English languages. Finally, the superior performance of XLMR over other models in almost all languages indicates better effectiveness of the multilingual PTLM model XLMR for ED in different languages (compared to mBERT). It also implies that traditional BiLSTM and CRF
layers for sequence labeling are less necessary for multilingual ED when a PTLM is employed for text encoding. As such, in the following experiments, we will employ Transformer as the main ED model for further analysis.
Monolingual PTLMs: To better understand the benefits of multilingual PTLMs (i.e., mBERT and XLMR) for multilingual ED, we further evaluate the performance the Transformer model when monolingual language-specific PTLMs are leveraged to encode input texts (i.e., replacing mBERT and XLMR). Accordingly, for monolingual language-specific PTLMs, we consider both BERT-based and RoBERTa-based models for comprehensiveness. Tables 6 (for BERT) and 7 (for RoBERTa) report the monolingual performance of Transformer when monolingual language-specific PTLMs are employed. Note that we only show ED performance for languages where monolingual PTLMs are publicly available. As can be seen, compared to multilingual PTLMs, monolingual PTLMs (based on BERT or RoBERTa) improve the performance of Transformer for English. However, for other languages, monolingual PTLMs are on-par (for BERT-based models) or significantly worse (for RoBERTa-based models) than multilingual PTLMs for ED, thus demonstrating the general
Language	P	R	F1
English (Devlin et al., 2019)	78.12	81.61	79.83
Spanish (Cañete et al., 2020)	72.73	62.25	67.08
Portuguese (Souza et al., 2020)	81.82	72.00	76.60
Polish (Kłeczek, 2021)	71.79	65.00	68.23
Turkish (MDZ, 2021)	67.75	60.57	63.96
Table 6: Test data performance of Transformer in the monolingual setting using available language-specific BERT models on MINION. The citations indicate the sources of the language-specific BERT.
Language	P	R	F1
English (Liu et al., 2019)	82.56	78.96	80.73
Spanish (MMG, 2021)	72.54	58.61	64.84
Polish (CLARIN-PL, 2021)	71.55	66.86	69.12
Hindi (Parmar, 2021)	71.70	50.67	59.38
Japanese (Wongso, 2021)	67.53	36.62	47.49
advantage of multilingual PTLMs for ED. In addition, it is suggestive that future work can explore methods to improve monolingual language-specific PTLMs for ED in different languages.
Cross-lingual Performance: To understand the transferability of ED knowledge and annotation across languages, we explore the cross-lingual evaluation setting where models are trained on English data (the source language) and directly evaluated on test data of other target languages in MINION. As such, we report the cross-lingual performance of Transformer with both mBERT and XLMR as the PTLMs in Table 8. Note that we inherit the same hyper-parameters selected for Transformer in the fine-tuning process of monolingual experiments for consistency.
Compared to the monolingual performance counterparts of mBERT and XLMR in Tables 4 and 5,
Table 7: Test data performance of Transformer in the monolingual setting using available language-specific RoBERTa models on MINION. The citations indicate the sources of the language-specific RoBERTa.
Language	mBERT	XLMR
P	R	F1	P	R	F1
English	77.37	79.69	78.51	81.45	77.79	79.58
Spanish	74.32	54.14	62.64	78.27	52.48	62.83
Portuguese	70.79	71.50	71.14	73.36	72.18	72.77
Polish	73.83	49.84	59.51	79.55	48.33	60.13
Turkish	69.25	35.14	46.62	66.84	36.49	47.21
Hindi	66.10	51.66	57.99	64.74	52.84	58.19
Japanese	52.44	25.29	34.13	55.39	25.71	35.12
Korean	67.24	42.27	55.52	80.69	43.80	56.78
Table 8: Cross-lingual performance of Transformer that is trained on English training data and evaluated on test data of other languages in MINION.
it is clear that the performance of Transformer in non-English languages decreases significantly in the cross-lingual evaluation, i.e., the average performance loss due to cross-lingual evaluation is  $15.2\%$  for both mBERT and XLMR. We also observe a wide range of cross-lingual performance for the target languages in Table 8, thus suggesting the diverse nature of the data and languages in MINION to support robust model development for ED. Among the target languages, Portuguese exhibits the smallest performance difference between monolingual and cross-lingual settings while the largest performance loss with cross-lingual transfer occurs in Japanese, Turkish, Korean, and Hindi. One possible reason for such performance loss is due to the language structure difference where Japanese, Turkish, Korean, and Hindi follow the Subject-Object-Verb word order while English and other languages in our dataset utilize the SubjectVerb-Object order. Another reason can be linked to different patterns/distributions of event triggers in different languages. For instance, some languages tend to mention the events using verbs (e.g., in English  $78\%$  of the triggers are verb) while other languages might use more diverse parts of speech to express event trigger (e.g., in Japanese only  $63\%$  of triggers are verbs). Also, Section 4 provide an additional explanation regarding the diversity of event triggers in different languages. In all, the cross-lingual performance in our MINION dataset demonstrates the challenges of transferring ED knowledge across languages that can be further studied in future work.
This section provides additional analysis to better understand the multilingual ED task in MINION.
Cross-dataset Evaluation: As the event ontology in MINION is inherited and pruned from the ACE 2005 dataset, it is helpful to learn how the annotated events in MINION is different from those in ACE 2005. To this end, we propose to evaluate model performance on the cross-dataset setting: models are trained on the English data of ACE 2005 and evaluated on test data of different languages in MINION. In particular, we utilize the standard data split from prior work (Nguyen and Grishman, 2015; Chen et al., 2015; Wang et al., 2019) to obtain English training and development data in ACE 2005. Also, we filter the ACE 2005 data so only triggers of event sub-types in our MINION
Language	P	R	F1
English	60.24	59.62	59.93
Spanish	61.48	44.63	51.72
Portuguese	61.98	46.09	52.87
Polish	70.23	38.06	49.37
Turkish	58.19	29.32	38.99
Hindi	57.8	21.63	31.48
Japanese	42.91	22.10	29.17
Korean	71.84	28.01	40.31
dataset are retained for a compatibility between two datasets. Due to its superior performance in previous experiments, we employ the Transformer model with XLMR in this experiment. The hyperparameters for the model is fine-tuned on the development data of ACE 2005. Table 9 shows the model performance in the cross-dataset evaluation. Compared to the corresponding cross-lingual performance of MINION in Table 8, it is clear that the performance on MINION is significantly worse when the model is trained on ACE 2005 data. As such, a possible explanation for this performance loss includes domain difference between ACE 2005 and MINION, i.e., MINION involve Wikipedia articles while ACE 2005 is based on news articles, conversational telephone speeches, and others. In addition, as the size of English training data in MINION (i.e., over 14K triggers) is significantly larger than those for ACE 2005 (i.e., less than 6K triggers), the training data in MINION might cover more event patterns to produce better performance for ED models. Future work can explore this cross-dataset evaluation setting to build more robust models for ED.
Trigger Diversity in Different Languages: To understand how events are expressed in different languages, we explore the ratio of unique trigger words over the total number of event triggers for an event sub-type (called unique ratio). Figure 2 shows the averages of unique ratios over event sub-types for different languages in our MINION dataset. As such the diagram shows that English is relatively simpler than other languages in ED as its diversity of event triggers for event types is the least among all the considered languages. Korean, Turkish, and Japanese are the languages that exhibit the largest diversities of event triggers. This further helps to explain the worst cross-lingual performance of models from English to Korean,
Table 9: Performance of the XLMR model that is trained on the English training data of ACE 2005 and evaluated on the test data of each language in MINION.
Language	Trained on English	Trained on Spanish
P	R	F1	P	R	F1
Portuguese	74.35	51.99	61.19	75.55	55.80	64.19
Polish	70.28	42.26	52.78	71.08	48.39	57.58
Turkish	60.16	30.14	40.16	62.19	34.42	44.31
Hindi	60.27	42.31	49.72	59.24	48.23	53.17
Japanese	49.31	21.78	30.21	55.48	22.60	32.12
Korean	72.98	36.37	48.55	73.90	40.16	52.04
Table 10: Performance of XLMR in the cross-lingual setting when it is trained on English and Spanish. For both languages, 3,000 random samples from the training set of the corresponding language are selected to train the model.
Turkish, and Japanese in Table 8.
Challenging the Supremacy of English for Event Detection: English has been the major language for ED research. In particular, in cross-lingual transfer learning for ED, English has often been considered as a high-resource source language to train ED models to apply to other target languages (M'hamdi et al., 2019; Nguyen et al., 2021b). In this experiment, we argue that English is not necessary the optimal source language for cross-lingual transfer learning of ED. In particular, using Transformer with XLMR as the base model, we train the model on the training data of both English and Spanish; the resulting models are evaluated on the test data of the other languages in MINION. To ensure a fair comparison, we use the same size of training data for English and Spanish, i.e., 3,000 annotated text segments randomly sampled in MINION. Table 10 presents the cross-lingual performance of the models. The table demonstrates that using Spanish as the source language can achieve better performance than English for all the target languages in MINION. As such, our findings suggest that choosing appropriate source languages for cross-lingual transfer learning of ED is important and can be further explored in future work.
Early attempts for ED have employed feature-based models (Ahn, 2006; Ji and Grishman, 2008; Patwardhan and Riloff, 2009; Liao and Grishman, 2010; Hong et al., 2011; Li et al., 2013; Miwa et al., 2014b; Yang and Mitchell, 2016) while deep learning has recently been proven to be a better approach for ED (Nguyen and Grishman, 2015; Chen et al., 2015; Nguyen et al., 2016; Sha et al., 2018; Yang et al., 2019; Wang et al., 2019; Cui et al., 2020; Lai et al., 2020, 2021a; Ngo Trung et al., 2021;
Figure 2: The average ratios of unique event triggers over event sub-types for each language in MINION.
Pouran Ben Veyseh et al., 2021a). There have also been recent efforts on creating new datasets for ED for different domains, including biomedical texts (Kim et al., 2009), literary texts (Sims et al., 2019), cybersecurity texts (Satyapanich et al., 2020; Man et al., 2020), Wikipedia texts (Wang et al., 2020), fine-grained event types (Le and Nguyen, 2021), and historical texts (Lai et al., 2021b). However, such prior works and datasets for ED are mainly devoted to English, ignoring challenges in many non-English languages. Non-English datasets for ED also exist (Kobylinski and Wasiluk, 2019; Sahoo et al., 2020); however, these datasets are only annotated for one language with divergent ontology and annotation guidelines, thus unable to support comprehensive studies and transferability research for ED on multiple languages.
Existing ED datasets that cover multiple languages involve ACE 2005 (Walker et al., 2006), TAC KBP (Mitamura et al., 2016, 2017), and TempEval-2 (Verhagen et al., 2010). Among such datasets, ACE 2005 is the most popular dataset used in prior multilingual/cross-lingual ED research (Chen and Ji, 2009; M'hamdi et al., 2019; Ahmad et al., 2021; Nguyen et al., 2021c; Nguyen and Nguyen, 2021). However, such multilingual datasets suffer from the issues of small data size, limited language coverage with greater focus on popular languages, and inaccessibility to the public as discussed in the introduction. Finally, we also note some prior works that claim event detection datasets for non-English datasets (Im et al., 2009; Küçük and Yazici, 2011; Lejeune et al., 2015). However, such datasets are not comparable to our dataset as their event detection task is indeed a sentence classification problem where established definition of events with event triggers are not followed and annotated.
We introduce a new dataset for ED in 8 typologically different languages. The dataset is significantly larger and covers more and newer languages than prior resources. Specifically, 31,226 text segments from language-specific articles of Wikipedia are manually annotated in the dataset. Our experiments and analysis demonstrate the high quality of the dataset and the multilingual challenges of ED, providing ample room for future research in this direction. In the future, we will extend the dataset to include event argument annotations.
Wasi Uddin Ahmad, Nanyun Peng, and Kai-Wei Chang. 2021. Gate: Graph attention transformer encoder for cross-lingual relation and event extraction. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI).
David Ahn. 2006. The stages of event extraction. In Proceedings of the Workshop on Annotating and Reasoning about Time and Events.
Giuseppe Attardi. 2015. Wikiextractor. https://github.com/attardi/wikiextractor.
Jose Cañete, Gabriel Chaperon, Rodrigo Fuentes, JouHui Ho, Hojin Kang, and Jorge Pérez. 2020. Spanish pre-trained bert model and evaluation data. In PML4DC at International Conference on Learning Representations (ICLR) 2020.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multipooling convolutional neural networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Zheng Chen and Heng Ji. 2009. Can one language bootstrap the other: A case study on event extraction. In Proceedings of the NAACL-HLT 2009 Workshop on Semi-supervised Learning for Natural Language Processing.
CLARIN-PL CLARIN-PL. 2021. Polish roberta.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Shiyao Cui, Bowen Yu, Tingwen Liu, Zhenyu Zhang, Xuebin Wang, and Jinqiao Shi. 2020. Edge-enhanced graph convolution networks for event detection with syntactic relation. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2329-2339, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence argument linking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Yu Hong, Jianfeng Zhang, Bin Ma, Jianmin Yao, Guodong Zhou, and Qiaoming Zhu. 2011. Using cross-entity inference to improve event extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Seohyun Im, Hyunjo You, Hayun Jang, Seungho Nam, and Hyopil Shin. 2009. KTimeML: Specification of temporal and event expressions in Korean text. In Proceedings of the 7th Workshop on Asian Language Resources (ALR7).
Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshi-nobu Kano, and Jun'ichi Tsujii. 2009. Overview of BioNLP'09 shared task on event extraction. In Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task.
Łukasz Kobyński and Michal Wasiluk. 2019. Deep learning in event detection in Polish. In Proceedings of the 10th Global Wordnet Conference, pages 216-221, Wroclaw, Poland. Global Wordnet Association.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Dilek Küçuk and Adnan Yazici. 2011. Exploiting information extraction techniques for automatic semantic video indexing with an application to turkish news videos. In Knowl. Based Syst.
Darek Kłeczek. 2021. Polish bert.
Viet Lai, Franck Dernoncourt, and Thien Huu Nguyen. 2021a. Learning prototype representations across few-shot tasks for event detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5270-5277, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Viet Lai, Minh Van Nguyen, Heidi Kaufman, and Thien Huu Nguyen. 2021b. Event extraction from historical texts: A new dataset for black rebellions. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2390-2400, Online. Association for Computational Linguistics.
Viet Dac Lai, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020. Event detection: Gate diversity and syntactic importance scores for graph convolution neural networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5405-5411, Online. Association for Computational Linguistics.
Duong Le and Thien Huu Nguyen. 2021. Fine-grained event trigger detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2745-2752, Online. Association for Computational Linguistics.
Gael Lejeune, Romain Brixtel, Antoine Doucet, and Nadine Lucas. 2015. Multilingual event extraction for epidemic detection. In Artif. Intell. Medicine.
Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Shasha Liao and Ralph Grishman. 2010. Filtered ranking for bootstrapping in event extraction. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Duc Trong Hieu Man, Duc Trong Le, Amir Pouran Ben Veyseh, Thuat Nguyen, and Thien Huu Nguyen. 2020. Introducing a new dataset for event detection in cybersecurity texts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Digital Library team MDZ. 2021. Turkish bert.
Meryem M'hamdi, Marjorie Freedman, and Jonathan May. 2019. Contextualized cross-lingual event trigger extraction with minimal resources. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL).
Teruko Mitamura, Zhengzhong Liu, and Eduard H. Hovy. 2016. Overview of TAC-KBP 2016 event nugget track. In Proceedings of the Text Analysis Conference (TAC).
Teruko Mitamura, Zhengzhong Liu, and Eduard H. Hovy. 2017. Events detection, coreference and sequencing: What's next? overview of the TAC KBP 2017 event track. In Proceedings of the Text Analysis Conference (TAC).
Makoto Miwa, Paul Thompson, Ioannis Korkontzelos, and Sophia Ananiadou. 2014a. Comparable study of event extraction in newswire and biomedical domains. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers.
Makoto Miwa, Paul Thompson, Ioannis Korkontzelos, and Sophia Ananiadou. 2014b. Comparable study of event extraction in newswire and biomedical domains. In Proceedings of the International Conference on Computational Linguistics (COLING).
MMG MMG. 2021. Spanish roberta.
Nghia Ngo Trung, Duy Phung, and Thien Huu Nguyen. 2021. Unsupervised domain adaptation for event detection using domain-specific adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4015-4025, Online. Association for Computational Linguistics.
Minh Van Nguyen, Viet Duc Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. 2021a. Trankit: A light-weight transformer-based toolkit for multilingual natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations.
Minh Van Nguyen and Thien Huu Nguyen. 2021. Improving cross-lingual transfer for event argument extraction with language-universal sentence structures. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 237–243, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
Minh Van Nguyen, Tuan Ngo Nguyen, Bonan Min, and Thien Huu Nguyen. 2021b. Crosslingual transfer learning for relation and event extraction via word category and class alignments. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5414-5426, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Minh Van Nguyen, Tuan Ngo Nguyen, Bonan Min, and Thien Huu Nguyen. 2021c. Crosslingual transfer learning for relation and event extraction via word category and class alignments. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5414-5426, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Thien Huu Nguyen and Ralph Grishman. 2015. Event detection and domain adaptation with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers).
Thien Huu Nguyen and Ralph Grishman. 2018. Graph convolutional networks with argument-aware pooling for event detection. In Proceedings of the AAAI Conference on Artificial Intelligence.
Suraj Parmar. 2021. Hindi roberta.
Rebecca Passonneau. 2006. Measuring agreement on set-valued items (masi) for semantic and pragmatic annotation. In The International Conference on Language Resources and Evaluation (LREC).
Siddharth Patwardhan and Ellen Riloff. 2009. A unified model of phrasal and sentential evidence for information extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Amir Pouran Ben Veyseh, Viet Lai, Franck Dernoncourt, and Thien Huu Nguyen. 2021a. Unleash GPT-2 power for event detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6271-6282, Online. Association for Computational Linguistics.