SlowGuess/NAACL_2022_OCR huggingface dataset:
From Tables 2 and 3, it should be noted that there is a substantial performance increase by performing the trivial change of replacing mBERT with XLM-RoBERTa as the encoder. Furthermore, our OACLED model clearly and consistently outperforms the baselines for all language pairings, with the exception of the Chinese-Arabic pair. We attribute this to the impaired performance of XLM-RoBERTa as the encoder for that specific pair as can be confirmed by the poor performance of the XLM-R-CRF baseline on the same configuration. Most importantly, OACLED's improvement over
Target
Source	Model	English	Chinese	Arabic
English	BERT-2TA	X	46.9*	29.3*
BERT-CRF	X	68.5*	30.9*
XLM-R-CRF	X	70.49±0.85	43.54±2.77
OACLED	X	74.64±0.73	44.86±3.1
Chinese	BERT-CRF	37.52±1.73	X	35.05±2.85
XLM-R-CRF	41.72±1.4	X	32.76±2.31
OACLED	45.77±1.45	X	34.48±2.43
Arabic	BERT-CRF	40.1±3.26	58.78±2.33	X
XLM-R-CRF	45.22±1.82	61.76±1.57	X
OACLED	47.98±2.07	63.13 ±1.7	X
Table 2: Results on the ACE05 dataset with standard deviation across random seeds. Entries marked * are taken directly from the original papers.
Target
Source	Model	English	Spanish
English	BERT-CRF	X	43.28±2.01
XLM-R-CRF	X	46.79±1.34
OACLED	X	47.69±1.63
Spanish	BERT-CRF	39.8±2.27	X
XLM-R-CRF	45.61±1.76	X
OACLED	47.5±1.89	X
Table 3: Results on ERE dataset with standard deviation across random seeds.
the XLM-R-CRF baseline is present in every configuration, which validates the effectiveness of our optimized approach to ALA training.
We identify 2 main components in our approach: using ALA to create refined language-invariant representations, and optimizing the adversarial training process by selecting a subset of samples chosen with OT to incorporate our measures of informativeness into the sample selection process. Of course, removing ALA training entirely restores the model to the baseline. However, adversarial training optimization via OT has various aspects to it. In order to understand the contribution of these aspects, we explore four different models: OACLED-OT presents the effects of removing sample selection entirely and using all available samples to train the LD; OACLED-L2 uses a constant distance between the unlabeled samples instead the standard L2 distance used in the Sinkhorn algorithm; OACLED-EP completely removes the EP module and a uniform distribution is used as the probability distributions for both languages; finally, OACLED-ED-Loss keeps the EP module, but removes its  $EP_{loss}$  term from Equation 10. The performance results of these models is presented in Table 4. In this and the following sections (3.5,
3.6.2), we present the results of experiments using English as the sole source language as it is the source language most ubiquitously used. We, however, found consistency in the displayed effects for different source/target language configurations.
Model version	Target Language
English	Chinese	Arabic	Spanish
OACLED-OT	70.94	40.55	44.96
OACLED-L2	71.35	41.79	44.39
OACLED-EP	73.08	42.81	46.99
OACLED-EP-Loss	72.93	43.4	46.35
OACLED	74.64	44.86	47.69
As expected, removing the sample selection through OT leads to the worst performance drop. This highlights the importance of selecting informative examples for the LD. Furthermore, removing the cost function also hurts performance greatly, which shows that a proper distance function is needed for the OT algorithm to work effectively. While the effects of removing the EP module and its corresponding loss term are not of the same magnitude, they are still significant. These results support our claim for the need and utility of all the components in our approach, showing that their inclusion is crucial in achieving state-of-the-art performance.
3.5 Language Model Finetuning
The key contribution of our approach is to exploit unlabeled data in the target language, which is usually abundant, by introducing it into the training process to improve our model's language-invariant qualities.
To confirm the utility of our approach, Table 5 contrasts our model's performance against a baseline whose encoder has been finetuned with the same unlabeled data using the standard masked language model objective.
Table 4: Ablation experiment results
Model Version	Target Language
English	Chinese	Arabic	Spanish
Finetuned XLM-R	71.06	43.71	47.82
OACLED	74.64	44.86	47.69
It can be observed that our model outperforms the finetuned baseline in two out of the three target languages. Additionally, the difference in performance in those two instances is considerably larger
(3.58% and 1.15%), than the setting in which the baseline performs better (0.13%).
3.6 Analysis
3.6.1 Learned Representation Distances
First, we look at the distance between the sentence-level representations  $h_{i0}$  generated by the encoder for different source/target language pairs. Figure 1 shows a plot of such distances using cosine distance as the distance function.
Figure 1: Distance between sentence representations for different language pairs.
When computing the correlation with the performance results in Table 2, we obtain a score  $R = -0.6616$ , meaning there is moderate negative correlation between the distance of the representations and model performance, i.e. closer representations lead to better performance.
Similarly, Table 6 shows a comparison of the distances between the representations generated by OACLED and those obtained by the XLM-R-CRF baseline.
Table 5: OACLED performance versus a baseline using an encoder finetuned with unlabeled data.
Cosine Distance
Source/Target	Baseline	OACLED
English/Chinese	3.64e-3	3.93e-6
English/Arabic	7.71e-2	2.08e-5
English/Spanish	5.4e-3	5.3e-6
Chinese/English	3.62e-3	3.87e-6
Arabic/English	4.16e-2	1.02e-5
Spanish/English	6.87e-3	1.49e-5
Table 6: Comparison of representation-vector distances for language pairs between our model and the baseline.
We observe that OACLED representations are closer, by several orders of magnitude, than those obtained by the baseline. This supports our claim that our model's encoder generates more refined
language-invariant representations than those obtained by the default version of XLM-RoBERTa.
3.6.2 Access to Labeled Target Data
Previously, we discussed how a key feature of our approach is that it does not require annotated data in the target language and, instead, leverages the use of unlabeled data which is readily available. Nonetheless, we also explore the performance of our model in the event that there exists a small amount of annotated target data available. Figure 2 shows the results of our experiments when using different amounts of labeled target data during training.
Figure 2: Model performance when training on small quantities of labeled target data. The  $X$  axis presents the percentage (0 - 10%) of data used out of the entire training set of the target language.
It can be observed that OACLED consistently outperforms the baseline even when there is some availability of annotated data. Additionally, performance steadily increases as more and more data is used. This conforms to expectations, and confirms that having labeled data in the target language available for training is ultimately beneficial to the model's performance.
3.6.3 Case Study
Next, we look into our model's predictions and analyse instances where it outperforms the baseline to exemplify the advantages of dealing with
optimized language-invariant representations. We identify two important patterns.
First, our model seems to better classify events in the target language that involve trigger words that have distinct connotations that depend on context. Specially those that are two distinct words in the source language. For example, the Spanish word "juicio" can have two distinct meanings that are different words in English: "trial" and "judgement". Our model correctly classifies it as a JUSTICE:TRIAL-HEARING trigger in the sentence "Dos llamaos a juicio fueon hechos por un jurado federal investigador". Meanwhile, the baseline fails to even recognize it as a trigger. Another example is the word "detenido", an adjective that can mean both "detained", in a criminal context, and "stopped", as in halted. Our model correctly classifies it in the sentence "Padilla no debería permanecer detenido durante把这些 alejado de los reos" as a JUSTICE:ARREST-JAIL trigger while the baseline fails to detect the event. We manually identified 23 of these polysemous triggers in the Spanish $^{2}$  test set and found that 19 (82.6%) were correctly classified by our OACLED model versus 14 (60.8%) by the baseline (27.8% improvement).
Additionally, we found our model correctly classifies verb conjugation variants that do not exist in the source language. For instance, our model correctly recognizes the words "venderlos", "vender", "vendes", and "vendedor" (variants of the verb "to buy") as TRANSACTION:TRANSFER-OWNERSHIP triggers whereas the baseline incorrectly classifies them as being of the TRANSACTION:TRANSFER-MONEY type. As previously mentioned, Majewska et al. (2021) propose injecting external verb-knowledge into the training to help with verb interpretation for event extraction. Our empirical results, however, outperform their reports which appears to imply that, at least for CLED, holistically learning the language-invariant features shared between the target and source languages works better than injecting language-specific verb knowledge.
We believe these findings illustrate how, by introducing additional context in the form of unlabeled data, the model is able to learn fine-grained word representations that better capture the semantics of the words in the target language, and successfully deal with difficult cross-lingual issues.
2We use Spanish for the analysis as it is the mother tongue of the first author.
Research efforts on monolingual ED are extensive and varied. Hand-crafted, feature-based, language-specific methods were the basis of early ED approaches (Ahn, 2006; Ji and Grishman, 2008; Patwardhan and Riloff, 2009; Liao and Grishman, 2010a,b; Hong et al., 2011; McClosky et al., 2011; Li et al., 2013; Miwa et al., 2014; Yang and Mitchell, 2016). More recent efforts have primarily made use of deep learning techniques such as convolutional neural networks (Nguyen and Grishman, 2015; Chen et al., 2015; Nguyen et al., 2016b), recurrent neural networks (Nguyen et al., 2016a; Sha et al., 2018; Lai et al., 2020), graph convolutional networks (Nguyen and Grishman, 2018; Yan et al., 2019; Nguyen et al., 2021a), adversarial networks (Hong et al., 2018; Zhang et al., 2019b), and pre-trained language models (Wadden et al., 2019; Zhang et al., 2019a; Yang et al., 2019; Zhang et al., 2020; Liu et al., 2020; Pouran Ben Veyseh et al., 2021b,a).
Works on cross-lingual ED are not as prevalent and generally make use of cross-lingual resources employed to address the differences between languages such as bilingual dictionaries or parallel corpora (Muis et al., 2018; Liu et al., 2019) and, more recently, pre-trained multilingual language models (M'hamdi et al., 2019; Hambardzumyan et al., 2020; Majewska et al., 2021). Unlike these previous efforts, our method leverages unlabeled data to further refine the language-invariant qualities of the language models.
Adversarial Language Adaptation, inspired by models in domain adaptation research (Ganin and Lempitsky, 2015; Naik and Rose, 2020; Ngo Trung et al., 2021), has been successfully applied at generating language-invariant models (Joty et al., 2017; Chen et al., 2018; Nguyen et al., 2021b). Our method improves upon these approaches optimizing the adversarial training process by selecting the most informative examples from the unlabeled data.
Additional examples of downstream applications of cross-lingual learning are document classification (Holger and Xian, 2018), named entity recognition (Xie et al., 2018) and part-of-speech tagging (Cohen et al., 2011). For a thorough review on cross-lingual learning, we refer the reader to Pikuliak et al. (2021).
We present OACLED, a new model for cross-lingual event detection that learns fine-grained language-invariant representations by optimizing the standard ALA training through optimal-transport-based sample selection. Our model achieves new state-of-the-art performance in our experiments on 8 different language pairs which demonstrate its robustness and effectiveness at generating refined language-invariant representations that allow for better event detection results. Our analysis of its intermediate outputs and predictions confirm that OACLED's representations are indeed closer to each other and that this proximity translates into better handling of difficult cross-lingual instances. We also note that, while this work focuses on the event detection task, our proposed optimization of the adversarial training process is task independent and can be generalized to other related IE tasks when leveraging ALA is deemed beneficial.
David Ahn. 2006. The stages of event extraction. In Proceedings of the Workshop on Annotating and Reasoning about Time and Events.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial Deep Averaging Networks for Cross-Linguual Sentiment Classification. In Transactions of the Association for Computational Linguistics.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multipooling convolutional neural networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Shay B. Cohen, Dipanjan Das, and Noah Smith. 2011. Unsupervised structure prediction with nonparallel multilingual guidance. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. In CoRR.
Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on Machine Learning, pages 1180-1189.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2020. The role of alignment of multilingual contextualized embeddings in zero-shot crosslingual transfer for event extraction. In Collaborative Technologies and Data Science in Artificial Intelligence Applications.
Schwenk Holger and Li Xian. 2018. A corpus for multilingual document classification in eight languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC).
Yu Hong, Jianfeng Zhang, Bin Ma, Jianmin Yao, Guodong Zhou, and Qiaoming Zhu. 2011. Using cross-entity inference to improve event extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Yu Hong, Wenxuan Zhou, Jingli Zhang, Guodong Zhou, and Qiaoming Zhu. 2018. Self-regulation: Employing a generative adversarial network to improve event detection. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 515-526, Melbourne, Australia. Association for Computational Linguistics.
Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Shafiq Joty, Preslav Nakov, Lluis Márquez, and Israa Jaradat. 2017. Cross-language learning with adversarial neural networks. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL), pages 226-237.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning.
Viet Dac Lai, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020. Event detection: Gate diversity and syntactic importance scores for graph convolution neural networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5405-5411, Online. Association for Computational Linguistics.
Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Shasha Liao and Ralph Grishman. 2010a. Filtered ranking for bootstrapping in event extraction. In Proceedings of the International Conference on Computational Linguistics (COLING).
Shasha Liao and Ralph Grishman. 2010b. Using document level cross-event inference to improve event extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020. Event extraction as machine reading comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Jian Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2019. Neural cross-lingual event detection with minimal parallel resources. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 738-748, Hong Kong, China. Association for Computational Linguistics.
Olga Majewska, Ivan Vulic, Goran Glavaš, Edoardo Maria Ponti, and Anna Korhonen. 2021. Verb knowledge injection for multilingual
event processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing.
David McClosky, Mihai Surdeanu, and Christopher Manning. 2011. Event extraction as dependency parsing. In BioNLP Shared Task Workshop.
Meryem M'hamdi, Marjorie Freedman, and Jonathan May. 2019. Contextualized cross-lingual event trigger extraction with minimal resources. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL).
Makoto Miwa, Paul Thompson, Ioannis Korkontzelos, and Sophia Ananiadou. 2014. Comparable study of event extraction in newswire and biomedical domains. In Proceedings of the International Conference on Computational Linguistics (COLING).
Aldrian Obaja Muis, Naoki Otani, Nidhi Vyas, Ruochen Xu, Yiming Yang, Teruko Mitamura, and Eduard Hovy. 2018. Low-resource cross-lingual event type detection via distant supervision with minimal effort. In Proceedings of the 27th International Conference on Computational Linguistics.
Aakanksha Naik and Carolyn Rose. 2020. Towards open domain event trigger identification using adversarial domain adaptation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Nghia Ngo Trung, Duy Phung, and Thien Huu Nguyen. 2021. Unsupervised domain adaptation for event detection using domain-specific adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4015-4025, Online. Association for Computational Linguistics.
Minh Van Nguyen, Viet Lai, and Thien Huu Nguyen. 2021a. Cross-task instance representation interactions and label dependencies for joint information extraction with graph convolutional networks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 27-38, Online. Association for Computational Linguistics.
Minh Van Nguyen, Tuan Ngo Nguyen, Bonan Min, and Thien Huu Nguyen. 2021b. Crosslingual transfer learning for relation and event extraction via word category and class alignments. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016a. Joint event extraction via recurrent neural networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Thien Huu Nguyen, Lisheng Fu, Kyunghyun Cho, and Ralph Grishman. 2016b. A two-stage approach for extending event detection to new types via neural networks. In Proceedings of the 1st ACL Workshop on Representation Learning for NLP (RepL4NLP).
Thien Huu Nguyen and Ralph Grishman. 2015. Event detection and domain adaptation with convolutional neural networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Thien Huu Nguyen and Ralph Grishman. 2018. Graph convolutional networks with argument-aware pooling for event detection. In Association for the Advancement of Artificial Intelligence (AAAI).
Trung Minh Nguyen and Thien Huu Nguyen. 2019. One for all: Neural joint modeling of entities and events. In Association for the Advancement of Artificial Intelligence (AAAI).
Siddharth Patwardhan and Ellen Riloff. 2009. A unified model of phrasal and sentential evidence for information extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Jonas Pfeiffer, Andreas Rückle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Matúš Pikuliak, Marián Šimko, and María Bieliková. 2021. Cross-lingual learning for text processing: A survey. In *Expert Systems with Applications*.
Amir Pouran Ben Veyseh, Viet Lai, Franck Dernoncourt, and Thien Huu Nguyen. 2021a. Unleash GPT-2 power for event detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6271-6282, Online. Association for Computational Linguistics.
Amir Pouran Ben Veyseh, Minh Van Nguyen, Nghia Ngo Trung, Bonan Min, and Thien Huu Nguyen. 2021b. Modeling document-level context for event detection via important context selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Lei Sha, Feng Qian, Baobao Chang, and Zhifang Sui. 2018. Jointly extracting event triggers and arguments by dependency-bridge rnn and tensor-based argument interaction. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI).
Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick,