SlowGuess/ACL_2020_OCR huggingface dataset:
candidate graph contains multiple ORG-AFF edges incident to the same node, the model will demote this graph by adding a negative value into its global score. We also observe that the weights of about  $9\%$  global features are almost not updated, which indicates that they are barely found in both gold-standard and predicted graphs. In Table 8, we perform qualitative analysis on concrete examples.
4.4 Porting to Another Language
As Table 7, we evaluate the proposed framework on ACE05-CN and ERE-ES. The results show that ONEIE works well on Chinese and Spanish data without any special design for the new language. We also observe that adding English training data can improve the performance on Chinese and Spanish.
4.5 Remaining Challenges
We have analyzed 75 of the remaining errors and in Figure 5 we present the distribution of various error types which need more features and knowledge acquisition to address in the future. In this section, we will discuss some main categories with examples.
Need background knowledge. Most of current IE methods ignore external knowledge such as entity attributes and scenario models. For exam-
Table 5: New benchmark results (F-score, %).
Positive Feature	Weight
1	A TRANSPORT event has only one DESTINATION argument	2.61
2	An ATTACK event has only one PLACE argument	2.31
3	A TRANSPORT event has only one ORIGIN argument	2.01
4	An END-POSITION event has only one PERSON argument	1.51
5	A PER-SOC relation exists between two PER entities	1.08
6	A GEN-AFF relation exists between ORG and LOC entities	0.96
7	A BENEFICIARY argument is a PER entity	0.93
8	A GEN-AFF relation exists between ORG and GPE entities	0.90
Negative Feature	Weight
9	An entity has an ORG-AFF relation with multiple entities	-3.21
10	An entity has an PART-WHOLE relation with multiple entities	-2.49
11	An event has two PLACE arguments	-2.47
12	A TRANSPORT event has multiple DESTINATION arguments	-2.25
13	An entity has a GEN-AFF relation with multiple entities	-2.02
14	An ATTACK event has multiple PLACE arguments	-1.86
15	An entity has a PHYS relation with multiple entities	-1.69
16	An event has multiple VICTIM arguments	-1.61
Table 6: Salient positive and negative global features.
Dataset	Training	Entity	Relation	Trig-C	Arg-C
ACE05-CN	CN	88.5	62.4	65.6	52.0
CN+EN	89.8	62.9	67.7	53.2
ERE-ES	ES	81.3	48.1	56.8	40.3
ES+EN	81.8	52.9	59.1	42.3
Table 7: Results on ACE05-CN and ERE-ES (F-score, %). For ACE05-CN, EN refers to  $\mathrm{ACE05 - E^{+}}$ . For ERE-ES, EN refers to ERE-EN.
Figure 5: Distribution of remaining errors.
ple, in the following sentence, "And Putin's media aide, Sergei Yastrzhembsky, told Kommersant Russia would not forgive the Iraqi debt", our model mistakenly identifies "Kommersan" as a person instead of organization. With entity linking, we
8005
Sentence & Analysis	Baseline	+Global Features
#1: Russia's foreign minister expressed outrage at suggestions from a top Washington official last week that Moscow should forgive the eight billion dollars in Soviet-era debt that Baghdad owes it, as a gesture of good will.
★ Global feature category: 8
★ Analysis: It is unlikely for a person to have an ORG-AFF relation with multiple entities.	Russia minister GPE ORG-AFF PER ORG-AFF ORG-AFF PER ORG-AFF GPE official Washington	Russia minister GPE ORG-AFF PER ORG-AFF GPE official Washington
#2: They also deployed along the border with Israel.
★ Global feature category: 9
★ Analysis: It is uncommon that an ORIGIN argument and a DES-TINATION argument have a PART-WHOLE relation.	border LOC PART-WHOLE Israel GPE destination origin Transport deployed	border LOC PART-WHOLE Israel GPE origin Transport deployed
#3: Prime Minister Abdullah Gul resigned earlier Tuesday to make way for Erdogan , who won a parliamentary seat in by-elections Sunday.
★ Global feature categories: 2 and 5
★ Analysis: 1. An ELECT usually has only one PERSON argument; 2. An entity is unlikely to act as a PERSON argument for END-POSITION and ELECT events at the same time.	ElectionpersonpersonElect End-Position resigned	ElectionpersonEnd-Position resigned
#4: Diller will continue to play a critical role in the future of Vivendi 's entertainment arm.
★ Global feature category: 6
★ Analysis: A PART-WHOLE relation should not exist between PER and ORG entities.	Vivendi PER PART-WHOLE arm ORG PER Diller	Vivendi PER ORG PER Diller
#5: He also brought a check from Campbell to pay the fines and fees.
★ Global feature category: 3
★ Analysis: As “Campbell” is likely to be an ENTITY argument of a FINE event, the model corrects its entity type from FAC to PER.	fines Campbell Fine FAC	fines entity Campbell Fine PER
Table 8: Examples showing how global features improve the quality of extracted information networks. For some sentences, we do not draw the whole information network.
can correct this error based on the first sentence in its Wikipedia page "Kommersant is a nationally distributed daily newspaper published in Russia mostly devoted to politics and business".
Rare words. The second challenge is the famous long-tail problem: many triggers, entity mentions (e.g., "caretaker", "Gazeta.ru") and contextual phrases in the test data rarely appear in the training data. While most event triggers are verbs or nouns, some adverbs and multi-word expressions can also serve as triggers.
Multiple types per trigger. Some trigger words may indicate both the procedure and the result status of an action. For example, "named" may indicate both NOMINATE and START-POSITION events; "killed" and "eliminate" may indicate both ATTACK and DIE events. In these cases the human ground truth usually only annotates the procedure types, whereas our system produces the resultant event types.
Need syntactic structure. Our model may benefit from deeper syntactic analysis. For example, in the following sentence "As well as previously holding senior positions at Barclays Bank, BZW and Kleinwort Benson, McCarthy was formerly a top civil servant at the Department of Trade and Industry", our model misses all of the employers "Barclays Bank", "BZW" and "Kleinwort Benson" for "McCarthy" probably because they appear in a previous sub-sentence.
Uncertain events and metaphors. Our model mistakenly labels some future planned events as specific events because its lacking of tense prediction and metaphor recognition. For example, START-ORG triggered by "formation" does not happen in the following sentence "The statement did not give any reason for the move, but said Lahoud would begin consultations Wednesday aimed at the formation of a new government". Our model also mistakenly identifies "camp" as a facility, and a
8006
DIE event triggered by "dying" in the following sentence "Russia hints 'peace camp' alliance with Germany and France is dying by Dmitry Zaks".
The IE community is lacking of newer data sets with end-to-end annotations. Unfortunately, the annotation quality of the ACE data set is not perfect due to some long-term debates on the annotation guideline; e.g., Should "government" be tagged as a GPE or an ORG? Should "dead" be both an entity and event trigger? Should we consider designator word as part of the entity mention or not?
Previous work (Roth and Yih, 2004; Li et al., 2011) encodes inter-dependency among knowledge elements as global constraints in an integer linear programming framework to effectively remove extraction errors. Such integrity verification results can be used to find knowledge elements that violate the constraints and identify possible instances of detector errors or failures. Inspired by these previous efforts, we propose a joint neural framework with global features in which the weights are learned during training. Similar to (Li et al., 2014)'s method, ONEIE also uses global features to capture cross-subtask and cross-instance interdependencies, while our features are language-independent and do not rely on other NLP tools such as dependency parsers. Our methods also differ in local features, optimization methods, and decoding procedures.
Some recent efforts develop joint neural models to perform extraction of two IE subtasks, such as entity and relation extraction (Zheng et al., 2017; Katiyar and Cardie, 2017; Bekoulis et al., 2018; Fu et al., 2019; Luan et al., 2019; Sun et al., 2019) and event and temporal relation extraction (Han et al., 2019). Wadden et al. (2019) design a joint model to extract entities, relations and events based on BERT and dynamic span graphs. Our framework extends (Wadden et al., 2019) by incorporating global features based on cross-subtask and cross-instance constraints. Unlike (Wadden et al., 2019) that uses a span-based method to extract mentions, we adopt a CRF-based tagger in our framework because it can extract mentions of any length, not restricted by the maximum span width.
We propose a joint end-to-end IE framework that incorporates global features to capture the inter
dependency between knowledge elements. Experiments show that our framework achieves better or comparable performance compared to the state of the art and prove the effectiveness of global features. Our framework is also proved to be language-independent and can be applied to other languages, and it can benefit from multi-lingual training.
In the future, we plan to incorporate more comprehensive event schemas that are automatically induced from multilingual multimedia data and external knowledge to further improve the quality of IE. We also plan to extend our framework to more IE subtasks such as document-level entity coreference resolution and event coreference resolution.
This research is based upon work supported in part by U.S. DARPA KAIROS Program No. FA8750-19-2-1004, U.S. DARPA AIDA Program No. FA8750-18-2-0014, Air Force No. FA8650-17-C-7715, the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract No. FA8650-17-C-9116. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ODNl, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Adversarial training for multi-context joint entity and relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multipooling convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP2015).
Jason Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association of Computational Linguistics (TACL).
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011.
8007
Natural language processing (almost) from scratch. Journal of Machine Learning Research.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT2019).
Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. In Transactions of the Association for Computational Linguistics (TACL).
Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019. GraphRel: Modeling text as relational graphs for joint entity and relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL2019).
Rujun Han, Qiang Ning, and Nanyun Peng. 2019. Joint event and temporal relation extraction with shared representations and structured prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP2019).
Heng Ji and Ralph Grishman. 2005. Improving name tagging by reference resolution and relation detection. In In Proceedings of ACL 05, Ann Arbor, USA.
Heng Ji, David Westbrook, and Ralph Grishman. 2005. Using semantic relations to refine coreference decisions. In *In Proceedings of HLT/EMNLP* 05, Vancouver, B.C., Canada.
Arzoo Katiyar and Claire Cardie. 2017. Going out on a limb: Joint extraction of entity mentions and relations without dependency trees. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL2017).
Johannes Kirschnick, Holmer Hemsen, and Volker Markl. 2016. JEDI: Joint entity and relation detection using type inference. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics System Demonstrations (ACL2016).
Qi Li, Sam Anzaroot, Wen-Pin Lin, Xiang Li, and Heng Ji. 2011. Joint inference for cross-document information extraction. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM2011).
Qi Li, Heng Ji, HONG Yu, and Sujian Li. 2014. Constructing information networks using one single model. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP2014).
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL2016).
Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP2015).
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT2019).
Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP2014).
Dan Roth and Wen-tau Yih. 2004. A linear programming formulation for global inference in natural language tasks. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL2004).
Avirup Sil and Alexander Yates. 2013. Re-ranking for joint named-entity recognition and linking. In Proceedings of the 22nd ACM international conference on Conference on Information & Knowledge Management (CIKM2013).
Changzhi Sun, Yeyun Gong, Yuanbin Wu, Ming Gong, Daxin Jiang, Man Lan, Shiliang Sun, and Nan Duan. 2019. Joint type inference on entities and relations via graph convolutional networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL2019).
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP2019).
Bishan Yang and Tom M. Mitchell. 2016. Joint extraction of events and entities within a document context. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT2016).
Tongtao Zhang, Heng Ji, and Avirup Sil. 2019. Joint entity and event extraction with generative adversarial imitation learning. Data Intelligence.
Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017. Joint extraction of entities and relations based on a novel tagging
scheme. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL2017).
8009