SlowGuess/NAACL_2022_OCR huggingface dataset:
Unlike contrastive loss and ii-loss which cannot be directly used for classifying samples in the test set and need to be combined with cross entropy loss, our proposed closed boundary loss can be easily adopted for classification by the following calculation:
$$
g (\mathbf {e} _ {i} \prime) = \left\{ \begin{array}{l l} 1 & \| \mathbf {e} _ {i} ^ {\prime} - \mathbf {c} \| ^ {2} - R ^ {2} <   0 \\ 0 & \| \mathbf {e} _ {i} ^ {\prime} - \mathbf {c} \| ^ {2} - R ^ {2} > 0 \end{array} \right.
$$
3.4 Entity Summary Graph
To make full use of the redundant argument information, we classify every entity in the document.
For the same argument, we may obtain multiple preliminary extraction results. The advantage is the robustness because the correct argument is more likely to be extracted from relatively simple positions. The challenge is how to merge the multiple extraction results. To address the challenge, we propose an entity summary graph.
Text Matching Module. We notice that most redundant expressions of the same entity are either character-level spelling similar or word-level semantics similar. In some cases, special domain knowledge is needed to determine if two expressions are the same. For example, "Army of National Liberation" and "ELN" are referred to the same entity. Therefore, we adopt a text matching model with both character embedding and word embedding to evaluate the spelling similarity and semantics similarity of extracted arguments. We also construct a text matching dataset from ground truth labels of the training set of our event extraction dataset to make the model learn necessary domain knowledge.
We build the text matching module (TMM) by adopting the structure of RE2 (Yang et al., 2019a) and adding character embedding to the RE2 model to enhance the model's capability of recognizing spelling similarity. We denote the initially predicted arguments as  $\mathrm{A} = \{\mathbf{a}_0, \mathbf{a}_1, \dots, \mathbf{a}_{k-1}\}$ . Then, we feed these entities into the text matching module to produce the matching score for each pair of arguments.
$$
\mathbf {M} _ {i j} = \operatorname {T M M} \left(\mathbf {a} _ {i}, \mathbf {a} _ {j}\right) \tag {9}
$$
where  $\mathbf{M}$  is the matching score matrix, which contains text matching score of every pair of entities from A.  $\mathbf{M} = [\mathbf{M}_{ij}], i,j = 1,2,\ldots ,k.$
Entity Summary Graph. The graph node is composed of preliminary predicted entities A. The i-th node and j-th node are connected if  $\mathbf{M}_{ij} > s$ , where  $s$  is a boundary score. The weight of each edge is the text matching score  $\mathbf{M}_{ij}$  of two entity nodes at the ends of the edge.
The constructed entity summary graph is mostly disconnected because there usually exist multiple argument clusters in a document. The argument cluster refers to a set of different expressions of the same argument, for example "the armed forces" and "military" refer to the same argument, thus forming an argument cluster. The entity summary graph consists of several connected subgraphs as shown in figure 3. Each subgraph
PerpInd	PerpOrg	Target	Victim	Weapon
GTT (Du et al., 2021)	65.48/39.86/49.55	66.04/42.68/51.85	55.05/44.12/48.98	76.32/61.05/67.84	61.82/56.67/59.13
NST (Du and Cardie, 2020a)	48.39/32.61/38.96	60.00/43.90/50.70	54.96/52.94/53.93	62.50/63.16/62.83	61.67/61.67/61.67
DYGIE++ (Wadden et al., 2019)	59.49/34.06/43.32	56.00/34.15/42.42	53.49/50.74/52.08	60.00/66.32/63.00	57.14/53.33/55.17
RICB	50.76/49.62/50.18	50.00/63.75/56.04	65.63/63.64/64.62	64.86/50.52/56.80	63.49/65.57/64.51
Table 1: Performance comparison with baseline models for each argument role on MUC-4 dataset. Results for each column are displayed in the order of precision, recall, and F1 score.
Models	P	R	F1
GTT (Du et al., 2021)	64.19	47.36	54.50
NST (Du and Cardie, 2020a)	56.82	48.92	52.58
DYGIE++ (Wadden et al., 2019)	57.04	46.77	51.40
RICB	57.68	58.03	57.85
Table 2: Averaged EAE result on the MUC-4 dataset. Precision (P), recall (R), and F1-score are used for evaluation.
corresponds to an argument cluster. We denote the entity summary graph  $\mathbf{G}$  and its subgraphs as  $\mathbf{G} = \left\{\mathbf{G}_{sub}^{(1)},\mathbf{G}_{sub}^{(2)},\dots ,\mathbf{G}_{sub}^{(u)}\right\}$ . The final predicted arguments are summarized by selecting an entity node with the largest sum of weights (LSW) from each subgraph.
$$
\mathrm {A} ^ {\prime} = \left\{\mathbf {a} _ {0} ^ {\prime}, \mathbf {a} _ {1} ^ {\prime}, \dots , \mathbf {a} _ {v - 1} ^ {\prime} \right\}, \quad \mathbf {a} _ {i} ^ {\prime} = \operatorname {L S W} \left(\mathrm {G} _ {s u b} ^ {(i)}\right)
$$
Our model is evaluated on the MUC-4 dataset (McLean, 1992). The dataset is composed of 1,700 documents, each containing an average of 400 tokens and 7 paragraphs. We use 1300 documents for training, 200 documents for testing, and 200 documents for the development set following (Du and Cardie, 2020a). Five argument roles are extracted in the dataset: perpetrator individual, perpetrator organization, target, victim, and weapon.
4.2 Baselines and Evaluation Metric
In this work, we propose a document-level EAE model leveraging Redundant Information and Closed Boundary Loss (RICB). We compare our model with the following baseline models: DY-GIE++ (Wadden et al., 2019) incorporates local and global contexts to build a multi-task framework for named entity recognition, relation extraction,
and event extraction. NST (Du and Cardie, 2020a) aggregates sentence representation and paragraph representation via a gate mechanism and treats document-level EAE as a sequence tagging problem. GTT (Du et al., 2021) proposes a generative transformer based framework for document-level EAE.
We evaluate the performance of our model by the CEAF-TF metric following (Du et al., 2021). The metric finds the best alignment of predicted arguments and gold arguments. It penalizes the system that does not merge multiple extraction results by setting a constraint that a gold argument can be aligned with at most one predicted argument. Precision (P), recall (R), and F1-score (F1) are used to evaluate the model's performance.
The per-role EAE results on the MUC-4 dataset of our RICB model and baseline models are summarized in Table 1, and the micro-averaged performance is shown in Table 2. Table 2 shows that our model consistently outperforms the latest baselines in F1-score and achieves the state-of-the-art (SOTA) performance. Specifically, the proposed model improves the absolute F1-score by  $3.35\%$ ,  $5.27\%$ , and  $6.45\%$  compared to baseline models. Noticeably, our model achieves an over  $9\%$  improvement in recall. In terms of the per-role extraction performance of our model, it achieves the highest F1-score in four of five argument roles: perpetrator individual, perpetrator organization, target, and weapon. Specifically, the absolute F1-score is improved by  $0.63\%$ ,  $4.19\%$ ,  $10.69\%$ , and  $2.84\%$  in these argument roles.
4.4 Effect of Graph2token Module
Graph structure is used in EAE to produce a comprehensive representation of coreference entities (Luan et al., 2019; Qian et al., 2019; Xu et al., 2021). In this work, we further adopt a graph2token module to feed back the comprehensive representation of coreference entities to their context tokens.
PerpInd	PerpOrg	Target	Victim	Weapon
Without graph2token	50.39/49.24/49.80	50.02/58.83/54.07	63.87/57.58/60.56	62.54/49.53/55.28	58.72/69.47/63.64
Cross entropy loss	50.00/50.34/50.17	48.57/63.75/55.14	62.04/64.39/63.19	49.55/58.95/53.85	55.13/70.49/61.87
String matching	48.80/45.86/47.28	45.30/66.25/53.81	65.71/63.44/64.56	59.49/49.47/54.02	58.57/67.21/62.60
RICB	50.76/49.62/50.18	50.00/63.75/56.04	65.63/63.64/64.62	64.86/50.52/56.80	63.49/65.57/64.51
Table 3: Ablation studies on graph2token module, closed boundary loss, and entity summary graph, respectively. The results in each column are displayed in the order of precision, recall, and F1 score.
The updated token representations can generate additional coreference-aware representations for entities near the coreference entity. For the ablation study, we experiment on without applying the graph2token module. We compare per-role extraction results with and without the graph2token module in Table 3. We find that the experiment without the graph2token module results in a performance drop on every argument role. In addition, the recall is decreased by  $0.38\%$ ,  $4.92\%$ ,  $6.06\%$ , and  $0.99\%$  in four argument roles. This indicates that the model can recognize more arguments by providing argument candidates with additional coreference-aware representations.
4.5 Effect of Closed Boundary Loss
We find that classifier trained by cross entropy loss could easily misclassify entities in the Universum class. We propose a closed boundary loss to address this issue. For the ablation study, we conduct experiments of applying cross entropy loss for argument classification, and compare the performance with our model. The comparison of two loss functions is summarized in Table 3, which shows that in all argument roles, closed boundary loss consistently outperforms cross entropy in the F1 score. We further notice that the precision of the model is improved in all argument roles at  $0.76\%$ ,  $1.43\%$ ,  $3.59\%$ ,  $15.31\%$ , and  $8.36\%$  by using closed boundary loss. The improvement in precision indicates that the use of closed boundary results in a smaller number of Universum samples that are misclassified as target samples.
4.6 Effect of Entity Summary Graph
To fully leverage the redundant argument information, we classify every entity in the document. For the same argument, we may obtain multiple preliminary extraction results. We propose the entity summary graph to merge the results. For the ablation study, we conduct experiments on merging multiple extraction results based on string matching following Zheng et al. (2019); Xu et al. (2021). We compare the string matching performance with our proposed entity summary graph in Table 3. It
shows that the entity summary graph outperforms the string matching method significantly in the F1-score. Furthermore, the precision of the model is improved in four of five argument roles by  $1.96\%$ ,  $4.70\%$ ,  $5.37\%$ , and  $4.92\%$  by using the entity summary graph, and this verifies the effect of our proposed entity summary graph, i.e. merging multiple extraction results and reducing false positives accordingly.
Figure 5 demonstrates an example of the differences in predicting event arguments between GTT (Du et al., 2021) and our proposed RICB method. To avoid involving excessive sentences in the document, only roles of perpetrator individual and perpetrator organization are used for illustration. RICB successfully extracts "Colonel Ponce" and "ARENA", while GTT fails. Both event arguments "Colonel Ponce" and "ARENA" appear multiple times in the document, which shows the redundant event information in the document. Specifically, among all their occurrences in the document, it is easier to recognize "Colonel Ponce" from sentence 8 and recognize "ARENA" from sentence 7. This is an illustration of our idea that by utilizing redundant event information in the document, we can extract arguments from relatively simple positions. In addition, to recognize "Colonel Ponce" from sentence 4, it is necessary to understand that "this action" refers to "the massacre". Our model can recognize it because the graph2token module can feed back the coreference information to "this action".
4.8 Further Analysis
Firstly, it is effective to leverage redundant event information in documents for document-level EAE, which is not only reflected in the F1 score, but also in the significant improvement in recall. The microaveraged recalls of baseline models are distributed between  $46\%$  to  $49\%$ , but our model reaches  $58\%$ . As we analyzed in the introduction, leveraging redundant argument information of a document allows the model to extract the argument from any
... [2] The massacre against the Salvadoran Workers National Union Federation (FENASTRAS) was planned in cold blood. ... [4] We have trustworthy information from our intelligence organs that this action was ordered by Colonel Ponce, that Cristiani knew about it and approved it. ... [6] Terrorism is an old practice of the Nationalist Republican Alliance (ARENA). [7] Only a few days ago, ARENA assassins tried to kill the president of the Mortgage bank, Mr Mason, for not following their orders. [8] The people demand the resignation of chief of staff Col Emilio Ponce because his responsibility in this criminal action is real. ... [18] The war of the armed forces, government, and ARENA is aimed against the people. ...
Peretrator Individual	Perpetrator Organization
GTT	ARENA assassins	-
RICB	Colonel Ponce, ARENA assassins	ARENA
Figure 5: An example of the differences in event argument extraction between GTT and our proposed RICB. The differences in extracting perpetrator individual and perpetrator organization are used for illustration. RICB successfully extracts Colonel Ponce and ARENA, while GTT fails. In the example, sentence numbers are marked in green, and identical entities are marked with the same color.
of its occurrences and relatively simple positions. Therefore, the difficulty of argument extraction is reduced and the recall is improved accordingly. We also notice a drop in precision rate in our model compared to baseline models. It is because baseline methods adopt sequence-to-sequence models and we classify a few arguments from a great number of entities in the document, which will naturally result in a decrease in precision. However, the precision and recall of our model are very close, which is more balanced compared to baseline models.
Secondly, leveraging redundant event information in a document is not simply classifying every entity in the document. On the one hand, better entity representations need to be produced, on the other hand, multiple extraction results need to be merged. Therefore, we add the graph2token module to the entity coreference graph, which improves the recall significantly. We also propose the entity summary graph to merge multiple extraction results, which successfully improves the precision.
Finally, we propose a novel closed boundary loss to better deal with the Universum class in our task. Its effectiveness is verified in ablation studies. We highlight two other potential benefits of closed boundary loss here. Firstly, since it generates a closed decision boundary for classifiers, it may also be valid for dealing with unseen samples in the test set. This property is not evaluated in this work. In addition, our dataset is highly imbalanced because only a small number of entities are arguments. Weighted cross entropy loss is cumbersome to adjust the appropriate weights,
however, the closed boundary loss does not need to adjust weights and works well with the imbalanced dataset.
In this work, we emphasize that the redundant event information in documents is beneficial but is often overlooked in document-level EAE. In addition, we find that classifiers trained by cross entropy loss are problematic in classifying the Universum class. Specifically, we generate a comprehensive and coreference-aware representation for every entity through the entity coreference graph with the graph2token module. In addition, we propose an entity summary graph to merge the multiple extraction results of the same argument. Furthermore, we propose a novel closed boundary loss to deal with the Universum class in classification. As a limitation, our proposed closed boundary loss is used for binary classification because we extract arguments in a role-by-role manner to make full use of the property of each argument role. In the future, we will extend it for multiclass classification and apply it to other tasks in natural language processing that face the problem of classifying Universum class. Experimental results show that our RICB model achieves the SOTA performance and outperforms prior approaches on the MUC-4 dataset.
The authors would like to thank Zijian Feng, Zixiao Zhu, Li Qi, and the anonymous reviewers for their constructive comments and suggestions. The
research was conducted at the Future Resilient Systems at the Singapore-ETH Centre, which was established collaboratively between ETH Zurich and the National Research Foundation Singapore. This research is supported by the National Research Foundation Singapore (NRF) under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
Wasi Uddin Ahmad, Nanyun Peng, and Kai-Wei Chang. 2021. Gate: Graph attention transformer encoder for cross-lingual relation and event extraction. In The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21).
Abhijit Bendale and Terrance E Boult. 2016. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563-1572.
Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. 2021. Padim: a patch distribution modeling framework for anomaly detection and localization. In International Conference on Pattern Recognition, pages 475-489. Springer.
X. Du, Alexander M. Rush, and Claire Cardie. 2021. Document-level event-based extraction using generative template-filling transformers. In EACL.
Xinya Du and Claire Cardie. 2020a. Document-level event role filler extraction using multi-granularity contextualized encoding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8010-8020, Online. Association for Computational Linguistics.
Xinya Du and Claire Cardie. 2020b. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 671-683, Online. Association for Computational Linguistics.
Sanghamitra Dutta, Liang Ma, Tanay Kumar Saha, Di Liu, Joel Tetreault, and Alejandro Jaimes. 2021. GTN-ED: Event detection using graph transformer networks. In Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15), pages 132-137, Mexico City, Mexico. Association for Computational Linguistics.
Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence argument linking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8057-8077, Online. Association for Computational Linguistics.
Joseph Fisher and Andreas Vlachos. 2019. Merge and label: A novel neural network architecture for nested
NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5840-5850, Florence, Italy. Association for Computational Linguistics.
Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), volume 2, pages 1735-1742. IEEE.
Mehadi Hassen and Philip K Chan. 2020. Learning a neural-network-based representation for open set recognition. In Proceedings of the 2020 SIAM International Conference on Data Mining, pages 154-162. SIAM.
Kung-Hsiang Huang and Nanyun Peng. 2021. Document-level event extraction with efficient end-to-end learning of cross-event dependencies. In Proceedings of the Third Workshop on Narrative Understanding, pages 36-47, Virtual. Association for Computational Linguistics.
Radu Tudor Ionescu, Fahad Shahbaz Khan, Mariana-Iuliana Georgescu, and Ling Shao. 2019. Object-centric auto-encoders and dummy anomalies for abnormal event detection in video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7842-7851.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Sha Li, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 894-908, Online. Association for Computational Linguistics.
Shasha Liao and Ralph Grishman. 2010. Using document level cross-event inference to improve event extraction. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 789-797, Uppsala, Sweden. Association for Computational Linguistics.
Chien-Liang Liu, Wen-Hoar Hsaio, Chia-Hoang Lee, Tao-Hsing Chang, and Tsung-Hsun Kuo. 2015. Semi-supervised text classification with universum learning. IEEE transactions on cybernetics, 46(2):462-473.
Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018. Jointly multiple events extraction via attention-based graph information aggregation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1247-1256, Brussels, Belgium. Association for Computational Linguistics.
Dongfang Lou, Zhilin Liao, Shumin Deng, Ningyu Zhang, and Huajun Chen. 2021. MLBiNet: A cross-sentence collective event detection network. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4829-4839, Online. Association for Computational Linguistics.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3036-3046, Minneapolis, Minnesota. Association for Computational Linguistics.
Qing Lyu, Hongming Zhang, Elior Sulem, and Dan Roth. 2021. Zero-shot event extraction via transfer learning: Challenges and insights. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 322-332, Online. Association for Computational Linguistics.
Virginia McLean. 1992. Fourth message understanding conference (muc-4).
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 300-309, San Diego, California. Association for Computational Linguistics.
Leo GM Noordman and Wietske Vonk. 1998. Memory-based processing in understanding causal information. Discourse Processes, 26(2-3):191-212.
Bertram Opitz and Angela D Friederici. 2004. Brain correlates of language learning: the neuronal dissociation of rule-based versus similarity-based learning. Journal of Neuroscience, 24(39):8436-8440.
Zhiquan Qi, Yingjie Tian, and Yong Shi. 2012. Twin support vector machine with universum data. Neural Networks, 36:112-119.
Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay. 2019. GraphIE: A graph-based framework for information extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 751-761, Minneapolis, Minnesota. Association for Computational Linguistics.
Bharat Richhariya and Muhammad Tanveer. 2020. A reduced universum twin support vector machine for class imbalance learning. Pattern Recognition, 102:107150.