SlowGuess/NAACL_2022_OCR huggingface dataset:
Hanzhang Zhou $^{1,3}$ , Kezhi Mao $^{2,3*}$
$^{1}$ Institute of Catastrophe Risk Management, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore
$^{2}$ School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore  $^{3}$ Future Resilient Systems Programme, Singapore-ETH Centre, CREATE campus, Singapore
hanzhang001@e.ntu.edu.sg
ekzmao@ntu.edu.sg
In document-level event argument extraction, an argument is likely to appear multiple times in different expressions in the document. The redundancy of arguments underlying multiple sentences is beneficial but is often overlooked. In addition, in event argument extraction, most entities are regarded as class "others", i.e. Universum class, which is defined as a collection of samples that do not belong to any class of interest. Universum class is composed of heterogeneous entities without typical common features. Classifiers trained by cross entropy loss could easily misclassify the Universum class because of their open decision boundary. In this paper, to make use of redundant event information underlying a document, we build an entity coreference graph with the graph2token module to produce a comprehensive and coreference-aware representation for every entity and then build an entity summary graph to merge the multiple extraction results. To better classify Universum class, we propose a new loss function to build classifiers with closed boundaries. Experimental results show that our model outperforms the previous state-of-the-art models by  $3.35\%$  in F1-score.
Event argument extraction (EAE) is a crucial subtask of event extraction (EE), aiming to identify the arguments of a given event and recognize the specific roles they play. Previous works are mostly focused on sentence-level EE (Liao and Grishman, 2010; Nguyen et al., 2016; Liu et al., 2018; Yang et al., 2019b; Du and Cardie, 2020b; Wei et al., 2021; Wang et al., 2021; Lyu et al., 2021). However, events are often described in the form of documents in the real world. Document-level event extraction has received consideration in recent years.
Research on document-level event extraction has been focused on tackling challenges such as
No.	Sentence	Entity label	Difficulty
s1	The killers, approximately 30 men in uniform, arrived before 0230.	1	★
s2	Soldiers with their faces painted black arrived in Cayara last Saturday. They broke down doors, looted stores, and burned several houses.	1	★★★
s3	The murder was carried out by soldiers.	1	★
s4	The house was surrounded by soldiers.	0	-
s5	The house was searched by the soldiers 2 days before the crime.	0	-
s6	How can men in uniform be in a militarized area?	0	-
s7	He said that the library was burned.	-	-
...	......	...	...
Argument role	Entity	Entity label	Summative label
Perpetrator individual	men in uniform	1	1
soldiers with their faces painted black	1	0
soldiers	1	0
Physical target	houses	1	1
library	1	1
...	...	...	...
Figure 1: An example of redundant event information in the document-level event argument extraction.
arguments-scattering and multiple-events (Zheng et al., 2019; Du and Cardie, 2020a; Du et al., 2021; Lou et al., 2021; Li et al., 2021; Huang and Peng, 2021; Xu et al., 2021; Yang et al., 2021; Ahmad et al., 2021; Ebner et al., 2020). The benefit of redundant event information in a document is largely neglected. We believe that the redundant event information in a document can be used to improve event extraction, as illustrated in the example in Figure 1. The upper part of Figure 1 shows seven simplified sentences selected from a document in the MUC-4 dataset. All entities marked in blue are the same entity "soldiers", which appears in different expressions in different sentences. For ease of description, we call it entity  $S$ . We can observe from Figure 1 that: 1) The argument information in the document is redundant since entity  $S$  appears in the article multiple times as an argument and we can successfully extract the argument by correctly recognizing any of these occurrences. This prop
Human Language Technologies, pages 3041 - 3052
Figure 2: A simplified illustration of closed boundary loss. Blue dots represent target samples, orange dots represent Universum samples. The red dotted line represents cross entropy loss, the purple solid line represents proposed closed boundary loss.
erty can be potentially used to improve the robustness of the model. 2) The difficulty of extracting the entity  $S$  as an argument in its different occurrences is different. Extracting entity  $S$  in sentence 1 and sentence 3 is much easier than extracting it from sentence 2. Hence, by utilizing the redundant event information of the document, we can extract arguments from relatively simple positions and reduce the difficulty of the task. 3) An entity may appear multiple times in the document, directly averaging them as the entity's feature representation (Xu et al., 2021) may introduce noise. For example, although entity  $S$  is an event argument in the document, its occurrences in s4, s5 and s6 should not be recognized as a correct pattern to identify the event argument. 4) The redundant argument information can result in redundant extraction results, as shown in the bottom table in Figure 1. The three entities extracted as perpetrator individual need to be merged into one. However, the extracted physical target "houses" and "library" are different entities and should not be merged. Therefore, the use of redundant event information underlying a document is not straightforward, a sophisticated algorithm for merging multiple extraction results is needed.
Extraction of arguments can be solved as an entity classification problem by treating entities as argument candidates (Zheng et al., 2019; Xu et al., 2021; Yang et al., 2021). In document-level event argument extraction, only a subset of the entities in a document are arguments, while the majority of entities are regarded as class "others" or "neither"(neither of the target classes). This kind of data was first studied by Weston et al. (2006) under the name Universum. The Universum data are
usually very diverse and do not have typical common features. In addition, Universum data is much more than the target class data, exhibiting a severe class imbalance problem. Figure 2 demonstrates a simplified distribution of data samples in document-level event argument extraction. The blue dots represent argument entities, the orange dots represent a large number of Universum class entities. Since the samples in the Universum class do not have typical common features, they tend to scatter in the feature space. This characteristic of the Universum data is largely overlooked in the information extraction community. Universum data is simply considered as another class "others", without any special consideration in the classifier design. Cross entropy loss is usually employed in classifier training (Zheng et al., 2019; Huang and Peng, 2021; Xu et al., 2021; Yang et al., 2021). However, classifiers trained by cross entropy loss have open decision boundaries, and hence some Universum samples, such as the orange dot on the upper right of the figure, could be easily misclassified. We think a classifier with a closed decision boundary could better deal with the Universum class in document-level event argument extraction, as illustrated by the purple line in Figure 2.
The contribution of this work is three-fold. Firstly, it is the pioneering work to leverage redundant event information in documents for event extraction. We propose the entity coreference graph with graph2token module and entity summary graph to leverage the redundant event information. Experimental results show that redundant information helps improve recall significantly. Secondly, we analyze the issue of Universum data in document-level event argument extraction and the problem of classifiers trained by cross entropy loss, and propose a closed boundary loss to address the problems. Finally, our model consistently outperforms the latest baseline models in F1-score and achieves state-of-the-art performance. Compared to the three baseline models, our proposed model improves the absolute F1-score by  $3.35\%$ ,  $5.27\%$ , and  $6.45\%$ , respectively.
2.1 Event Argument Extraction
Most previous event argument extraction models make predictions at sentence-level (Nguyen et al., 2016; Liu et al., 2018; Yang et al., 2019b; Du and Cardie, 2020b; Wei et al., 2021; Wang et al.,
2021; Dutta et al., 2021). Considering that the real world events are often distributed across sentences, document-level event extraction has attracted more attention recently. Zheng et al. (2019) propose the Doc2EDAG model to overcome the argument scattering problem. Du and Cardie (2020a) first argue the importance of document-level extraction and adopt a sequence model for document-level event extraction. Lou et al. (2021) investigate a novel bidirectional decoder to overcome the long-range forgetting problem. Li et al. (2021) formulate the document-level event extraction model as conditional generation based on templates. Huang and Peng (2021) attach importance to event coreference and entity coreference in document-level event extraction tasks. Xu et al. (2021) build a heterogeneous graph with the Tracker module to deal with problems of event scattering and multiple events. Yang et al. (2021) adopt parallel prediction networks to extract events parallelly from document-level representations. However, none of these works pay attention to the characteristic of information redundancy in the document, which we believe is a unique and beneficial property for document-level event argument extraction. In addition, to our knowledge, closed boundary classification has never been adopted in event extraction. Classification-based event argument extraction models (Huang and Peng, 2021; Xu et al., 2021; Yang et al., 2021) all employ cross entropy loss for classifier training, without considering the characteristics of Universum class: scattered distribution in the feature space due to heterogeneity and diversity of the samples in this class.
2.2 Closed Boundary Loss
We found that a classifier trained by cross entropy could easily misclassify entities in the class "others", i.e. Universum class. We found the root cause of the problem is the open decision boundary of the classifier. To address this problem, we propose a novel closed boundary loss for classifier training.
Research works in Universum usually employ additional unlabeled Universum data to provide prior knowledge for the task, such as Universum support vector machine (SVM) (Weston et al., 2006; Qi et al., 2012; Richhariya and Tanveer, 2020), and semi-supervised learning (Liu et al., 2015; Xiao et al., 2021). However, the SVM-based methods above are developed for structured data and are hard to integrate with deep neural network
based representation learning to form an end-to-end training procedure for natural language processing tasks. One possible solution is to use a deep neural network to learn representations first, and then feed the representations learned to the Universum SVM classifiers. But the disadvantage of this two-step procedure is that the classification result cannot be back-propagated to representation learning. It is desired that the closed boundary classifier could be integrated with deep neural network-based representation learning to form end-to-end training for optimal performance.
Closed boundary classification methods are also developed in anomaly detection and open set recognition, such as deep one-class learning (Ruff et al., 2018; Defard et al., 2021), auto-encoder based anomaly detection (Ionescu et al., 2019), OpenMax layer for open set recognition (Bendale and Boult, 2016). However, these methods cannot use the information in outlier samples due to task setting.
A closed boundary classifier works best in feature space with compact class distribution. In the literature, some loss functions have been proposed to generate such feature space such as Deep SVDD (Ruff et al., 2018), contrastive loss (Hadsell et al., 2006), and ii-loss (Hassen and Chan, 2020). However, Deep SVDD only minimizes the intra-class distance and cannot maximize the inter-class distance. Contrastive loss and ii-loss need to be combined with cross entropy loss to classify samples. But cross entropy loss generates open decision boundaries for the classifier.
In this paper, we propose a new loss function that could train a classifier with a closed decision boundary. In addition, it can be directly integrated with representation learning layers in a neural network to form an end-to-end training procedure to produce a feature space with minimum intra-class difference and maximum inter-class difference, which in turn leads to improved performance.
As shown in Figure 3, our model consists of four main components: context encoding module (Sec 3.1), entity coreference graph (Sec 3.2), closed boundary loss (Sec. 3.3), and entity summary graph (Sec. 3.4), which are illustrated in this section.
Figure 3: The overall model structure. Blue dots represent entity nodes, green dots represent sentence nodes.
3.1 Context Encoding
Given the input document, we apply a Bi-LSTM to obtain token representations of the document:  $\mathbf{D} = \{\mathbf{d}_0,\mathbf{d}_1,\dots ,\mathbf{d}_{n - 1}\} \in \mathbb{R}^{n\times l}$  where  $n$  is the document length, and  $l$  is the hidden state dimension. We construct entity representation and sentence representation from the start and end tokens in an entity or sentence:
$$
\mathbf {e} _ {i} = \left(\mathbf {e} _ {\text {m e m o r y}} ^ {(i)}; \mathbf {e} _ {\text {r u l e}} ^ {(i)}\right) \tag {1}
$$
$$
\mathbf {s} _ {i} = \left(\mathbf {s} _ {\text {m e m o r y}} ^ {(i)}; \mathbf {s} _ {\text {r u l e}} ^ {(i)}\right) \tag {2}
$$
$$
\mathbf {e} _ {\text {m e m o r y}} ^ {(i)} = \left(\mathrm {D} \left[ \operatorname {e n t} _ {\text {s t a r t}} ^ {(i)} [ l: ] \right]; \mathrm {D} \left[ \operatorname {e n t} _ {\text {e n d}} ^ {(i)} [: l ] \right]\right)
$$
$$
\mathbf {e} _ {\text {r u l e}} ^ {(i)} = \left(\mathrm {D} \left[ \operatorname {e n t} _ {\text {s t a r t}} ^ {(i)} [: l ] \right]; \mathrm {D} \left[ \operatorname {e n t} _ {\text {e n d}} ^ {(i)} [ l: ] \right]\right)
$$
$$
\mathbf {s} _ {\text {m e m o r y}} ^ {(i)} = \left(\mathrm {D} \left[ \operatorname {s e n t} _ {\text {s t a r t}} ^ {(i)} [ l: ] \right]; \mathrm {D} \left[ \operatorname {s e n t} _ {\text {e n d}} ^ {(i)} [ : l ] \right]\right)
$$
$$
\mathbf {s} _ {\text {r u l e}} ^ {(i)} = \left(\mathrm {D} \left[ s e n t _ {\text {s t a r t}} ^ {(i)} [ : l ] \right]; \mathrm {D} \left[ s e n t _ {\text {e n d}} ^ {(i)} [ l: ] \right]\right)
$$
where D is the output of the Bi-LSTM encoding layer,  $ent_{\text{start}}^{(i)}$ ,  $ent_{\text{end}}^{(i)}$ ,  $sent_{\text{start}}^{(i)}$  and  $sent_{\text{end}}^{(i)}$  are the start and end position of the i-th entity and the i-th sentence, respectively, and []; denotes the concatenation operation.  $\mathbf{e}_{\text{memory}}^{(i)}$  and  $\mathbf{s}_{\text{memory}}^{(i)}$  mainly contain the information inside the entity and sentence.  $\mathbf{e}_{\text{rule}}^{(i)}$  and  $\mathbf{s}_{\text{rule}}^{(i)}$  mainly contain the context information outside the entity and sentence. The model predicts memory representations mainly based on remembering entity names and predicts rule representations mainly based on recognizing
Figure 4: An example of coreference in a document and its impact on entity understanding and document-level event argument extraction
the contextual patterns. Therefore, we separate the memory representation and rule representation as they correspond to the memory-based and the rule-based learning process of humans (Noordman and Vonk, 1998; Opitz and Friederici, 2004).
3.2 Entity Coreference Graph
Leveraging redundant event information in a document is not straightforward to classify every entity in the document. On the one hand, better entity representation is needed. Therefore, we construct an entity coreference graph with graph2token module to produce a comprehensive and coreference-aware representation for every entity.
The entity coreference graph is inspired by the observation of coreference's role in document understanding. Firstly, for the repeatedly referred entity (coreference entity), the understanding to this entity itself is constantly enriched or enhanced by each reference. For the example illustrated in Figure 4, "the massacre" and "this action" are two different mentions of the same entity. The understanding of this entity is enriched by combining
the location of the massacre mentioned in the first sentence and the commander of the massacre mentioned in the second sentence. Secondly, for other entities located in the context of the coreference entity, their meanings are clearer by recognizing the connotation of the coreference entity. For example, "the colonel" cannot be recognized as an argument unless the model understands that "this action" refers to "the massacre". Research works in event extraction (Xu et al., 2021; Luan et al., 2019; Qian et al., 2019) consider the first observation but neglect the second one. Specifically, previous works in event extraction use graph structure to merge information in different mentions of the same entity. However, such a graph structure cannot feed back the fused information to the context of coreference entities because the representations of the context tokens are fixed from the initial encoding process. In this sense, for the representation of "the colonel", its context information still excludes "the massacre". Therefore, we adopt a graph2token module to feed back the comprehensive entity information obtained through graph structure to tokens, and then rebuild entity representations that are both comprehensive and coreference-aware.
Graph Construction. There are two types of nodes in the entity graph: entity nodes and sentence nodes. Entities are recognized from document following Fisher and Vlachos (2019). Entity nodes and sentence nodes are denoted as  $\mathbf{E} = \{\mathbf{e}_0, \mathbf{e}_1, \dots, \mathbf{e}_p\}$ , and  $\mathbf{S} = \{\mathbf{s}_0, \mathbf{s}_1, \dots, \mathbf{s}_q\}$ , respectively.
There are two types of edges in the entity graph: 1) entity-entity edge is created according to the coreference relationship. We use SpanBERT (Joshi et al., 2020) to implement coreference resolution on documents during preprocessing. 2) entity-sentence edge is the connection between the entity node and the sentence node where it is located.
Graph Propagation. After the graph is constructed, Graph Attention Network (Velicković et al., 2017) is applied to propagate information between connected nodes. Assuming that graph nodes are denoted by  $\mathrm{H} = \{\mathrm{E},\mathrm{S}\} = \{\mathbf{h}_0,\mathbf{h}_1,\dots ,\mathbf{h}_{p + q}\} \in \mathbb{R}^{(p + q)\times 2l}$ , the information that a node receives from its neighbors is formulated as:
$$
\mathbf {h} _ {i} ^ {\prime} = \operatorname {R E L U} \left(\sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i j} \mathbf {W h} _ {j}\right) \tag {3}
$$
$$
\alpha_ {i j} = \frac {\exp \left(L \left(\mathbf {W} _ {e _ {i j}} [ \mathbf {W} \mathbf {h} _ {i} ; \mathbf {W} \mathbf {h} _ {j} ]\right)\right)}{\sum_ {k \in \mathcal {N} _ {i}} \exp \left(L \left(\mathbf {W} _ {e _ {i}} [ \mathbf {W} \mathbf {h} _ {i} ; \mathbf {W} \mathbf {h} _ {k} ]\right)\right)} \tag {4}
$$
where  $\mathbf{h}_i^{\prime}$  is the neighbor information of the i-th node,  $\mathbf{h}_j$  is the representation of the j-th node,  $\mathbf{W}$ ,  $\mathbf{W}_{e_i}$  are weight matrixes,  $\mathcal{N}_i$  denotes the set of neighbors of node  $i$ , and  $L(\cdot)$  is the LeakyReLU function.
The representation of the i-th node  $\mathbf{h}_i$  and its neighbor information  $\mathbf{h}_i^{\prime}$  is fused by the gated mechanism:
$$
\beta_ {i} = \sigma \left(f \left(\mathbf {h} _ {i}; \mathbf {h} _ {i} ^ {\prime}\right)\right) \tag {5}
$$
where  $\sigma (\cdot)$  is the sigmoid function,  $f(\cdot)$  denotes the linear transformation. The fused representation of the i-th node  $\mathbf{h}_i^{\prime \prime}$  is obtained as:
$$
\mathbf {h} _ {i} ^ {\prime \prime} = \beta_ {i} \odot \mathbf {h} _ {i} + (1 - \beta_ {i}) \odot \mathbf {h} _ {i} ^ {\prime} \tag {6}
$$
where  $\odot$  stands for element-wise multiplication. Through propagating and fusing information of coreference entities and the corresponding sentence, a comprehensive representation of the entity is obtained.
Graph2token. To address the second insight we put forward in this section, we adopt the graph2token module to feed back the information behind coreference entities to their neighboring tokens.
We concatenate the original token representation  $\mathbf{d}_i$  with the entity representation  $\mathbf{h}_j^{\prime \prime}$  in which it is located, and feed it to an LSTM layer. In this way, the comprehensive entity representation  $\mathbf{h}_j^{\prime \prime}$  is propagated to context tokens outside the entity and a coreference-aware token representation  $\mathbf{d}_i^{\prime}$  is generated:
$$
\mathbf {d} _ {i} ^ {\prime} = \operatorname {L S T M} \left(\mathbf {d} _ {i}; \mathbf {h} _ {j} ^ {\prime \prime}\right) \tag {7}
$$
Then, we build coreference-aware entity representations from updated token representations.
$$
\mathbf {e} _ {\text {r u l e}} ^ {(i) ^ {\prime}} = \left(D ^ {\prime} \left[ \operatorname {e n t} _ {\text {s t a r t}} ^ {(i)} [: l ] \right]; D ^ {\prime} \left[ \operatorname {e n t} _ {\text {e n d}} ^ {(i)} [ l: ] \right]\right)
$$
where  $\mathbf{D}' = \{\mathbf{d}_0', \mathbf{d}_1', \dots, \mathbf{d}_{n-1}'\}$ . Finally, a comprehensive and coreference-aware entity representation  $\mathbf{E}' = \{\mathbf{e}_0', \mathbf{e}_1', \dots, \mathbf{e}_p'\}$  is obtained by concatenation:
$$
\mathbf {e} _ {i} ^ {\prime} = \left(\mathbf {h} _ {i} ^ {\prime \prime}; \mathbf {e} _ {\text {r u l e}} ^ {(i)} ^ {\prime}\right) \tag {8}
$$
3.3 Closed Boundary Loss
We have analyzed that classifiers trained by cross entropy loss have open decision boundaries and could easily misclassify the Universum class. To
address this problem, we propose a novel loss function that could be used to train classifiers with closed decision boundaries.
Comprehensive and coreference-aware entity representations  $\mathbf{E}^{\prime} = \{\mathbf{e}_0^{\prime},\mathbf{e}_1^{\prime},\dots ,\mathbf{e}_p^{\prime}\}$  are obtained in the last section. We treat entities as argument candidates and classify entities by classifiers trained by our proposed closed boundary loss:
$$
\begin{array}{l} \mathbf {L} _ {\mathbf {C B}} = R ^ {2} + \frac {1}{n} \sum_ {i = 1} ^ {n} \max  \left(0, \left\| \mathbf {e} _ {i} ^ {\prime} - \mathbf {c} \right\| ^ {2} - R ^ {2}\right) \\ + \frac {1}{m} \sum_ {i = 1} ^ {m} \max  \left(0, (1 + \mu) R ^ {2} - \left\| \mathbf {e} _ {i} ^ {\prime} - \mathbf {c} \right\| ^ {2}\right) \\ \end{array}
$$
where  $n$  is the number of target class samples,  $m$  is the number of Universum class samples, the center  $c$  is initialized as the mean of target samples in the feature space, and the radius  $R$  is initialized as  $\nu$ -quantile of the distance of target samples to the center  $c$  in the feature space.  $R$  and  $c$  are initialized after a few warm-up epochs. The closed boundary loss intends to include samples of each target class using a hypersphere characterized by center  $c$  and radius  $R$  in the feature space and locate Universum samples outside the hypersphere. Due to the heterogeneous nature of Universum samples, we allow them to scatter outside the hypersphere and do not require them to be aggregated like cross entropy loss.
The goal of the first term  $R^2$  is to minimize the volume of the hypersphere. The second term aims to enclose target class samples by the hypersphere. If the Euclidean distance between the sample  $\mathbf{h}_i''$  and the center  $\mathbf{c}$  exceeds the radius, it will lead to a penalty in the loss function. The third term aims to keep the universe samples outside the hypersphere. Parameter  $\mu$  is introduced to adjust the gap between the closed boundary hypersphere and Universum samples.