SlowGuess/NAACL_2022_OCR huggingface dataset:
Yuan Liang; Zhuoxuan Jiang; Di Yin and Bo Ren
Tencent Cloud, China
{ericyliang, alexzxjiang, endymecyyin, timren} @tencent.com
In document-level event extraction (DEE) task, event arguments always scatter across sentences (across-sentence issue) and multiple events may lie in one document (multi-event issue). In this paper, we argue that the relation information of event arguments is of great significance for addressing the above two issues, and propose a new DEE framework which can model the relation dependencies, called Relation-augmented Document-level Event Extraction (ReDEE). More specifically, this framework features a novel and tailored transformer, named as Relation-augmented Attention Transformer (RAAT). RAAT is scalable to capture multi-scale and multi-amount argument relations. To further leverage relation information, we introduce a separate event relation prediction task and adopt multi-task learning method to explicitly enhance event extraction performance. Extensive experiments demonstrate the effectiveness of the proposed method, which can achieve state-of-the-art performance on two public datasets. Our code is available at https://github.com/TencentYoutuResearch/RAAT.
Event extraction (EE) task aims to detect the event from texts and then extracts corresponding arguments as different roles, so as to provide a structural information for massive downstream applications, such as recommendation (Gao et al., 2016; Liu et al., 2017), knowledge graph construction (Wu et al., 2019; Bosselut et al., 2021) and intelligent question answering (Boyd-Graber and Borschinger, 2020; Cao et al., 2020).
Most of the previous methods focus on sentence-level event extraction (SEE) (Ahn, 2006; Liao and Grishman, 2010; Li et al., 2013; Chen et al., 2015; Nguyen et al., 2016; Zhao et al., 2018; Sha et al., 2018; Yan et al., 2019; Du and Cardie, 2020; Li
et al., 2020; Paolini et al., 2021; Lu et al., 2021), extracting events from a single sentence. However, SEE is mostly inconsistent with actual situations. For example, event arguments may scatter across different sentences. As illustrated in Figure 1, the event argument [ORG1] of event role Pledger is mentioned in Sentence 4 and the corresponding argument [ORG2] of event role Pledger is in Sentence 5 and 6. We call this across-sentence issue. Another situation involves the multi-event issue, which means that multiple events may exist in the same document. As seen in the example in Figure 1, where two event records coincide, we should recognize that they may partially share common arguments.
Recently, document-level event extraction (DEE) attracts great attention from both academic and industrial communities, and is regarded as a promising direction to tackle the above issues (Yang et al., 2018; Zheng et al., 2019; Xu et al., 2021b; Yang et al., 2021; Zhu et al., 2021). However, by our observation, we discover that the relations between event arguments have patterns which are an important indicator to guide the event extraction. This information is neglected by existing DEE methods. Intuitively, the relation information could build long-range relationship knowledge of event roles among multiple sentences, which could relieve the across-sentence issue. For multi-event issue, shared arguments within one document could be distinguished to different roles based on the different prior relation knowledge. As illustrated in Figure 1, [ORG1] and [ORG2] have a prior relation pattern of Pledger and Pledgergee, as well as [ORG1] and [SHARE1] for the relation pattern between Pledger and its Pledged Shares. Therefore, the relation information could increase the DEE accuracy if it is well modeled.
In this paper, we propose a novel DEE framework, called Relation-augmented Document-level Event Extraction (ReDEE), which is able to model
Human Language Technologies, pages 4985 - 4997
Event Records for the Equity Pledge (EP) Event Type (Two Examples)
Record ID	Pledger	Pledged Shares	Pledgee	Total Holding Shares	Total Holding Ratio	Total Pledged Shares	Start Date	Release Date
1	[ORG1]	[SHARE1]	[ORG2]	[SHARE2]	[RATIO1]	[SHARE2]	[TIME1]	[TIME2]
2	[ORG1]	[SHARE2]	[ORG3]	[SHARE2]	[RATIO1]	[SHARE2]	[TIME2]	-
Mark-Entity Mapping Table
Mark	Entity
[ORG1]	Shenzhen Nanfang Tongzheng Investment Co., Ltd.
[ORG2]	Chengqing Branch of China Zheshang Bank Co., Ltd.
[ORG3]	Jiatianxia Asset Management Co., Ltd
[TIME1]	September 18, 2017
[TIME2]	September 20, 2018
[SHARE1]	10,000,000
[SHARE2]	10,072,158
[RATIO1]	6.57%
ID	Selected Sentences of a Document
S4	Chongqing Wanli New Energy Co., Ltd. (hereinafter referred to as the “Company” or “the Company”) received on September 21, 2018 that [ORG1] ...
S5	On [TIME1], Nanfang Tongzheng pledged its [SHARE1] unrestricted tradable shares of the company to [ORG2] for ...
S6	Nanfang Tongzheng has released all the above-mentioned [SHARE1] shares pledged to [ORG2], and ... on [TIME2] ...
S8	According to ... [ORG3], Nanfang Tongcheng pledged its [SHARE2] unrestricted tradable shares of the company to [ORG3], ...([TIME2]) until ...
S9	As ... [SHARE2] shares of the company, accounting for [RATIO1] of the company’s total share capital, and the cumulative number of ledged shares is [SHARE2] ...
Event-related Relations between Entities Table (Two Examples)
Head Entity	Tail Entity	Relation
[ORG1]	[ORG2]	Pledger and Pledgee
[ORG1]	[SHARE1]	Pledger with Pledged Shares
Figure 1: An example document for the event type of Equity Pledge, including selected sentences that are involved in multiple event records and where the event arguments scatter across sentences. We can observe that the relations between these entity mentions have intuitive patterns that could be leveraged to enhance the event extraction task. More information of entity color and complete event-related relations can be found in Appendix A.2.
the relation information between arguments by designing a tailored transformer structure. This structure can cover multi-scale and multi-amount relations and is general for different relation modeling situations. We name the structure as Relation-augmented Attention Transformer (RAAT). To fully leverage the relation information, we introduce a relation prediction task into the ReDEE framework and adopt multi-task learning method to optimize the event extraction task. We conduct extensive experiments on two public datasets. The results demonstrate the effectiveness of modeling the relation information, as well as our proposed framework and method.
- We propose a Relation-augmented Document-level Event Extraction (ReDEE) framework. It is the first time that relation information is implemented in the document-level event extraction field.
- We design a novel Relation-augmented Attention Transformer (RAAT). This network is general to cover multi-scale and multi-amount relations in DEE.
- We conduct extensive experiments and the results demonstrate that our method outper
form the baselines and achieve state-of-the-art performance by  $1.6\%$  and  $2.8\%$  F1 absolute increasing on two datasets respectively.
Previously, most of the related works focus on sentence-level event extraction. For example, a neural pipeline model is proposed to identify triggers first and then extracts roles and arguments (Chen et al., 2015). Then a joint model is created to extract triggers and arguments simultaneously via multi-task learning (Nguyen et al., 2016; Sha et al., 2018). To utilize more knowledge, some studies propose to leverage document contexts (Chen et al., 2018; Zhao et al., 2018), pre-trained language models (Yang et al., 2019), and explicit external knowledge (Liu et al., 2019a; Tong et al., 2020). However, these sentence-level models fail to extract multiple qualified events spanning across sentences, while document-level event extraction is a more common need in real-world scenarios.
Recently, DEE has attracted a great attention from both academic and industrial communities. At first, the event is identified from a central sentence and
other arguments are extracted from neighboring sentences separately (Yang et al., 2018). Later, an innovative end-to-end model Doc2EDAG, is proposed (Zheng et al., 2019), which can generate event records via an entity-based directed acyclic graph to fulfill the document-level event extraction effectively. Based on Doc2EDAG, there are some variants appearing. For instance, GIT (Xu et al., 2021b) designs a heterogeneous graph interaction network to capture global interaction information among different sentences and entity mentions. It also introduces a specific Tracker module to memorize the already extracted event arguments for assisting record generation during next iterations. DE-PPN (Yang et al., 2021) is a multi-granularity model that can generate event records via limiting the number of record queries. Not long ago, a pruned complete graph-based non-autoregressive model PTPCG was proposed to speedup the record decoding and get competitive overall evaluation results (Zhu et al., 2021). In summary, although those existing works target for solving across-sentence and multi-event issues of the DEE task from various perspectives, to our best knowledge, we conduct a pioneer investigation on relation modeling towards this research field in this paper.
2.3 Trigger-aware Event Extraction
Previously a lot of works((Ji and Grishman, 2008; Liao and Grishman, 2010; Li et al., 2013; Chen et al., 2015; Nguyen et al., 2016; Liu et al., 2018)) deal with event extraction in two stages: firstly, trigger words are detected, which are usually nouns or verbs that clearly express event occurrences. And secondly, event arguments, the main attributes of events, are extracted by modeling relationships between triggers and themselves. In our work, we unify task as a whole to avoid error propagation between sub-tasks.
Firstly, we clarify several key concepts in event extraction tasks. 1) entity: a real world object, such as person, organization, location, etc.2) entity mention: a text span in document referring to an entity object. 3) event role: an attribute corresponding a pre-defined field in an event. 4) event argument: an entity playing a specific event role. 5) event record: a record expressing an event itself, including a series of event arguments.
In document-level event extraction task, one doc
ument can contain multiple event records, and an event record may miss a small set of event arguments. Further more, a entity can have multiple event mentions.
In this section, we introduce the proposed architecture first and then the key components in detail.
4.1 Architecture Overview
End-to-end training methods for DEE usually involve a pipeline paradigm, including three subtasks: named entity recognition, event role prediction and event argument extraction. In this paper, we propose the Relation-augmented Document-level Event Extraction (ReDEE) framework coordinated with the paradigm. Our framework features leverage the relation dependency information in both encoding and decoding stages. Moreover, a relation prediction task is added into the framework to fully utilize the relation knowledge and enhance the event extraction task.
More specifically, shown in Figure 2, there are four key components in our ReDEE framework: Entity Extraction and Representation(EER), Document Relation Extraction(DRE), Entity and Sentence Encoding(ese), and Event Record Generation(ERG). In the following, we would introduce the detailed definition of each component.
4.2 Entity Extraction and Representation
We treat the component of entity extraction as a sequence labeling task. Given a document  $D$  with multiple sentences  $\{s_1,s_2,\dots,s_i\}$ , we use a native transformer encoder to represent the token sequence. Specifically, we use the BERT (Devlin et al., 2019) encoder pre-trained in Roberta setting (Liu et al., 2019). Then we use the Conditional Random Field(CRF) (Lafferty et al., 2001) to classify token representations into labels of named entities. We adopt the classical BIOSE sequence labeling scheme. The labels are predicted by the following calculation:  $\hat{y}_{ne} = CRF(Trans(D))$ . Then all the intermediate embeddings of extracted entity mentions and sentences are concatenate into a matrix  $M_{ne + s}\in \mathbb{R}^{(j + i)\times d_e}$  by max-pooling operation on each sentence and entity mention span, where  $j$  and  $i$  are the numbers of entity mentions and sentences, and  $d_{e}$  is the dimension of embeddings. The loss function for named entity recognition is
Figure 2: Overall of our proposed ReDEE framework.
denoted:
$$
\mathcal {L} _ {n e} = - \sum_ {s _ {i} \in D} \log P \left(y _ {i} \mid s _ {i}\right) \tag {1}
$$
where  $s_i$  denotes the  $i^{th}$  sequence sentence in document, and  $y_i$  is the corresponding ground truth label sequence.
4.3 Document Relation Extraction
The DRE component takes the document text  $(D)$  and entities  $\left(\{e_1,e_2,\dots ,e_j\}\right)$  extracted in the previous step as inputs, and outputs the relation pairs among entities, in a form of triples  $\left(\left\{\left[e_1^h,e_1^t,r_1\right],\left[e_2^h,e_2^t,r_2\right],\ldots ,\left[e_k^h,e_k^t,r_k\right]\right\}\right)$ .  $\left[e_k^h,e_k^t,r_k\right]$  means the head entity, the tail entity and the relationship of the  $k^{th}$  triple respectively.
An important aspect is how to define and collect the relations from data. Here we assume that every two arguments in an event record can be connected by a relation. For example, Pledger and Pledgerge in the EquityPledge event could have a relation named as Pledger2Pledgerge, and the order of head and tail entities is determined by the pre-order of event arguments (Zheng et al., 2019). In this way, every event record with  $n$  arguments could create  $C_n^2$  relation samples. Note that this method to build relations is general to event extraction tasks from various domains, and the supervised relation information just comes from event record data itself, without any extra human labeling work. We do statistics for the relation types for ChiFinAnn dataset. Table 1 shows a snippet of statistics and the full edition can be found in Appendix A.3.
Relation Type	#Train	#Dev	#Test
Pledger2PledgedShares	20002	2567	2299
Pledger2Pledgee	20002	2567	2299
PledgedShares2Pledgee	20002	2567	2299
Start2EndDate	19615	2239	1877
Pledger2TotalHoldingShares	18552	2412	2173
Table 1: The example relations with top 5 quantities in the ChiFinAnn dataset. The complete statistic can refer to the Appendix A.3.
To predict the argument relations in this step, we adopt the structured self attention network (Xu et al., 2021a) which is the latest method for document-level relation extraction. However, different from previous work using multi-class binary cross-entropy loss, we use normal cross-entropy loss to predict only one label for each entity pair. The relation type is inferred by this function:
$$
\hat {y} _ {i, j} = \operatorname {a r g m a x} \left(e _ {i} ^ {T} W _ {r} e _ {j}\right) \tag {2}
$$
where  $e_i, e_j \in \mathbb{R}^d$  denote entity embedding from encoder module of DRE and  $d$  is the dimension of embeddings.  $W_r \in \mathbb{R}^{d \times c \times d}$  denotes biaffine matrix trained by DRE task and  $c$  is the total number of relations. And the loss function for optimize the relation prediction task is denoted:
$$
\mathcal {L} _ {d r e} = - \sum_ {y _ {i, j} \in Y} \log P \left(y _ {i, j} \mid D\right) \tag {3}
$$
where  $y_{i,j}$  denotes ground truth label between the  $i^{th}$  and  $j^{th}$  entity,  $D$  for document text and  $Y$  for set of all relation pairs among entities.
Figure 3: RAAT structure. Firstly each relation between entities and sentences are represented as matrices. Then the matrices are clustered by the head entities. At last the clustered matrices are integrated into the transformer structure for attention calculation.
4.4 Entity and Sentence Encoding
Now we have embeddings of entity mentions and sentences from EER component and a list of predicted triple relations from DRE component. Then this component encodes data mentioned above and output embeddings effectively integrated with relation information. In this subsection, we would introduce the method that translates triple relations to calculable matrices and the novel RAAT structure for encoding all the above data.
4.4.1 Entity and Sentence Dependency
First, we introduce a mechanism: entity and sentence dependency, which not only includes relation triples, but also describes links among sentences and entities beyond triples.
Co-relation and Co-reference are defined to represent entity-entity dependency. For the former one, two entities have a Co-relation dependency between them if they belong to a predicted relation triple. Entity pairs are considered having different Co-relation if their involved triples have different relations. Co-reference shows dependency between entity mentions pointing to same entities. That is, if an entity has several mentions existing across document, then each two of them has Co-reference dependency. However, in the case that
sentence	entity
sentence	NA	Co-existence/NA
entity	Co-existence/NA	Co-relation/Co-reference/NA
Table 2: All types of dependency among sentences and entities
head and tail entities in relation triple are the same (i.e. StartDate and EndDate share same entities in some event records), then Co-relation and Co-reference are both held between them.
We use Co-existence to describe dependency between entities and sentences where entity mentions come from. To be more specific, the entity mention together with its belonged sentence has Coexistence. For remaining entity-entity and entity-sentence pairs without any dependency mentioned above, we uniformly treat them as  $NA$  dependency.
Table 2 shows the complete dependency mechanism. Co-relation differs from NA, Co-reference, and Co-existence in that it has several sub-types, with number equaling to that of relation types defined in document relation extraction task.
4.4.2 RAAT
In order to effectively encode entity and sentence dependencies, we design the RAAT which takes advantage of a calculable matrix representing dependencies and integrates it into attention computation. According to the architecture shown in Figure 3, RAAT is inherited from native transformer but has a distinct attention computation module which is made up of two parts: self-attention and relation-augmented attention computation.
Given a document shown as  $D = \{s_1, s_2, \ldots, s_j\}$ , all entity mentions in this document as  $E^m = \{e_1^m, e_2^m, \ldots, e_t^m\}$ , where  $e_i^m$  denotes entity mentions with the superscript  $m$  denotes mention, and the subscript  $i$  denotes index, and a list of triples  $\{[e_1^h, e_1^t, r_1], [e_2^h, e_2^t, r_2], \ldots, [e_k^h, e_k^t, r_k]\}$ , we build a matrix  $T \in \mathbb{R}^{c \times (t+j) \times (t+j)}$  where  $c$  for the number of dependencies, and  $t$  and  $j$  for the number of sentences and entity mentions respectively.  $T$  is comprised of  $c$  matrices with same dimensions  $R \in \mathbb{R}^{(t+j) \times (t+j)}$ , and each  $R$  represents one type of dependency  $r \in \{Co - relation_k, Co - reference, Co - existence, NA\}$ ,  $k = 1, 2, \ldots, N$ ,  $N$  as the number of relation types. For element within  $T$ ,  $t_{k,i,j}$  represent the dependency between node  $i$  and node  $j$ . Specifically,  $t_{k,i,j} = 1$  if they have the  $k^{th}$  dependency, otherwise,  $t_{k,i,j} = 0$ . Here, node  $k \in \{e_1^m, e_2^m, \ldots, e_t^m, s_1, s_2, \ldots, s_j\}$  can
be either entity mention or sentence.
However,  $T$  would be giant and sparse if we use the above strategy. To squeeze  $T$  and decrease training parameters, we cluster Co-relation dependency based on the type of head entity in relation triple. For example, Pledger2Pledgergee and Pledger2PledgedShares are clustered as one Co-relation dependency, and two matrice  $R_{a}$  and  $R_{b}$  corresponding to them are merged into one matrix. As a result, we finally get  $T \in \mathbb{R}^{(3 + H) \times (t + j) \times (t + j)}$  where  $\mathrm{H}$  denotes the number of head entity type in Co-relation, and 3 covers NA, Co-reference, and Co-existence. Let  $X \in \mathbb{R}^{(t + j) \times d}$  as input embeddings of attention module,  $W_{rq}, W_{rk}, W_{q}, W_{k}, W_{v} \in \mathbb{R}^{d \times d}$ ,  $M \in \mathbb{R}^{(3 + H) \times d \times d}$  as weight matrices, we compute relation-augmented attention in the following steps:
$$
Q _ {r} = X W _ {r q}, K _ {r} = X W _ {r k} \tag {4}
$$
$$
S _ {a} = \frac {\sum_ {i = 1} ^ {3 + H} Q _ {r} M [ i , \therefore , : ] K _ {r} ^ {T} \cdot T [ i , \therefore , : ]}{\sqrt {d}} + b i a s _ {i} \tag {5}
$$
where  $S_{a}$  denotes score matrix of relationaugmented attention,  $\cdot$  denotes element-wise multiplication. We compute self attention score and combine it with  $S_{a}$  in the following way:
$$
Q = X W _ {q}, K = X W _ {k}, W _ {v} = X W _ {v} \tag {6}
$$
$$
S _ {b} = \frac {Q K ^ {T}}{\sqrt {d}} \tag {7}
$$
$$
O = \left(S _ {a} + S _ {b}\right) V \tag {8}
$$
where  $O$  is the output of attention module. Similar to the structure of native transformer, RAAT has multiple identical blocks stacking up layer by layer. Furthermore,  $T$  is extensive since the number of Co-relation can be selected. RAAT can be adaptive to the change of input length, which is equivalent to the total number of sentences and entity mentions.
4.5 Event Record Generation
With the outputs from previous component, the embeddings of entities and sentences, this ERG component actually includes two sub-modules: event type classifier and event record decoder.
4.5.1 Event Type Classifier
Given the embeddings of sentences, we adopt several binary classifiers on every event type to predict whether the corresponding event is identified or
not. If there is any classifier identifying an event type, the following event record decoder would be activated to iteratively generate every argument for the corresponding event type. The loss function to optimize this classifier is as the following:
$$
\mathcal {L} _ {p r e d} = - \sum_ {i} \log (P \left(y _ {i} \mid S\right)) \tag {9}
$$
where  $y_{i}$  denotes the label of the  $i^{th}$  event type,  $y_{i} = 1$  if there exists event record with event type i, otherwise,  $y_{i} = 0$ .  $S$  denotes input embeddings of sentences.
4.5.2 Event Record Decoder
To iteratively generate every argument for a specific event type, we refer to the entity-based directed acyclic graph (EDAG) method (Zheng et al., 2019). EDAG is a sequence of iterations with the length equaling to number of roles for certain event type. The objective of each iteration is to predict event argument of certain event role. Inputs of each iteration are come up with entities and sentences embeddings. And the predicted arguments of outputs will be a part of inputs for next iteration. However, different from EDAG, we substitute its vanilla transformer part with our proposed RAAT structure (i.e. RAAT-2 as shown in Figure 2). More specifically, the EDAG method uses a memory structure to record extracted arguments and adds role type representation to predict current-iteration arguments. However, this procedure hardly captures dependency between entities both in memory and argument candidates and sentences. In our method, RAAT structure can connect entities in memory and candidate arguments via relation triples extracted by the DRE component, and it can construct a structure to represent dependencies. In detail, before predicting event argument for current iteration, Matrix  $T$  is constructed in the way shown above so that dependency is integrated into attention computation. After extracting the argument, it is added into memory, meanwhile, a new  $T$  is generated to adapt next iteration prediction.
Therefore, the RAAT can strengthen the relation signal for attention computation. The RAAT-2 has the same structure with RAAT-1 but independent parameters. The formal definition of loss function for event recorder decoder is:
$$
\mathcal {L} _ {a} = - \sum_ {v \in V _ {D}} \sum_ {e} \log (P \left(y _ {e} | (v, s)\right)) \tag {10}
$$
where  $V_{D}$  denotes node set in event records graph,  $v$  denotes extracted event arguments of event record