SlowGuess/NAACL_2022_OCR huggingface dataset:
Runxin Xu $^{1*}$ , Peiyi Wang $^{1*}$ , Tianyu Liu $^{3}$ , Shuang Zeng $^{1,2}$ , Baobao Chang $^{1}$ , Zhifang Sui $^{1}$
<sup>1</sup>Key Laboratory of Computational Linguistics, Peking University, MOE, China
3Tencent Cloud Xiaowei
{runxinxu, wangpeiyi9979}  $@$  gmail.com, rogertyliu@tecent.com
{zengs,chbb,szf}@pku.edu.cn
Most previous studies aim at extracting events from a single sentence, while document-level event extraction still remains under-explored. In this paper, we focus on extracting event arguments from an entire document, which mainly faces two critical problems: a) the long-distance dependency between trigger and arguments over sentences; b) the distracting context towards an event in the document. To address these issues, we propose a Two-Stream Abstract meaning Representation enhanced extraction model (TSAR). TSAR encodes the document from different perspectives by a two-stream encoding module, to utilize local and global information and lower the impact of distracting context. Besides, TSAR introduces an AMR-guided interaction module to capture both intra-sentential and inter-sentential features, based on the locally and globally constructed AMR semantic graphs. An auxiliary boundary loss is introduced to enhance the boundary information for text spans explicitly. Extensive experiments illustrate that TSAR outperforms previous state-of-the-art by a large margin, with 2.54 F1 and 5.13 F1 performance gain on the public RAMS and WikiEvents datasets respectively, showing the superiority in the cross-sentence arguments extraction. We release our code in https://github.com/ PKUnlp-icler/TSAR.
Event Argument Extraction (EAE) aims at identifying the entities that serve as event arguments, and predicting the roles they play in the event, which is a key step for Event Extraction (EE). It helps to transform the unstructured text into structured event knowledge that can be further utilized in recommendation systems (Li et al., 2020), dialogue systems (Zhang et al., 2020a), and so on. Most previous studies assume that the events are only
[1] A ship carrying half a million barrels of oil that was pumped in the U.S. docked at a terminal owned by Venezuela last week, according to oil data research firm ClipperData. Vehicle origin artifact
[2] The shipment was sent to a facility located on Dutch Island
Transport destination importer
of Curacao in Caribbean. [3] The fact that Venezuela is import
ing American oil is raising eyebrows because Venezuela has
298 billion barrels of oil reserves, according to the Energy
Information Administration. [4] That's more than Saudi Arabia,
Russia or Iran and eight times the reserves of the United States.
Figure 1: A document from RAMS dataset (Ebner et al., 2020). A transport event is triggered by shipment, with five event arguments scattering across the document.
expressed by a single sentence and hence focus on sentence-level extraction (Chen et al., 2015; Liu et al., 2018; Zhou et al., 2021). However, in real-life scenarios, the events are often described through a whole document consisting of multiple sentences (e.g., a news article or a financial report), which still remains under-explored.
Figure 1 illustrates an example of document-level EAE, in which a Transport event is triggered by shipment. Different from sentence-level EAE, extracting arguments out of the entire document faces two critical challenges. (1) Long-distance dependency among trigger and arguments. The arguments are usually located in different sentences from the trigger word and their distance can be quite far away. In Figure 1, while the trigger shipment is in Sentence 2, the vehicle, origin, artifact, and importer arguments are located in Sentence 1 or 3, which highly increases the extraction difficulty. To accommodate the long-range extraction, not only intra-sentential but also inter-sentential semantics should be well modeled. (2) Distracting context. While a document naturally encompasses more context than a single sentence, some distracting context can mislead the argument extraction.
Human Language Technologies, pages 5025 - 5036
As shown in Figure 1, the origin argument U.S. can be identified more easily without Sentence 4, which does not offer useful information for the event, but contains many place entities that can be distracting, like Saudi Arabia and Russia or Iran. It remains challenging to pinpoint the useful context while discarding the distracting information.
Recently, Du and Cardie (2020a) use a tagging-based method, which fails to handle nested arguments. Instead, span-based methods predict argument roles for candidate spans (Ebner et al., 2020; Zhang et al., 2020b). Some studies directly generate arguments based on sequence-to-sequence model (Li et al., 2021). However, how to model long-distance dependency among trigger and arguments, and how to handle distracting context explicitly, remain largely under-explored.
In this paper, to tackle the aforementioned two problems, we propose a Two-Stream AMR-enhanced extraction model (TSAR). In order to take advantage of the essential context in the document, and avoid being misled by distractions, we introduce a two-stream encoding module. It consists of a global encoder that encodes global semantics with as much context as possible to gather adequate context information, and a local encoder that focuses on the most essential information and prudently takes in extra context. In this way, TSAR can leverage complementary advantages of different encoding perspectives, and therefore make better use of the feasible context to benefit the extraction. Besides, to model the long-distance dependency, we introduce an AMR-guided interaction module. Abstract Meaning Representation (AMR, Banarescu et al., 2013) graph contains rich hierarchical semantic relations among different concepts, which makes it favorable for complex event extraction. From such a linguistic-driven angle, we turn the linear structure of the document into both global and local graph structures, followed by a graph neural network to enhance the interactions, especially for those non-local elements. Finally, as TSAR extracts arguments in span level, where the span boundary may be ambiguous, we introduce an auxiliary boundary loss to enhance span representation with calibrated boundary.
To summarize, our contributions are three-fold. 1) We propose a two-stream encoding module for document-level EAE, which encodes the document through two different perspectives to better utilize the context. 2) We introduce an AMR-guided in
teraction module to facilitate the semantic interactions within the document, so that long-distance dependency can be better captured. 3) Our experiments show that TSAR outperforms the previous start-of-the-art model by large margins, with 2.54 F1 and 5.13 F1 improvements on public RAMS and WikiEvents datasets respectively, especially on cross-sentence event arguments extraction.
2.1 Sentence-level Event Extraction
Previous studies mainly focus on sentence-level event extraction. Li et al. (2014) and Judea and Strube (2016) use handcrafted features to extract events from the sentence. Chen et al. (2015) firstly propose a neural pipeline model to extract events, while Nguyen et al. (2016) utilize a joint model to mitigate error propagation. To better model the interactions among words, Liu et al. (2018); Yan et al. (2019); Ma et al. (2020) make use of the dependency tree, and Wadden et al. (2019) enumerates all possible spans and propagate information in the span graph. Data augmentation is also considered (Yang et al., 2019). Moreover, some works try to reformulate the event extraction task as other tasks. For example, Du and Cardie (2020b) and Zhou et al. (2021) cast event extraction as question answering, and Xiangyu et al. (2021) model it as a sequence-to-sequence task. However, all of these models can only extract events from a single sentence. Thus, they fail to handle the much more common cases, where event arguments usually spread over multiple sentences within the document.
2.2 Document-level Event Extraction
In order to extract events from a whole piece of article with multiple sentences, document-level event extraction has attracted more and more attention recently. Yang and Mitchell (2016) utilize well-defined features to extract arguments across sentences, while most recent methods are based on neural networks. Some studies first identify entities in the document, followed by assigning these entities as specific argument roles (Yang et al., 2018; Zheng et al., 2019; Xu et al., 2021). Differently, some studies try to jointly extract entities and argument roles simultaneously, which can be further divided into tagging-based and span-based methods. Tagging-based methods directly conduct sequence labeling for each token in the document with BIO-schema (Du and Cardie, 2020a; Veyseh
Figure 2: Overview of our TSAR. Firstly, taking an entire document as input, TSAR first encodes the document by the two-stream encoding module, where the global and local encoders with different attention reception fields are used to capture the context in different scopes. Then the AMR-guided interaction module constructs global AMR graphs and local ones to stimulate the interactions among concepts in the document, especially those far away from each other, based on graph neural network. Next, the information fusion module fuses the two-stream representations, and also strengthens the boundary information through a boundary loss. Finally, the classification module makes predictions for candidate spans. For conciseness, we assume the document has three sentences,  $S_{1}$ ,  $S_{2}$ ,  $S_{3}$ , and the event is triggered by  $y_{2}$  with  $[z_{2}, z_{3}]$  being a candidate argument span.
et al., 2021), while span-based methods predict the argument role for candidate text spans which usually have a maximum length limitation (Ebner et al., 2020; Zhang et al., 2020b). Another line of studies reformulate the task as a sequence-to-sequence task (Du et al., 2021a,b; Li et al., 2021), or machine reading comprehension task (Wei et al., 2021).
As a span-based method, TSAR is different from prior methods that simply encode it as a long sentence. Instead, TSAR introduces a two-stream encoding module and AMR-guided interactions module to model intra-sentential and inter-sentential semantics, along with an auxiliary boundary loss to enhance span boundary information.
Following Ebner et al. (2020), we formulate doc-level event argument extraction as follows. We define that a document  $\mathcal{D}$  consists of  $N$  sentences, and a sentence is comprised of a sequence of words, i.e.,  $\mathcal{D} = \{w_1, w_2, \dots, w_{|\mathcal{D}|}\}$ , and  $\mathrm{SEN}(w_i) \in [1, N]$  refers to the sentence that  $w_i$  belongs to. We also define the event types set  $\mathcal{E}$  and the corresponding argument roles set  $\mathcal{R}_e$  for each event type  $e \in \mathcal{E}$ .
Then, given a document  $\mathcal{D}$  and the trigger  $t\in \mathcal{D}$  triggering the event type  $e\in \mathcal{E}$ , the task aims to detect all  $(r,s)$  pairs for the event, where  $r\in \mathcal{R}_e$  is an argument role for the event type  $e$ , and  $s\subseteq \mathcal{D}$  is a contiguous text span in the document.
Figure 2 shows the overall architecture of our model TSAR. The document is fed into the two-stream encoding module, followed by the AMR-guided interaction module to derive both global and local contextualized representations. The information fusion module fuses these two-stream representations, and the classification module finally predicts argument roles for candidate spans.
4.1 Two-Stream Encoding Module
Although more context is provided by the document, it also inevitably introduces irrelevant and distracting information towards the event. These noise signals can be harmful to the argument extraction as shown in Figure 1. To capture useful information and filter distracting one, we propose a two-stream encoding module, consisting of a global
He was then arrested in Iran and was reportedly tried for treason.
Figure 3: The AMR graph provides abstract and logical semantic information, where the nodes denote the concepts and the edges refer to different relation types. The corresponding text spans for nodes are omitted.
encoder that is aware of all context, and a local encoder that only prudently focuses on the most essential information. Therefore, we can leverage their complementary advantages to make better use of the context information.
Specifically, the global and local encoders share the same Transformer-based pre-trained language model such as BERT. By controlling the reception field of the words in the self-attention module, we can encode the document from different perspectives. In the global encoder, the attention technique is the same as the traditional Transformer:
$$
\operatorname {A t t e n t i o n} ^ {G} (Q, K, V) = \operatorname {s o f t m a x} \left(\frac {Q K ^ {\top}}{\sqrt {d _ {m}}}\right) V
$$
where  $Q, K, V$  refers to query, key, and value matrix, and  $d_m$  is the model dimension. However, in the local encoder, we introduce a mask matrix  $M$ , such that tokens can only attend to the sentence itself and the sentence where the trigger locates, to avoid redundant distracting information:
$$
\operatorname {A t t e n t i o n} ^ {L} (Q, K, V) = \operatorname {s o f t m a x} \left(\frac {Q K ^ {\top} + M}{\sqrt {d _ {m}}}\right) V
$$
$$
M _ {i j} = \left\{ \begin{array}{l l} 0, & \text {S E N} (w _ {j}) \in \{\text {S E N} (w _ {i}), \text {S E N} (t) \} \\ - \infty , & \text {O t h e r w i s e} \end{array} \right.
$$
where  $\mathrm{SEN}(w_i)$  is the sentence that the word  $w_i$  belongs to, and  $t$  refers to the trigger of the event.
Hence, we encode the document with two different streams, a global encoder  $\mathrm{Encoder}^G$  and a local encoder  $\mathrm{Encoder}^L$ , finally deriving two representations,  $Z^G$  and  $Z^L$ :
$$
\begin{array}{l} Z ^ {G} = \left[ z _ {1} ^ {G}, z _ {2} ^ {G}, \dots , z _ {| \mathcal {D} |} ^ {G} \right] = \operatorname {E n c o d e r} ^ {G} \left(\left[ w _ {1}, w _ {2}, \dots , w _ {| \mathcal {D} |} \right]\right) \\ Z ^ {L} = \left[ z _ {1} ^ {L}, z _ {2} ^ {L}, \dots , z _ {| \mathcal {D} |} ^ {L} \right] = \operatorname {E n c o d e r} ^ {L} \left(\left[ w _ {1}, w _ {2}, \dots , w _ {| \mathcal {D} |} \right]\right) \\ \end{array}
$$
4.2 AMR-Guided Interaction Module
One key challenge to extract arguments from the document is to capture the intra-sentential and
inter-sentential features. Therefore, we propose an AMR-guided interaction module that adopts Abstract Meaning Representation (AMR, Banarescu et al., 2013) graph to provide rich semantic structure to facilitate the interactions among concepts, which also offers logical meanings of the document from a linguistic-driven perspective to benefit the language understanding.
AMR semantic graph models the meaning representations of a sentence as a rooted, directed, labeled graph. Concretely, with an AMR parser, a natural sentence can be parsed into an AMR graph  $G = (V, E)$ . The node  $v = (a, b) \in V$  represents a concept that corresponds to the span ranging from  $w_{a}$  to  $w_{b}$  in the origin sentence, while the edge represents a specific AMR relation (detail in Appendix A). Thus, AMR focuses on semantic relations rather than syntactic ones, which is more high-level and beneficial to event understanding, and the structures are more close to the event trigger-arguments structures. For example, Figure 3 demonstrates how a sentence is parsed into an AMR semantic graph. As event arguments play essential roles in the text, most of them would be involved, if not all, in the AMR graphs (90% and 88% arguments in RAMS and WikiEvents datasets). We use the state-of-the-art AMR parser Fernandez Astudillo et al. (2020), which achieves satisfactory results (up to 81.3 Smatch on AMR2.0 data) for downstream application. As the number of AMR relation types is large, which results in too many demanded parameters, we also follow Zhang and Ji (2021) to cluster the relation types into main categories. More details can be found in Appendix A.
The AMR-guided interaction module is attached after the global and local encoders as shown in Figure 2. We use the AMR graphs as skeletons for information interactions, under a composition, interaction, and decomposition paradigm.
From the local perspective, we construct AMR graphs for each sentence in the document, and they are isolated from each other. For initialization, the vector representation of node  $u = (a_u, b_u)$  is composed by averaging the local representations of its corresponding text span:
$$
h _ {u} ^ {0} = \frac {1}{| b _ {u} - a _ {u} + 1 |} \sum_ {i = a _ {u}} ^ {b _ {u}} z _ {i} ^ {L}
$$
Similar to Zeng et al. (2020), we then use  $L$ -layer stacked Graph Convolution Network (Kipf and
Welling, 2017) to model the interactions among different concept nodes through edges with different relation types. Given node  $u$  at the  $l$ -th layer, the information interaction and aggregation operation is defined as follows:
$$
h _ {u} ^ {(l + 1)} = \operatorname {R e L U} \left(\sum_ {k \in \mathcal {K}} \sum_ {v \in \mathcal {N} _ {k} (u) \bigcup \{u \}} \frac {1}{c _ {u , k}} W _ {k} ^ {(l)} h _ {v} ^ {(l)}\right)
$$
where  $\mathcal{K}$  denotes different relation types,  $\mathcal{N}_k(u)$  denotes the neighbors for  $u$  connected with  $k$ -th relation types and  $c_{u,k}$  is a normalization constant. Besides,  $W_k^{(l)} \in \mathbb{R}^{d_m \times d_m}$  is a trainable parameter.
Finally, we concatenate vectors in all layers and derive the final node representation by  $h_u = W_1[h_u^0; h_u^1; \ldots; h_u^L] \in \mathbb{R}^{d_m}$ . Then  $h_u$  is decomposed into the local representations of corresponding words, followed by token-wise aggregation, where  $\mathbb{I}(\cdot)$  refers to the indication function:
$$
\widetilde {h} _ {i} ^ {L} = z _ {i} ^ {L} + \frac {\sum_ {u} \mathbb {I} (a _ {u} <   = i \wedge b _ {u} > = i) h _ {u}}{\sum_ {u} \mathbb {I} (a _ {u} <   = i \wedge b _ {u} > = i)}
$$
From the global perspective, we first construct the global AMR graphs by fully connecting the root nodes of AMR graphs of different sentences, since the root nodes contain the core semantics according to the AMR core-semantic principle (Cai and Lam, 2019)  $^{1}$ . Then similar graph-based interaction methods are used to obtain the AMR-enhanced global representations  $\widetilde{h}_i^G$ , but based on global AMR graphs instead. In this way, the intersentential information can flow through the sentence boundaries, and therefore long-distance dependency can also be better captured.
4.3 Information Fusion Module
In the information fusion module, we fuse the global representations  $\widetilde{H}^G = \left[\widetilde{h}_1^G,\widetilde{h}_2^G,\ldots ,\widetilde{h}_{|\mathcal{D}|}^G\right]$  and local representations  $\widetilde{H}^{L} = \left[\widetilde{h}_{1}^{L},\widetilde{h}_{2}^{L},\ldots ,\widetilde{h}_{|\mathcal{D}|}^{L}\right]$ , to construct the final vector representations for the candidate spans.
In detail, we use a gated fusion to control how much information is incorporated from the two-stream representations. Given  $\widetilde{h}_i^G$  and  $\widetilde{h}_i^L$ , we calculate the gate vector  $g_{i}$  with trainable parameters  $W_{2}$  and  $W_{3}$ ,  $g_{i} = \mathrm{sigmoid}(W_{2}\widetilde{h}_{i}^{G} + W_{3}\widetilde{h}_{i}^{L} + b)$ . Then we derive the fused representations  $\widetilde{h}_i$ :
$$
\widetilde {h} _ {i} = g _ {i} \odot \widetilde {h} _ {i} ^ {G} + (1 - g _ {i}) \odot \widetilde {h} _ {i} ^ {L}
$$
For a candidate text span ranging from  $w_{i}$  to  $w_{j}$ , its fused representation consists of the start representation  $\widetilde{h}_{i}^{start}$ , the end representation  $\widetilde{h}_{j}^{end}$  and the average pooling of the hidden state of the span with  $W_{span} \in \mathbb{R}^{d_m \times (3 \times d_m)}$ :
$$
s _ {i: j} = W _ {s p a n} \left[ \widetilde {h} _ {i} ^ {s t a r t}; \widetilde {h} _ {i} ^ {e n d}; \frac {1}{j - i + 1} \sum_ {k = i} ^ {j} \widetilde {h} _ {k} \right]
$$
where  $\widetilde{h}_{i}^{start} = W_{s}\widetilde{h}_{i}$  and  $\widetilde{h}_{i}^{end} = W_{e}\widetilde{h}_{i}$ .
Since we extract arguments in span level, whose boundary may be ambiguous, we introduce an auxiliary boundary loss to enhance boundary information for the  $\widetilde{h}_i^{start}$  and  $\widetilde{h}_i^{end}$ . In detail, we predict whether the word  $w_i$  is the first or last word of a golden argument span with token-wise classifiers. We use a linear transformation followed by a sigmoid function, to derive the probability of the word  $w_i$  being the first or last word of a golden argument span, i.e.,  $P_i^s$  and  $P_i^e$ .
$$
P _ {i} ^ {s} = \mathrm {s i g m o i d} \left(W _ {4} \widetilde {h} _ {i} ^ {s t a r t}\right), P _ {i} ^ {e} = \mathrm {s i g m o i d} \left(W _ {5} \widetilde {h} _ {i} ^ {e n d}\right)
$$
Finally, the boundary loss is defined as the following cross-entropy losses of detecting the start and end position.
$$
\begin{array}{l} \mathcal {L} _ {b} = - \sum_ {i = 1} ^ {| \mathcal {D} |} \left[ y _ {i} ^ {s} \log P _ {i} ^ {s} + \left(1 - y _ {i} ^ {s}\right) \log \left(1 - P _ {i} ^ {s}\right) \right. \\ \left. + y _ {i} ^ {e} \log P _ {i} ^ {e} + \left(1 - y _ {i} ^ {e}\right) \log \left(1 - P _ {i} ^ {e}\right) \right] \tag {1} \\ \end{array}
$$
where,  $y_{i}^{s}$  and  $y_{i}^{e}$  denote the golden labels. In this way, we introduce an explicit supervision signal to inject boundary information of the start and end representation of an span, which is shown to be necessary and important to the extraction in our exploring experiments.
4.4 Classification Module
In the classification module, we predict what argument role the candidate span plays, or it does not belong to any specific argument roles. Besides the span representation  $s_{i:j}$ , we also consider the trigger, event type, and the length of the span. Specifically, we concatenate the following representations to obtain the final prediction vector  $I_{i:j}$ : 1) the trigger representation  $\widetilde{h}_t$ , and the span representation  $s_{i:j}$ , with their absolute difference  $\left|\widetilde{h}_t - s_{i:j}\right|$ , and element-wise multiplication,  $\widetilde{h}_t \odot s_{i:j}$ ; 2) the embedding of the event type  $E_{type}$ . 3) the embedding of the span length  $E_{len}$ ;
<sup>1</sup>We find more elaborate methods yield no further improvements, so we adopt this simple connection paradigm.
Dataset	Split	# Doc.	# Event	# Argument
RAMS	Train	3,194	7,329	17,026
Dev	399	924	2,188
Test	400	871	2,023
WikiEvents	Train	206	3,241	4,542
Dev	20	345	428
Test	20	365	566
$$
I _ {i: j} = \left[ \widetilde {h} _ {t}; s _ {i: j}; \left| \widetilde {h} _ {t} - s _ {i: j} \right|; \widetilde {h} _ {t} \odot s _ {i: j}; \mathrm {E} _ {t y p e}; \mathrm {E} _ {l e n} \right]
$$
We then use the cross entropy  $\mathcal{L}_c$  as loss function:
$$
\mathcal {L} _ {c} = - \sum_ {i = 1} ^ {| \mathcal {D} |} \sum_ {j = i} ^ {| \mathcal {D} |} y _ {i: j} \log P \left(r _ {i: j} = y _ {i: j}\right) \tag {2}
$$
where  $y_{i:j}$  is the golden argument role, and  $P(r_{i:j})$  is derived by a feed-forward network based on  $I_{i:j}$ .
Finally, we train the model in an end-to-end way with the final loss function  $\mathcal{L} = \mathcal{L}_c + \lambda \mathcal{L}_b$  with hyperparameter  $\lambda$ .
We evaluate our model on two public document-level event argument extraction datasets, RAMS v1.0 (Ebner et al., 2020) and WikiEvents (Li et al., 2021). RAMS contains 9,124 human-annotated examples, with 139 event types and 65 kinds of argument roles, and more than 21k arguments. WikiEvents is another human-annotated dataset, with 50 event types and 59 event argument roles, and more than 3.9k events. We follow the official train/dev/test split for RAMS and WikiEvents datasets, and use the evaluation script provided by Ebner et al. (2020) to evaluate the performance. The detailed data statistics of RAMS and WikiEvents datasets are shown in Table 1.
5.2 Experiment Setups and Metrics