Venue: NAACL 2022

# A Two-Stream AMR-enhanced Model for Document-level

Event Argument Extraction

Runxin Xu\({}^{1*}\), Peiyi Wang\({}^{1}\); Tianyu Liu\({}^{3}\), Shuang Zeng\({}^{1,2}\), Baobao Chang\({}^{1}\), Zhifang Sui\({}^{1}\)

\({}^{1}\)Key Laboratory of Computational Linguistics, Peking University, MOE, China

\({}^{2}\)School of Software and Microelectronics, Peking University, China

\({}^{3}\)Tencent Cloud Xiaowei {runxinxu,wangpeiyi9979}@gmail.com, rogertyliu@tecent.com

###### Abstract

Most previous studies aim at extracting events from a single sentence, while document-level event extraction still remains under-explored. In this paper, we focus on extracting event arguments from an entire document, which mainly faces two critical problems: a) the long-distance dependency between trigger and arguments over sentences; b) the distracting context towards an event in the document. To address these issues, we propose a **Two-Stream**A**bstract meaning **R**epresentation enhanced extraction model (**Tsar**). Tsar encodes the document from different perspectives by a two-stream encoding module, to utilize local and global information and lower the impact of distracting context. Besides, Tsar introduces an AMR-guided interaction module to capture both intra-sentential and inter-sentential features, based on the locally and globally constructed AMR semantic graphs. An auxiliary boundary loss is introduced to enhance the boundary information for text spans explicitly. Extensive experiments illustrate that Tsar outperforms previous state-of-the-art by a large margin, with \(2.54\) F1 and \(5.13\) F1 performance gain on the public RAMS and WikiEvents datasets respectively, showing the superiority in the cross-sentence arguments extraction. We release our code in [https://github.com/PKUnlp-icler/TSAR](https://github.com/PKUnlp-icler/TSAR).

+
Footnote †: Corresponding authors.

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5025 - 5036


As shown in Figure 1, the _origin_ argument _U.S._ can be identified more easily without Sentence 4, which does not offer useful information for the event, but contains many place entities that can be distracting, like _Saudi Arabia_ and _Russia or Iran_. It remains challenging to pinpoint the useful context while discarding the distracting information.

Recently, Du and Cardie (2020) use a tagging-based method, which fails to handle nested arguments. Instead, span-based methods predict argument roles for candidate spans Ebner et al. (2020); Zhang et al. (2020). Some studies directly generate arguments based on sequence-to-sequence model Li et al. (2021). However, how to model long-distance dependency among trigger and arguments, and how to handle distracting context explicitly, remain largely under-explored.

In this paper, to tackle the aforementioned two problems, we propose a Two-Stream AMR-enhanced extraction model (Tsar). In order to take advantage of the essential context in the document, and avoid being misled by distractions, we introduce a two-stream encoding module. It consists of a global encoder that encodes global semantics with as much context as possible to gather adequate context information, and a local encoder that focuses on the most essential information and prudently takes in extra context. In this way, Tsar can leverage complementary advantages of different encoding perspectives, and therefore make better use of the feasible context to benefit the extraction. Besides, to model the long-distance dependency, we introduce an AMR-guided interaction module. Abstract Meaning Representation AMR, Banarescu et al. (2013) graph contains rich hierarchical semantic relations among different concepts, which makes it favorable for complex event extraction. From such a linguistic-driven angle, we turn the linear structure of the document into both global and local graph structures, followed by a graph neural network to enhance the interactions, especially for those non-local elements. Finally, as Tsar extracts arguments in span level, where the span boundary may be ambiguous, we introduce an auxiliary boundary loss to enhance span representation with calibrated boundary.

To summarize, our contributions are three-fold. 1) We propose a two-stream encoding module for document-level EAE, which encodes the document through two different perspectives to better utilize the context. 2) We introduce an AMR-guided interaction module to facilitate the semantic interactions within the document, so that long-distance dependency can be better captured. 3) Our experiments show that Tsar outperforms the previous start-of-the-art model by large margins, with \(2.54\) F1 and \(5.13\) F1 improvements on public RAMS and WikiEvents datasets respectively, especially on cross-sentence event arguments extraction.

## 2 Related Work

### Sentence-level Event Extraction

Previous studies mainly focus on sentence-level event extraction. Li et al. (2014) and Judea and Strube (2016) use handcrafted features to extract events from the sentence. Chen et al. (2015) firstly propose a neural pipeline model to extract events, while Nguyen et al. (2016) utilize a joint model to mitigate error propagation. To better model the interactions among words, Liu et al. (2018); Yan et al. (2019); Ma et al. (2020) make use of the dependency tree, and Wadden et al. (2019) enumerates all possible spans and propagate information in the span graph. Data augmentation is also considered Yang et al. (2019). Moreover, some works try to reformulate the event extraction task as other tasks. For example, Du and Cardie (2020) and Zhou et al. (2021) cast event extraction as question answering, and Xiangyu et al. (2021) model it as a sequence-to-sequence task. However, all of these models can only extract events from a single sentence. Thus, they fail to handle the much more common cases, where event arguments usually spread over multiple sentences within the document.

### Document-level Event Extraction

In order to extract events from a whole piece of article with multiple sentences, document-level event extraction has attracted more and more attention recently. Yang and Mitchell (2016) utilize well-defined features to extract arguments across sentences, while most recent methods are based on neural networks. Some studies first identify entities in the document, followed by assigning these entities as specific argument roles Yang et al. (2018); Zheng et al. (2019); Xu et al. (2021). Differently, some studies try to jointly extract entities and argument roles simultaneously, which can be further divided into tagging-based and span-based methods. Tagging-based methods directly conduct sequence labeling for each token in the document with BIO-schema Du and Cardie (2020); Veyseh 

et al., 2021), while span-based methods predict the argument role for candidate text spans which usually have a maximum length limitation Ebner et al. (2020); Zhang et al. (2020). Another line of studies reformulate the task as a sequence-to-sequence task Du et al. (2021a, b); Li et al. (2021), or machine reading comprehension task Wei et al. (2021).

As a span-based method, Tsar is different from prior methods that simply encode it as a long sentence. Instead, Tsar introduces a two-stream encoding module and AMR-guided interactions module to model intra-sentential and inter-sentential semantics, along with an auxiliary boundary loss to enhance span boundary information.

## 3 Task Formulation

Following Ebner et al. (2020), we formulate doclevel event argument extraction as follows. We define that a document \({\cal D}\) consists of \(N\) sentences, and a sentence is comprised of a sequence of words, i.e., \({\cal D}=\left\{w_{1},w_{2},\ldots,w_{|{\cal D}|}\right\}\), and \(\operatorname*{SEN}\left(w_{i}\right)\in[1,N]\) refers to the sentence that \(w_{i}\) belongs to. We also define the event types set \({\cal E}\) and the corresponding argument roles set \({\cal R}_{e}\) for each event type \(e\in{\cal E}\). Then, given a document \({\cal D}\) and the trigger \(t\in{\cal D}\) triggering the event type \(e\in{\cal E}\), the task aims to detect all \((r,s)\) pairs for the event, where \(r\in{\cal R}_{e}\) is an argument role for the event type \(e\), and \(s\subseteq{\cal D}\) is a contiguous text span in the document.

## 4 Methodology

Figure 2 shows the overall architecture of our model Tsar. The document is fed into the two-stream encoding module, followed by the AMR-guided interaction module to derive both global and local contextualized representations. The information fusion module fuses these two-stream representations, and the classification module finally predicts argument roles for candidate spans.

### Two-Stream Encoding Module

Although more context is provided by the document, it also inevitably introduces irrelevant and distracting information towards the event. These noise signals can be harmful to the argument extraction as shown in Figure 1. To capture useful information and filter distracting one, we propose a two-stream encoding module, consisting of a global

Figure 2: **Overview of our Tsr. Firstly, taking an entire document as input, Tsar first encodes the document by the two-stream encoding module, where the global and local encoders with different attention reception fields are used to capture the context in different scopes. Then the AMR-guided interaction module constructs global AMR graphs and local ones to stimulate the interactions among concepts in the document, especially those far away from each other, based on graph neural network. Next, the information fusion module fuses the two-stream representations, and also strengthens the boundary information through a boundary loss. Finally, the classification module makes predictions for candidate spans. For conciseness, we assume the document has three sentences, \(S_{1}\), \(S_{2}\), \(S_{3}\), and the event is triggered by \(y_{2}\) with \([z_{2},z_{3}]\) being a candidate argument span.**



encoder that is aware of all context, and a local encoder that only prudently focuses on the most essential information. Therefore, we can leverage their complementary advantages to make better use of the context information.

Specifically, the global and local encoders share the same Transformer-based pre-trained language model such as BERT. By controlling the reception field of the words in the self-attention module , we can encode the document from different perspectives. In the global encoder, the attention technique is the same as the traditional Transformer:

\[{\rm Attention}^{G}\left(Q,K,V\right)={\rm softmax}\left(\frac{ QK^{\top}}{\sqrt{d_{m}}}\right)V\]

where \(Q\), \(K\), \(V\) refers to query, key, and value matrix, and \(d_{m}\) is the model dimension. However, in the local encoder, we introduce a mask matrix \(M\), such that tokens can only attend to the sentence itself and the sentence where the trigger locates, to avoid redundant distracting information:

\[{\rm Attention}^{L}\left(Q,K,V\right)={\rm softmax}\left(\frac{ QR^{\top}+M}{\sqrt{d_{m}}}\right)V\]

\[M_{ij}=\left\{\begin{array}{ll}0,&{\rm SEN}\left(w_{j}\right)\in\{{\rm SEN} \left(w_{i}\right),{\rm SEN}\left(t\right)\}\\ -\infty,&Otherwise\end{array}\right.\]

where \({\rm SEN}\left(w_{i}\right)\) is the sentence that the word \(w_{i}\) belongs to, and \(t\) refers to the trigger of the event.

Hence, we encode the document with two different streams, a global encoder\({\rm Encoder}^{G}\) and a local encoder\({\rm Encoder}^{L}\), finally deriving two representations, \(Z^{G}\) and \(Z^{L}\):

\[Z^{G} =\left[z_{1}^{G},z_{2}^{G},\ldots,z_{|\mathcal{D}|}^{G}\right]={ \rm Encoder}^{G}\left(\left[w_{1},w_{2},\ldots,w_{|\mathcal{D}|}\right]\right)\] \[Z^{L} =\left[z_{1}^{L},z_{2}^{L},\ldots,z_{|\mathcal{D}|}^{L}\right]={ \rm Encoder}^{L}\left(\left[w_{1},w_{2},\ldots,w_{|\mathcal{D}|}\right]\right)\]

### AMR-Guided Interaction Module

One key challenge to extract arguments from the document is to capture the intra-sentential and inter-sentential features. Therefore, we propose an AMR-guided interaction module that adopts Abstract Meaning Representation (AMR, Banarescu et al., 2013) graph to provide rich semantic structure to facilitate the interactions among concepts, which also offers logical meanings of the document from a linguistic-driven perspective to benefit the language understanding.

AMR semantic graph models the meaning representations of a sentence as a rooted, directed, labeled graph. Concretely, with an AMR parser, a natural sentence can be parsed into an AMR graph \(G=(V,E)\). The node \(v=(a,b)\in V\) represents a concept that corresponds to the span ranging from \(w_{a}\) to \(w_{b}\) in the origin sentence, while the edge represents a specific AMR relation (detail in Appendix A). Thus, AMR focuses on semantic relations rather than syntactic ones, which is more high-level and beneficial to event understanding, and the structures are more close to the event trigger-arguments structures. For example, Figure 3 demonstrates how a sentence is parsed into an AMR semantic graph. As event arguments play essential roles in the text, most of them would be involved, if not all, in the AMR graphs (\(90\%\) and \(88\%\) arguments in RAMS and WikiEvents datasets). We use the state-of-the-art AMR parser Fernandez Astudillo et al. (2020), which achieves satisfactory results (up to \(81.3\) Smatch on AMR2.0 data) for downstream application. As the number of AMR relation types is large, which results in too many demanded parameters, we also follow Zhang and Ji (2021) to cluster the relation types into main categories. More details can be found in Appendix A.

The AMR-guided interaction module is attached after the global and local encoders as shown in Figure 2. We use the AMR graphs as skeletons for information interactions, under a _composition, interaction, and decomposition_ paradigm.

From the local perspective, we construct AMR graphs for each sentence in the document, and they are isolated from each other. For initialization, the vector representation of node \(u=(a_{u},b_{u})\) is _composed_ by averaging the local representations of its corresponding text span:

\[h_{u}^{0}=\frac{1}{|b_{u}-a_{u}+1|}\sum_{i=a_{u}}^{b_{u}}z_{i}^{ L}\]

Similar to Zeng et al. (2020), we then use \(L\)-layer stacked Graph Convolution Network (Kipf and

Figure 3: The AMR graph provides abstract and logical semantic information, where the nodes denote the concepts and the edges refer to different relation types. The corresponding text spans for nodes are omitted.

 

Welling, 2017) to model the _interactions_ among different concept nodes through edges with different relation types. Given node \(u\) at the \(l\)-th layer, the information interaction and aggregation operation is defined as follows:

\[h^{(l+1)}_{u}=\mathrm{ReLU}\left(\sum_{k\in K}\sum_{v\in\mathcal{K}_{k}(u)\bigcup \{u\}}\frac{1}{c_{u,k}}W^{(l)}_{k}h^{(l)}_{v}\right)\]

where \(\mathcal{K}\) denotes different relation types, \(\mathcal{N}_{k}(u)\) denotes the neighbors for \(u\) connected with \(k\)-th relation types and \(c_{u,k}\) is a normalization constant. Besides, \(W^{(l)}_{k}\in\mathbb{R}^{d_{m}\times d_{m}}\) is a trainable parameter.

Finally, we concatenate vectors in all layers and derive the final node representation by \(h_{u}=W_{1}[h^{0}_{u};h^{1}_{u};\ldots;h^{L}_{u}]\in\mathbb{R}^{d_{m}}\). Then \(h_{u}\) is _decomposed_ into the local representations of corresponding words, followed by token-wise aggregation, where \(\mathbb{I}(\cdot)\) refers to the indication function:

\[\widetilde{h}^{L}_{i}=z^{L}_{i}+\frac{\sum_{u}\mathbb{I}(a_{u}<=i\wedge b_{u}> =i)h_{u}}{\sum_{u}\mathbb{I}(a_{u}<=i\wedge b_{u}>=i)}\]

From the global perspective, we first construct the global AMR graphs by fully connecting the root nodes of AMR graphs of different sentences, since the root nodes contain the core semantics according to the AMR core-semantic principle (Cai and Lam, 2019)1. Then similar graph-based interaction methods are used to obtain the AMR-enhanced global representations \(\widetilde{h}^{G}_{i}\), but based on global AMR graphs instead. In this way, the inter-sentential information can flow through the sentence boundaries, and therefore long-distance dependency can also be better captured.

Footnote 1: We find more elaborate methods yield no further improvements, so we adopt this simple connection paradigm.

### Information Fusion Module

In the information fusion module, we fuse the global representations \(\widetilde{H}^{G}=\left[\widetilde{h}^{G}_{1},\widetilde{h}^{G}_{2},\dots, \widetilde{h}^{G}_{|\mathcal{D}|}\right]\) and local representations \(\widetilde{H}^{L}=\left[\widetilde{h}^{L}_{1},\widetilde{h}^{L}_{2},\dots, \widetilde{h}^{L}_{|\mathcal{D}|}\right]\), to construct the final vector representations for the candidate spans.

In detail, we use a gated fusion to control how much information is incorporated from the two-stream representations. Given \(\widetilde{h}^{G}_{i}\) and \(\widetilde{h}^{L}_{i}\), we calculate the gate vector \(g_{i}\) with trainable parameters \(W_{2}\) and \(W_{3}\), \(g_{i}=\operatorname*{sigmoid}(W_{2}\widetilde{h}^{G}_{i}+W_{3}\widetilde{ h}^{L}_{i}+b)\). Then we derive the fused representations \(\widetilde{h}_{i}\):

\[\widetilde{h}_{i}=g_{i}\odot\widetilde{h}^{G}_{i}+(1-g_{i})\odot\widetilde{ h}^{L}_{i}\]

For a candidate text span ranging from \(w_{i}\) to \(w_{j}\), its fused representation consists of the start representation \(\widetilde{h}^{start}_{i}\), the end representation \(\widetilde{h}^{end}_{j}\) and the average pooling of the hidden state of the span with \(W_{span}\in\mathbb{R}^{d_{m}\times(3\times d_{m})}\):

\[s_{i:j}=W_{span}\left[\widetilde{h}^{start}_{i};\widetilde{h}^{end}_{i};\frac{ 1}{j-i+1}\sum_{k=i}^{j}\widetilde{h}_{k}\right]\]

where \(\widetilde{h}^{start}_{i}=W_{s}\widetilde{h}_{i}\) and \(\widetilde{h}^{end}_{i}=W_{e}\widetilde{h}_{i}\).

Since we extract arguments in span level, whose boundary may be ambiguous, we introduce an auxiliary boundary loss to enhance boundary information for the \(\widetilde{h}^{start}_{i}\) and \(\widetilde{h}^{end}_{i}\). In detail, we predict whether the word \(w_{i}\) is the first or last word of a golden argument span with token-wise classifiers. We use a linear transformation followed by a sigmoid function, to derive the probability of the word \(w_{i}\) being the first or last word of a golden argument span, i.e., \(P^{s}_{i}\) and \(P^{e}_{i}\).

\[P^{s}_{i}=\operatorname*{sigmoid}\left(W_{4}\widetilde{h}^{start}_{i}\right) ,P^{e}_{i}=\operatorname*{sigmoid}\left(W_{5}\widetilde{h}^{end}_{i}\right)\]

Finally, the boundary loss is defined as the following cross-entropy losses of detecting the start and end position.

\[\begin{split}\mathcal{L}_{b}&=-\sum_{i=1}^{| \mathcal{D}|}[y^{s}_{i}\text{log}P^{s}_{i}+(1-y^{s}_{i})\log{(1-P^{s}_{i})}\\ &\qquad+y^{e}_{i}\text{log}P^{e}_{i}+(1-y^{e}_{i})\log{(1-P^{e}_{ i})}]\end{split} \tag{1}\]

where, \(y^{s}_{i}\) and \(y^{e}_{i}\) denote the golden labels. In this way, we introduce an explicit supervision signal to inject boundary information of the start and end representation of an span, which is shown to be necessary and important to the extraction in our exploring experiments.

### Classification Module

In the classification module, we predict what argument role the candidate span plays, or it does not belong to any specific argument roles. Besides the span representation \(s_{i:j}\), we also consider the trigger, event type, and the length of the span. Specifically, we concatenate the following representations to obtain the final prediction vector \(I_{i:j}\): 1) the trigger representation \(\widetilde{h}_{t}\), and the span representation \(s_{i:j}\), with their absolute difference \(\left|\widetilde{h}_{t}-s_{i:j}\right|\), and element-wise multiplication, \(\widetilde{h}_{t}\circ s_{i:j}\); 2) the embedding of the event type E\({}_{type}\). 3) the embedding of the span length E\({}_{len}\) 

\[I_{i:j}=\left[\widetilde{h}_{t};s_{i:j};\left|\widetilde{h}_{t}-s_{i :j}\right|;\widetilde{h}_{t}\odot s_{i:j};\mathrm{E}_{type};\mathrm{E}_{len}\right]\]

We then use the cross entropy \(\mathcal{L}_{c}\) as loss function:

\[\mathcal{L}_{c}=-\sum_{i=1}^{\left|\mathcal{D}\right|}\sum_{j=i}^{\left| \mathcal{D}\right|}y_{i:j}\mathrm{log}P\left(r_{i:j}=y_{i:j}\right) \tag{2}\]

where \(y_{i:j}\) is the golden argument role, and \(P\left(r_{i:j}\right)\) is derived by a feed-forward network based on \(I_{i:j}\).

Finally, we train the model in an end-to-end way with the final loss function \(\mathcal{L}=\mathcal{L}_{c}+\lambda\mathcal{L}_{b}\) with hyperparameter \(\lambda\).

## 5 Experiments

### Datasets

We evaluate our model on two public document-level event argument extraction datasets, RAMS v\(1.0\)(Ebner et al., 2020) and WikiEvents Li et al. (2021). RAMS contains \(9,124\) human-annotated examples, with \(139\) event types and \(65\) kinds of argument roles, and more than \(21\)k arguments. WikiEvents is another human-annotated dataset, with \(50\) event types and \(59\) event argument roles, and more than \(3.9\)k events. We follow the official train/dev/test split for RAMS and WikiCvents datasets, and use the evaluation script provided by Ebner et al. (2020) to evaluate the performance. The detailed data statistics of RAMS and WikiEvents datasets are shown in Table 1.

### Experiment Setups and Metrics

In our implementation, we use BERT\({}_{\rm base}\)Devlin et al. (2019) and RoBERTa\({}_{\rm large}\)Liu et al. (2019) as our backbone encoder for Tsar, with global and local encoders sharing parameters. Detailed hyperparameters are listed in Appendix B.

Following Zhang et al. (2020), we report the Span F1 and Head F1 for RAMS dataset. Span F1 requires the predicted argument spans to fully match the golden ones, while Head F1 relaxes the constraint and evaluates solely on the head word of the argument span. The head word of a span is defined as the word that has the smallest arc distance to the root in the dependency tree. In addition, following Li et al. (2021), we report the Head F1 and CoreF1 scores for WikiEvents dataset. The model is given full credit in Coref F1 if the extracted argument is coreferential with the golden argument as used by Ji and Grishman (2008).

### Main Results

We compare Tsar with the following baselines. 1) **BERT-CRF**Shi and Lin (2019) is a tagging-based method, which adopts a BERT-based BIO-styled sequence labeling model. 2) **Two-Step**Zhang et al. (2020) is a span-based method, which first identifies the head word of possible argument span, and then extends to the full span. **BERT-CRF\({}_{\rm TCD}\)** and **Two-Step\({}_{\rm TCD}\)** refers to adopting Type-Constraint Decoding mechanism as used in Ebner et al. (2020). 3) **FEAE**Wei et al. (2021), Frame-aware Event Argument Extraction, is a concurrent work based on question answering. 4) **BERT-QA**Du and Cardie (2020) is also a QA-based model. BERT-QA and BERT-QA-Doc extract run on sentence-level and document-level, respectively. 5) **BART-Gen**Li et al. (2021) formulate the task as a sequence-to-sequence task and uses BART\({}_{\rm large}\)Lewis et al. (2020) to generate corresponding arguments in a predefined format.

Table 2 illustrates the results in both dev and test set on RAMS dataset. As is shown, among models based on BERT\({}_{\rm base}\), Tsar outperforms other previous methods. For example, Tsar yields an improvement of \(4.93\sim 7.13\) Span F1 and

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Dataset** & **Split** & **\# Doc.** & **\# Event** & **\# Argument** \\ \hline  & Train & 3,194 & 7,329 & 17,026 \\ RAMS & Dev & 399 & 924 & 2,188 \\  & Test & 400 & 871 & 2,023 \\ \hline  & Train & 206 & 3,241 & 4,542 \\ WikiEvents & Dev & 20 & 345 & 428 \\  & Test & 20 & 365 & 566 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of RAMS and WikiEvents datasets.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Dev**} & \multicolumn{2}{c}{**Test**} \\ \cline{2-5}  & Span F1 & Head F1 & Span F1 & Head F1 \\ \hline BERT-CRF & 38.1 & 45.7 & 39.3 & 47.1 \\ BERF-CRF\({}_{\rm TCD}\) & 39.2 & 46.7 & 40.5 & 48.0 \\ Two-Step & 38.9 & 46.4 & 40.1 & 47.7 \\ Two-Step\({}_{\rm TCD}\) & 40.3 & 48.0 & 41.8 & 49.7 \\ FEAE & - & - & 47.40 & - \\ TSA\({}_{\rm base}\) (Ours) & **45.23** & **51.70** & **48.06** & **55.04** \\ \hline \hline BART-Gen & - & - & 48.64 & 57.32 \\ TSA\({}_{\rm large}\) (Ours) & **49.23** & **56.76** & **51.18** & **58.53** \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Comparison between Tsar and other methods on RAMS dataset. Models above the double line are based on BERT\({}_{\rm base}\). Tsar consistently outperforms others on Span F1 and Head F1. Compared with BART-Gen, Tsar improves \(2.54\) Span F1 in the test set.

 

\(3.70\sim 6.00\) Head F1 compared with the previous method in the dev set, and up to \(8.76\) Span F1 in the test set. Besides, among models based on large pre-trained language models, Tsar outperforms BART-Gen by \(2.54\) Span F1 and \(1.21\) Head F12. These results suggest that encoding the document in a two-stream way, and introducing AMR graphs to facilitate interactions, is beneficial to capturing intra-sentential and inter-sentential features, and thus improves the performance.

Footnote 2: We use \({\tt TSA}_{\rm large}\) based on RoBERTa\({}_{\rm large}\) to compare with BART-Gen based on BARTlarge, as they are pre-trained on the same corpus with the same batch size and training steps.

Moreover, we follow Li et al. (2021) to evaluate both argument identification and argument classification, and report the Head F1 and Coref F1. Identification requires the model to correctly detect the argument span boundary, while classification has to further correctly predict its argument role. As illustrated in Table 3, Tsar consistently outperforms others in both tasks. Compared with BART-Gen, Tsar improves up to \(4.87\)/\(3.23\) Head/Coref F1 for argument identification, and \(5.13\)/\(3.68\) Head/Coref F1 for argument classification. Similar results also appear among models based on BERTbase, with \(5.69\sim 36.37\) and \(11.95\sim 33.34\) Head F1 improvement for identification and classification. These results show that Tsar is superior to other methods in not only detecting the boundary of argument spans, but also predicting their roles.

## 6 Analysis

### Cross-sentence Argument Extraction

Since there are multiple sentences in the document, some event arguments are located far away from the trigger, which highly increases the difficulty of extraction. To explore the effect of handling such cross-sentence arguments of our Tsar, we divide the event arguments in RAMS dataset into five bins according to the sentence distance between arguments and trigger, i.e., \(d=\{-2,-1,0,1,2\}\). We report the Span F1 on the RAMS dev set for different methods. As shown in Table 4, the Span F1 for cross-sentence arguments (\(d\neq 0\)) is much lower than local arguments (\(d=0\)), suggesting the huge challenge to capture long-distance dependency between triggers and cross-sentence arguments. However, Tsar still surpasses other strong baselines. In detail, \({\tt Tsar}_{\rm base}\) improves \(0.4\) and \({\tt Tsar}_{\rm large}\) improves \(0.7\) F1 compared with the previous state-of-the-art, respectively. More importantly, when extracting cross-sentence arguments, \({\tt Tsar}_{\rm base}\) and \({\tt Tsar}_{\rm large}\) yield an improvement of up to \(2.3\) and \(2.7\) on average. The results support our claims that Tsar is good at capturing both intra-sentential and inter-sentential features, especially the long-distance between trigger and arguments.

### Ablation Study

We conduct an ablation study to explore the effectiveness of different modules in Tsar. Table 5 show the results on RAMS datasets for \({\tt Tsar}_{\rm large}\). We also provide results for \({\tt Tsar}_{\rm base}\), and those on WikiEvents datasets in Appendix C.

Firstly, we remove the global or local encoder in

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Method** & \multicolumn{2}{c}{**Arg Identification**} & \multicolumn{2}{c}{**Arg Classification**} \\ \cline{2-5}  & Head F1 & Coref F1 & Head F1 & Coref F1 \\ \hline BERT-CRF & 69.83 & 72.24 & 54.48 & 56.72 \\ BERT-QA & 61.05 & 64.59 & 56.16 & 59.36 \\ BERT-QA-Doc & 39.15 & 51.25 & 34.77 & 45.96 \\ \({\tt Tsar}_{\rm none}\) (Ours) & **75.52** & **73.17** & **68.11** & **66.31** \\ \hline \hline BART-Gen & 71.75 & 72.29 & 64.57 & 65.11 \\ \({\tt Tsar}_{\rm large}\) (Ours) & **76.62** & **75.52** & **69.70** & **68.79** \\ \hline \end{tabular}
\end{table}
Table 3: **Comparison between Tsar and other methods on WikiEvents dataset. Models above the double line are based on BERTbase. Tsar yields evident improvements in argument identification and classification sub-tasks. Compared with BART-Gen, Tsar improves Head F1 in argument classification by \(5.13\) score.**

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Dev**} & \multicolumn{2}{c}{**Test**} \\ \cline{2-5}  & Span F1 & Head F1 & Span F1 & Head F1 \\ \hline TsARlarge & 49.23 & 56.76 & 51.18 & 58.53 \\ - Global Encoder & 46.71 & 54.26 & 48.21 & 55.49 \\ - Local Encoder & 48.43 & 55.44 & 48.69 & 56.82 \\ - AMR-guided Graph & 48.63 & 55.24 & 49.21 & 56.70 \\ - Boundary Loss & 47.93 & 55.14 & 50.47 & 57.75 \\ \hline \hline \end{tabular}
\end{table}
Table 5: **Ablation study on RAMS for \({\tt Tsar}_{\rm large}\). The score would decrease without any kind of module.** 

the two-stream encoding module. As shown in Table 5, the removal causes drop in performance, e.g., \(3.04\) and \(1.71\) Head F1 drop on the test set without global and local encoder. It suggests the global and local encoders are complementary to each other, and both of them are necessary for Tsar.

Secondly, once we remove the AMR-guided interaction module, the Head F1 would decrease by \(1.83\) on the test set. It shows the semantic structure provided by AMR graphs is helpful to the arguments extraction of the document.

Finally, the removal of boundary loss causes the boundary information lost in span representations, which also leads to \(1.62\) and \(0.78\) Head F1 decrease on dev and test set.

### Case Study

In this section, we show a specific case of the extraction results among different methods. As shown in Figure 5, _stabbings_ triggers an _Attack_ event with three arguments in color. Since _Nine people_ is located near the trigger, all the methods correctly predict it as the _target_. However, extracting _Minnesota_ and _Dahir Adan_ asks for capturing long-distance dependency. Although Two-Step and BART-Gen wrongly predict the _place_ as _Iraq and Syria_, and Two-Step even fails to extract the _Attacker_, Tsar manage to extract the cross-sentence arguments. It can be attributed to that our AMR-enhanced module catches _Minnesota_ is the _place_ of _attack_ that is highly related to the trigger _stabbings_ in semantics.

### Error Analysis

To further explore the errors made by different models and analyze the reasons in detail, we randomly choose \(200\) examples from the RAMS test set and compare the predictions with golden annotations manually. We divide the errors into five categories, which is shown in Figure 4. _Wrong Span_ refers to assigning a specific role to a wrong span non-overlapped with the golden one. We find it is usually due to the negative words like _not_, and the coreference spans for the golden one. _Over-extract_ denotes the model predicts an argument role while it does not exist in the document. Some extracted spans are the sub-strings of the golden spans (_Partial_), or have some overlaps with them (_Overlap_). These two kinds of errors are usually attributed to

Figure 4: **Error analysis on RAMS dataset. The triggers are in bold with corresponding event types in green. The underlined spans refer to golden arguments, with their roles in blue. The blacked] spans denote the predicted arguments, with their roles noted in red. We illustrate the number of different kinds of errors for Two-step and our Tsar, which has \(275\) and \(233\) errors in total, respectively. Compared with Two-step, Tsar decreases errors in most error categories, especially for _Wrong Role_ and _Over-extract_.**

Figure 5: **An extraction case, where an Attack event is triggered by stabbings with three arguments. Tsar manages to extract the cross-sentence argument Minnesota data from the trigger, while other methods fail.**



the annotation inconsistency in the dataset, such as whether the adjective, quantifier, and article (e.g., \(a\) and _the_) before the noun should belong to the golden argument. Besides, the _Partial_ error also usually occurs in cases where there is punctuation like a comma in the golden span as shown in Figure 4. Finally, though the model succeeds to identify the golden span, it can still assign wrong argument role to the span (_Wrong Role_). We compare the errors of Two-step\({}_{\text{TCD}}\) and Tsar\({}_{\text{base}}\). We observe Tsar decrease the number of errors from \(275\) to \(233\), especially for _Wrong Role_ and _Over-extract_, with \(27\) and \(16\) errors reduction, respectively.

## 7 Conclusion

It is challenging to extract event arguments from a whole document, owing to the long-distance dependency between trigger and arguments over sentences and the distracting context. To tackle these problems, we propose **Two**-**Stream **AMR**-enhanced extraction model (**Tsar**). Tsar uses two-stream encoders to encode the document from different perspectives, followed by an AMR-guided interaction module to facilitate the document-level semantic interactions. An auxiliary boundary loss is introduced to enhance the boundary information for spans. Experiments on RAMS and WikiEvents datasets demonstrate that Tsar outperform previous state-of-the-art methods by a large margin, with \(2.51\) and \(5.13\) F1 improvements respectively, especially for cross-sentence argument extraction.

## Acknowledgements

This paper is supported by the National Key R&D Program of China under Grand No.2018AAA0102003, the National Science Foundation of China under Grant No.61936012 and 61876004.

## References

* L. Banarescu, C. Bonial, S. Cai, M. Georgescu, K. Griffitt, U. Hermjakob, K. Knight, P. Koehn, M. Palmer, and N. Schneider (2013)Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, Cited by: SS1.
* M. Bevilacqua, R. Blloshmi, and R. Navigli (2021)One spring to rule them both: symmetric amr semantic parsing and generation without a complex pipeline. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, Cited by: SS1.
* D. Cai and W. Lam (2019)Core semantic first: A top-down approach for AMR parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1.
* L. Chen, P. Wang, Runxin Xu, T. Liu, Z. Sui, and Baobao Chang (2022)ATP: AMRize Then Parse! Enhancing AMR Parsing with PseudoAMRs. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: SS1.
* Y. Chen, L. Xu, K. Liu, D. Zeng, and J. Zhao (2015)Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: SS1.
* X. Du and C. Cardiie (2020)Document-level event role filler extraction using multi-granularity contextualized encoding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: SS1.
* X. Du and C. Cardiie (2020)Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1.
* X. Du and C. Cardiie (2020)Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1.
* X. Du, A. Rush, and C. Cardiie (2021)GRIT: generative role-filler transformers for document-level event entity extraction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Cited by: SS1.
* X. Du, A. Rush, and C. Cardiie (2021)Template filling with generative transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: SS1.
* S. Ebner, P. Xia, R. Culkin, K. Rawlins, and B. Van Durme (2020)Multi-sentence argument linking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: SS1.
* R. Fernandez Astudillo, M. Ballesteros, T. Naseem, A. Blodgett, and R. Florian (2020)Transition-based parsing with stack-transformers. In Findings of the Association for Computational Linguistics (EMNLP), 

Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: SS1, SS3.
* A. Judea and M. Strube (2016)Incremental global event extraction. In Proceedings of the 26th International Conference on Computational Linguistics (COLING), Cited by: SS1, SS3.
* Diederik P. Kingma and J. Ba (2015)Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), Cited by: SS1, SS3.
* T. N. Kipf and M. Welling (2017)Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations (ICLR), Cited by: SS1, SS3.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: SS1, SS3.
* M. Li, A. Zareian, Y. Lin, X. Pan, S. Whitehead, B. Chen, B. Wu, H. Ji, S. Chang, C. Voss, D. Naipierski, and M. Freedman (2020)GAIA: a fine-grained multimedia knowledge extraction system. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL), Cited by: SS1, SS3.
* Q. Li, H. Ji, Y. Hong, and S. Li (2014)Constructing information networks using one single model. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1, SS3.
* S. Li, H. Ji, and J. Han (2021)Document-level event argument extraction by conditional generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: SS1, SS3.
* X. Liu, Z. Luo, and H. Huang (2018)Jointly multiple events extraction via attention-based graph information aggregation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1, SS3.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)RoBERTA: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS1, SS3.
* J. Ma, S. Wang, R. Anubhai, M. Ballesteros, and Y. Al-Onaizan (2020)Resource-enhanced neural model for event argument extraction. In Findings of the Association for Computational Linguistics (EMNLP), Cited by: SS1, SS3.
* T. Huu Nguyen, K. Cho, and R. Girshick (2016)Joint event extraction via recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: SS1, SS3.
* P. Shi and J. Lin (2019)Simple BERT models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255. Cited by: SS1, SS3.
* 25th Pacific-Asia Conference (PAKDD), Cited by: SS1, SS3.
* D. Wadden, U. Wennberg, Y. Luan, and H. Hajishirzi (2019)Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1, SS3.
* M. Wang, L. Yu, D. Zheng, Q. Gan, Y. Gai, Z. Ye, M. Li, J. Zhou, Q. Huang, C. Ma, Z. Huang, Q. Guo, H. Zhang, H. Lin, J. Zhao, J. Li, A. J. Smola, and Z. Zhang (2019)Deep graph library: towards efficient and scalable deep learning on graphs. arXiv preprint arXiv:1909.01315. Cited by: SS1, SS3.
* P. Wang, L. Chen, T. Liu, D. Dai, Y. Cao, B. Chang, and Z. Sui (2022)Hierarchical curriculum learning for amr parsing. External Links: 2203.03086 Cited by: SS1, SS3.
* K. Wei, X. Sun, Z. Zhang, J. Zhang, G. Zhi, and L. Jin (2021)Trigger is not sufficient: exploiting frame-aware knowledge for implicit event argument extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: SS1, SS3.
* T. Wolf, L. Debutt, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Fontowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jerneitre, J. Pu, C. Xu, T. Le Scao, S. Gugger, M. Dramee, Quentin Lhoest, and Alexander Rush (2020)Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP), Cited by: SS1, SS3.
* Xi. Xiangyu, W. Ye, S. Zhang, Q. Wang, H. Jiang, and W. Wu (2021)capturing event argument interaction via a bi-directional entity-level recurrent decoder. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: SS1, SS3.
* R. Xu, T. Liu, L. Li, and B. Chang (2021)Document-level event extraction via heterogeneous graph-based interaction model with a tracker..

 

In _The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Yan et al. (2019) Haoran Yan, Xiaolong Jin, Xiangbin Meng, Jiafeng Guo, and Xueqi Cheng. 2019. Event detection with multi-order graph convolution and aggregated attention. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Yang and Mitchell (2016) Bishan Yang and Tom M. Mitchell. 2016. Joint extraction of events and entities within a document context. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
* Yang et al. (2018) Hang Yang, Yubo Chen, Kang Liu, Yang Xiao, and Jun Zhao. 2018. DCFEE: A document-level Chinese financial event extraction system based on automatically labeled training data. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Yang et al. (2019) Sen Yang, Dawei Feng, Linbo Qiao, Zhigang Kan, and Dongsheng Li. 2019. Exploring pre-trained language models for event extraction and generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Zeng et al. (2020) Shuang Zeng, Runxin Xu, Baobao Chang, and Lei Li. 2020. Double graph based reasoning for document-level relation extraction. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* 18th International Conference on Artificial Intelligence in Medicine (AIME)_.
* Zhang et al. (2020b) Zhisong Zhang, Xiang Kong, Zhengzhong Liu, Xuezhe Ma, and Eduard Hovy. 2020b. A two-step approach for implicit event argument detection. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Zhang and Ji (2021) Zixuan Zhang and Heng Ji. 2021. Abstract Meaning Representation guided graph encoding and decoding for joint information extraction. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
* Zheng et al. (2019) Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. 2019. Doc2EDAG: An end-to-end document-level framework for Chinese financial event extraction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Zhou et al. (2021) Yang Zhou, Yuko Chen, Jun Zhao, Yin Wu, Jiexin Xu, and JinLong Li. 2021. What the role is vs. what plays the role: Semi-supervised event argument extraction via dual question answering. In _Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI)_ 

Abstract Meaning Representation (AMR) Graph

There are many AMR parsing approaches Bevilacqua et al. (2021); Fernandez Astudillo et al. (2020); Wang et al. (2022); Chen et al. (2022). To obtain AMR semantic graphs with the align information between text spans and AMR nodes, we use the transition-based AMR parser proposed by Fernandez Astudillo et al. (2020), which is a state-of-the-art AMR parser and can achieve satisfactory results for downstream application (up to \(81.3\) Smatch on AMR2.0 data). As the number of AMR relation types is large, which results in too many demanded parameters, we follow Zhang and Ji (2021) to cluster the relation types into main categories as shown in Table 6.

## Appendix B Hyperparameters Setting

We set the dropout rate to \(0.1\), batch size to \(8\), and train Tsar using Adam Kingma and Ba (2015) as optimizer with \(3\text{e}{-5}\) learning rate. We train Tsar for \(50\) epochs for RAMS dataset and \(100\) epochs for WikiEvents dataset. We search the boundary loss weight \(\lambda\) from \(\{0.05,0.1,0.2\}\), and \(L\) from \(\{3,4\}\), and select the best model using dev set. Our code is based on Transformers Wolf et al. (2020) and DGL libraries Wang et al. (2019).

## Appendix C Ablation Study

In the main body of the paper, we illustrate the results of the ablation study for Tsar\({}_{\rm large}\) on RAMS dataset. To thoroughly show the effect of different modules of Tsar, we also provide the results of the ablation study for Tsar\({}_{\rm base}\) on RAMS dataset. Table 7 shows the results on RAMS dataset, from which we can observe removing different modules would cause \(1.34\sim 2.77\) Span F1 on test set.

Besides, we do ablation study on WikiEvents. As shown in Table 8, the Head F1 decreases by \(0.70\sim 2.02\) and \(0.88\sim 2.96\) for Arg Identification and Arg Classification sub-tasks respectively, once different modules are removed from Tsar\({}_{\rm base}\). Similar conclusions can be drawn from the results of Tsar\({}_{\rm large}\), which is shown in Table 9.