SlowGuess/NAACL_2021_OCR huggingface dataset:
MinhTranPhu
VinAI Research
v.minhttp@vinai.io
Thien Huu Nguyen
thien@cs.uoregon.edu
We study the problem of Event Causality Identification (ECI) to detect causal relation between event mention pairs in text. Although deep learning models have recently shown state-of-the-art performance for ECI, they are limited to the intra-sentence setting where event mention pairs are presented in the same sentences. This work addresses this issue by developing a novel deep learning model for document-level ECI (DECI) to accept inter-sentence event mention pairs. As such, we propose a graph-based model that constructs interaction graphs to capture relevant connections between important objects for DECI in input documents. Such interaction graphs are then consumed by graph convolutional networks to learn document context-augmented representations for causality prediction between events. Various information sources are introduced to enrich the interaction graphs for DECI, featuring discourse, syntax, and semantic information. Our extensive experiments show that the proposed model achieves state-of-the-art performance on two benchmark datasets.
Event Causality Identification (ECI) is an important problem in Information Extraction that seeks to predict causal relation between a pair of events mentioned in text. For instance, in the sentence "The building was nearly destroyed by a fire early Tuesday morning", an ECI system should be able to recognize the causal relation between the two events triggered by "destroyed" and "fire" (called event mentions), i.e., "fire" cause "destroyed". ECI finds its applications for a wide range of problems in natural language processing (NLP), including machine reading comprehension (Berant et al., 2014), future event forecasting (Hashimoto, 2019), and why-question answering (Oh et al., 2016).
The early approach for ECI has involved feature-based methods (Do et al., 2011; Hashimoto, 2019;
Figure 1: An example for document-level ECI.
Ning et al., 2018; Gao et al., 2019) while the recent approach has examined deep learning methods to deliver state-of-the-art performance for this task (Kadowski et al., 2019; Liu et al., 2020). Despite the good performance, the existing deep learning methods for ECI are limited in that they only model the context at the sentence level, assuming the event mention pairs of interest to be in the same sentences (i.e., intra-sentence setting). On the one hand, this assumption fails to cover the inter-sentence scenario where the input pairs of event mentions can appear in different sentences in the documents, e.g., in the recent EventStoryLine dataset for ECI (Caselli and Vossen, 2017). On the other hand, the sole modeling of sentence context cannot benefit from the document-level information that can provide useful evidence to facilitate the causality prediction for events. An example can be seen in Figure 1 where the interested pair of event mentions involves  $damaged_{2}$  and  $quake_{2}$  in the last (green) sentence. A system that only considers sentence context might find it challenging to predict causal relation in this case due to the long distance and the appearance of many irrelevant words between  $damaged_{2}$  and  $quake_{2}$  in the sentence. However, if a system relies on document-level information and recognizes the coreference of the event mention pairs ( $damaged_{2}$ ,  $damaged_{1}$ ) and ( $quake_{2}$ ,  $quake_{1}$ ), it can exploit the clear ev
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3480-3490 June 6-11, 2021. ©2021 Association for Computational Linguistics
idence of "damaged<sub>1</sub> due to the quake<sub>1</sub>" to infer the causal relation for damaged<sub>2</sub> and quake<sub>2</sub>.
To fill this gap, this work aims to develop a deep learning model for document-level ECI (DECI) where input event mentions can reside in different sentences of an input document. As such, a major challenge in modeling document-level context with deep learning involves capturing necessary interactions/connections between relevant objects for ECI. For instance, in our example in Figure 1, relevant objects include the event mentions and the important context words (i.e., "due to) while necessary connections involve event coreference and interactions of event mentions with context words (i.e., between damaged<sub>1</sub>, quake<sub>1</sub>, and "due to"). Motivated by this intuition, we design the graph-based model for DECI where interaction graphs over relevant objects for documents are explicitly generated and consumed by Graph Convolutional Networks (GCN) (Kipf and Welling, 2017; Nguyen and Grishman, 2018) to induce representation vectors for prediction. To our knowledge, this is the first work that employs interaction graphs for documents and GCNs for ECI.
How can interaction graphs for documents (i.e., nodes and edges) be formed to learn effective representation vectors for ECI? First, the intuitive approach to design nodes for interaction graphs is to leverage relevant objects for ECI in documents. Accordingly, we employ all the words, event mentions and entity mentions in a document to establish nodes for its interaction graph. Here, we note that entity mentions (e.g., names, pronouns, nominals) might also be helpful for ECI as entity mentions can serve as arguments (participants) of events and events with the same arguments might have better chance to involve in the causal relation.
Second, for edges of interaction graphs, we propose to exploit different knowledge sources or information types to create different types of connections for the graph nodes. Such connection types are then combined to produce a single rich interaction graph for an input document for representation learning in ECI. In particular, we focus on three major types of information for node connections for ECI in this work, i.e., discourse-based, syntax-based, and semantic-based information. As such, the discourse-based information explores the sentence boundary and coreference of entity/event mentions in documents to link the nodes in interaction graphs (motivated by our example in Figure
1). The syntax-based information connects words based on their syntactic relations in dependency trees of sentences, suggested by the use of shortest dependency paths between event mentions as features for ECI in prior work (Gao et al., 2019). In contrast, the intuition for semantic-based information is that semantically related words/entity/event mentions in documents can also provide useful evidences to infer the causal relation for events. For instance, consider the following sentence:
"The violence in and near the Yida refugee camp, located 10 miles south of the border, came one day after bombings were reported in another region of South Sudan, an attack that provoked strong condemnation from the U.S. State Department."
Here, the causal relation between "attack" and "condemnation" can be easily predicted due to the direct evidence in the context (i.e., via "provoked"). However, the more complicated and implicit context between "bombings" and "condemnation" would make it more difficult for ECI systems to realize the causality in this case. Fortunately, the systems can combine the causal relation between "attack" and "condemnation" and the close semantic similarity between the two events "bombings" and "attack" to facilitate the causality prediction between "bombings" and "condemnation".
Finally, we propose a novel mechanism to regularize interaction graphs and representation vectors to further improve the representation learning for DECI. As such, we aim to constrain the model so edges with small weights in the generated graphs have minimized contribution to representation vectors. In this way, we expect the model to be more robust against irrelevant/noisy edges in the graphs and still promote useful edges for representation learning. We conduct extensive experiments on two datasets for DECI. The results demonstrate the effectiveness of the proposed model and lead to state-of-the-art performance for DECI.
We formulate DECI as a binary classification problem. The input to the models include a document  $D = w_{1},w_{2},\ldots ,w_{N}$  (of  $N$  words/tokens) that can have multiple sentences, and two event mentions of interest  $e_s$  and  $e_t$  in  $D$ . The goal of DECI is to predict whether there exists a causal relation between  $e_s$  and  $e_t$  in  $D$ . Our model for DECI involves three major components: (i) Document Encoder to transform the words into representation vectors,
(ii) Structure Generation to generate an interaction graph for  $D$ , and (iii) Representation Regularization to regularize the representation vectors. We provide details for these components below.
2.1 Document Encoder
In the first step, we transform each word  $w_{i} \in D$  into a representation vector  $x_{i}$  using the contextualized embeddings BERT (Devlin et al., 2019) (i.e., the BERT<sub>base</sub> version). In particular, as BERT might split  $w_{i}$  into several word-pieces, we employ the average of the hidden vectors for the word-pieces of  $w_{i}$  in the last layer of BERT as the representation vector  $x_{i}$  for  $w_{i}$ . To handle long documents with BERT, we divide  $D$  into segments of 512 word-pieces to be encoded separately. The resulting sequence  $X = x_{1}, x_{2}, \ldots, x_{n}$  for  $D$  is then sent to the next steps for further computation.
2.2 Structure Generation
The goal of this section is to generate an interaction graph  $\mathcal{G} = \{\mathcal{N},\mathcal{E}\}$  for  $D$  to facilitate representation learning for DECI. As such, the nodes and edges in  $\mathcal{G}$  for our DECI problem are constructed as follows:
Nodes: The node set  $\mathcal{N}$  for our interaction graph  $\mathcal{G}$  should capture relevant objects for the causal prediction between the two event mentions of interest  $e_s$  and  $e_t$  in  $D$ . As motivated earlier, we consider all the context words  $w_i$ , event mentions, and entity mentions in  $D$  as relevant objects for our DECI problem. Formally, let  $E = \{e_1, e_2, \dots, e_{|E|}\}$  and  $M = \{m_1, m_2, \dots, m_{|M|}\}$  be the sets of event mentions and entity mentions in  $D$  respectively (i.e.,  $e_s, e_t \in E$ ). The node set  $\mathcal{N}$  for  $\mathcal{G}$  is thus formed by the union of  $D$ ,  $E$ , and  $M$ :  $\mathcal{N} = D \cup E \cup M = \{n_1, n_2, \dots, n_{|\mathcal{N}|}\}$ . In this work, we use the provided event mentions in the datasets for  $E$ , following prior work on DECI (Gao et al., 2019) while the Stanford CoreNLP toolkit is employed to obtain the entity mentions  $M$ .
Edges: To formally represent the edges between the nodes in  $\mathcal{N}$  for  $\mathcal{G}$ , we use the adjacency matrix  $A = \{a_{ij}\}_{i,j=|\mathcal{N}|}$  ( $a_{ij} \in \mathbb{R}$ ). Here, as we aim to use  $A$  as the input for Graph Convolutional Networks (GCN) to learn representation vectors for DECI, the value/score  $a_{ij}$  between two nodes  $n_i$  and  $n_j$  in  $\mathcal{N}$  is expected to estimate the importance (or the level of interaction) of  $n_j$  for the representation computation of  $n_i$ . In this way,  $n_i$  and  $n_j$  can directly interact and influence the representation computation for each other even if they are sequen
tially far away from each other in  $D$ . As presented in the introduction, three types of information are exploited to design the edges  $\mathcal{E}$  (or compute the interaction scores  $a_{ij}$ ) for  $\mathcal{G}$  in our model, including the discourse-based, syntax-based and semantic-based information.
**Discourse-based Edges:** As the input document  $D$  can involve multiple sentences and event/entity mentions, understanding where they span and how they relate to each other is crucial to effectively encode the document for DECI. As such, we propose to exploit three types of discourse information to obtain the interaction graph  $\mathcal{G}$  for  $D$ , i.e., the sentence boundary, the coreference structure, and the mention span for event/entity mentions in  $D$ .
Sentence Boundary: The intuition for this type of information is that two event/entity mentions appearing in the same sentences tend to be more contextually related to each other than those in different sentences. This suggests the better usefulness of event/entity mentions in the same sentences for the representation computation of each other. To capture this intuition, we propose to compute the sentence boundary-based interaction score  $a_{ij}^{sent}$  for the nodes  $n_i$  and  $n_j$  in  $\mathcal{N}$  where  $a_{ij}^{sent} = 1$  if  $n_i$  and  $n_j$  are the event/entity mentions of the same sentences in  $D$  (i.e.,  $n_i, n_j \in E \cup M$ ); and 0 otherwise.  $a_{ij}^{sent}$  will be used as an input to compute the overall interaction score  $a_{ij}$  for  $\mathcal{G}$  later.
Coreference Structure: Instead of considering within-sentence information as in  $a_{ij}^{sent}$ , coreference structure concerns the connection of event and entity mentions across sentences to enrich their representations with the contextual information of the coreferring ones (illustrated in Figure 1). As such, to enable the interaction of representations for coreferring event/entity mentions, we compute the conference-based score  $a_{ij}^{\text{coref}}$  for each pair of nodes  $n_i$  and  $n_j$  to contribute to the overall score  $a_{ij}$  for representation learning. Here,  $a_{ij}^{\text{coref}}$  is set to 1 if  $n_i$  and  $n_j$  are coreferring event/entity mentions in  $D$ , and 0 otherwise. Note that we use the Stanford CoreNLP toolkit to determine the coreference of entity mentions while similar to (Gao et al., 2019), golden event coreference information in the DECI datasets is utilized in this work.
Mention Span: The sentence boundary and coreference structure scores only model interactions of event and entity mentions in  $D$  based on discourse information. To further connect event and entity mentions with context words  $w_{i}$  for representation
learning, we employ the mention span-based interaction score  $a_{ij}^{span}$  as another input for  $a_{ij}$ , where  $a_{ij}^{span}$  is only set to 1 (i.e., 0 otherwise) if  $n_i$  is a word  $(n_i \in D)$  in the span of the entity/event mention  $n_j$  ( $n_j \in E \cup M$ ) or vice versa. Note that  $a_{ij}^{span}$  is important as it allows representation vectors for event/entity mentions to be grounded on the contextual information in  $D$ .
Syntax-based Edges: Prior work has leveraged dependency parsing trees of sentences in documents as an useful source of information to generate features for DECI systems, e.g., using the shortest dependency paths between the two event mentions of interest (Gao et al., 2019). As such, we expect the dependency trees of the sentences in  $D$  can also provide beneficial information to connect the nodes in  $\mathcal{N}$  to learn effective representation vectors for DECI. To this end, we propose to employ the dependency relations/connections between the words in  $D$  to obtain a syntax-based interaction score  $a_{ij}^{dep}$  for each pair of nodes  $n_i$  and  $n_j$  in  $\mathcal{N}$ , serving as an additional input for  $a_{ij}$ . In particular, directly inheriting the graph structures of the dependency trees of the sentences in  $D$ , we set  $a_{ij}^{dep}$  to 1 if  $n_i$  and  $n_j$  are two words in the same sentence (i.e.,  $n_i, n_j \in D$ ) and they are connected to each other in the corresponding dependency tree, and 0 otherwise. Thus, two words are considered important to each other for representation learning in DECI if they are neighbors in the dependency trees<sup>1</sup>.
Semantic-based Edges: This information exploits the semantic similarity of the nodes in  $\mathcal{N}$  to enrich the overall interaction scores  $a_{ij}$  for  $\mathcal{G}$ . The motivation is that a node  $n_i$  would contribute more to the representation vector of another node  $n_j$  for DECI if  $n_i$  is more semantically related to  $n_j$  (illustrated in the introduction). To this end, we propose two complementary methods to compute the semantic similarity between the nodes for  $a_{ij}$  based on context-based and knowledge-based information.
Context-based Semantic: In this method, we seek to first obtain a representation vector  $v_{i}$  for the semantic of each node  $n_i$  in  $\mathcal{N}$  based on its context in  $D$ . The context-based semantic similarity  $a_{ij}^{context}$  for the nodes is then be computed via such representation vectors and fed into the estimation of the overall interaction score  $a_{ij}$ . In particular, the context-based representation vector  $v_{i}$  for a word node  $n_i \in D$  is directly inherited from the contextualized embedding vector  $x_{c} \in X$
(i.e.,  $v_{i} = x_{c}$ ) of the corresponding word  $w_{c}$  for  $n_i$ . In contrast, for event and entity mentions, their representation vectors are computed by max-pooling the contextualized embedding vectors in  $X$  that correspond to the words in the event/entity mentions' spans. Eventually, the context-based similarity score  $a_{ij}^{\text{context}}$  for two nodes  $n_i$  and  $n_j$  in  $\mathcal{N}$  is obtained via the normalized score:
$$
k _ {i} = U _ {k} v _ {i}, q _ {i} = U _ {q} v _ {i}
$$
$$
a _ {i j} ^ {\text {c o n t e x t}} = \exp \left(k _ {i} q _ {j}\right) / \sum_ {u = 1.. | \mathcal {N} |} \exp \left(k _ {i} q _ {u}\right) \tag {1}
$$
where  $U_{k}$  and  $U_{q}$  are trainable weight matrices, and the biases are omitted for brevity in this work.
Knowledge-based Semantic: Instead of using contextual information, this method leverages the external knowledge of the nodes from knowledge bases to capture their semantic for node similarity computation. We expect the external knowledge for the nodes to provide complementary information for the contextual information in  $D$ , thus further enriching the semantic similarity scores (and overall interaction scores  $a_{ij}$ ) for the nodes in  $\mathcal{N}$ . To this end, we propose to utilize WordNet (Miller, 1995), a rich knowledge base for word meanings, to obtain external knowledge for the words in  $D$ . As such, WordNet involves a network of word meanings (i.e., synsets) that are connected to each other via various semantic relations (e.g., synonyms, hyponyms). Our first step to generate knowledge-based similarity scores involves mapping each word node  $n_i \in D \cap \mathcal{N}$  to a synset node  $M_i$  in WordNet using a Word Sense Disambiguation (WSD) tool. In particular, we employ WordNet 3.0 and the state-of-the-art BERT-based WSD model in (Blevins and Zettlemoyer, 2020) to perform the word-synset mapping in this work. Afterward, we compute a knowledge-based similarity score  $a_{ij}^{struct}$  for each pair of word nodes  $n_i$  and  $n_j$  in  $D \cap \mathcal{N}$  using the structure-based similarity of their linked synsets  $M_i$  and  $M_j$  in WordNet (i.e.,  $a_{ij}^{struct} = 0$  if either  $n_i$  or  $n_j$  is not a word node in  $D \cap \mathcal{N}$ ). Accordingly, the Lin similarity measure (Lin et al., 1998) for synset nodes in WordNet is utilized for this purpose:  $a_{ij}^{struct} = \frac{2*\mathrm{IC(LCS}(M_i,M_j))}{\mathrm{IC}(M_i) + \mathrm{IC}(M_j)}$ , where IC and LCS amount to the information content of synset nodes and the least common subsumer of two synsets in the WordNet hierarchy (the most specific ancestor node) respectively<sup>2</sup>.
2We use the nltk tool to obtain the Lin similarity: https://www.nltk.org/howto/wordnet.html.
<sup>1</sup>We use Stanford CoreNLP to parse the sentences.
Structure Combination: Up to now, six scores have been generated to capture the level of interactions in representation learning for each pair of nodes  $n_i$  and  $n_j$  in  $\mathcal{N}$  according to different information sources (i.e.,  $a_{ij}^{sent}, a_{ij}^{coref}, a_{ij}^{span}, a_{ij}^{dep}, a_{ij}^{context}$  and  $a_{ij}^{struct}$ ). For convenience, we group the six scores for each node pair  $n_i$  and  $n_j$  into a vector  $d_{ij} = [a_{ij}^{sent}, a_{ij}^{coref}, a_{ij}^{span}, a_{ij}^{dep}, a_{ij}^{context}, a_{ij}^{struct}]$  of size 6. To unify the scores in  $d_{ij}$  to form an overall rich interaction score  $a_{ij}$  for  $n_i$  and  $n_j$  in  $\mathcal{G}$ , we use the following normalization:
$$
a _ {i j} = \exp \left(d _ {i j} q ^ {T}\right) / \sum_ {u = 1.. | \mathcal {N} |} \exp \left(d _ {i u} q ^ {T}\right) \tag {2}
$$
where  $q$  is a learnable vector of size 6.
As mentioned above, given the combined interaction graph  $\mathcal{G}$  with the adjacency matrix  $A = \{a_{ij}\}_{i,j = |\mathcal{N}|}$ , we use GCNs to induce representation vectors for the nodes in  $\mathcal{N}$  for DECI. In particular, the GCN model in our work takes the context-based representation vectors  $v_{i}$  of the nodes  $n_i\in \mathcal{N}$  as the input. For convenience, we organize  $v_{i}$  into rows of the input matrix  $H_0 = [v_1,\dots ,v_{|\mathcal{N}|\} .$  The GCN model then involves  $G$  layers that generate the matrix  $H_{l}$  at the  $l$  -th layer for the nodes in  $\mathcal{N}$ $(1\leq l\leq G)$  via:  $H_{l} = RLU(AH_{l - 1}W_{l})$  ( $W_{l}$  is the weight matrix for the  $l$  -th layer). The output of the GCN model after  $G$  layers is  $H_{G}$  whose rows are denoted by  $H_{G} = [h_{1},\ldots ,h_{|\mathcal{N}|}]$ , serving as more abstract representation vectors for the nodes  $n_i$  for causality prediction. This GCN-based computation of  $H_{L}$  is written as  $H_{G} = [h_{1},\dots ,h_{|\mathcal{N}|}] =$ $GCN(H_0,A,\mathcal{G})$  for convenience.
2.3 Representation Regularization
Our model so far renders  $\mathcal{G}$  as a fully connected graph for representation learning whose edge weights are induced and recorded in the adjacency matrix  $A = \{a_{ij}\}_{i,j=1..|\mathcal{N}|}$  ( $0 < a_{ij} < 1$ ). However, it is intuitive that not all the edges in  $\mathcal{G}$  are relevant/necessary for the representation vectors in DECI. Some edges might even introduce noisy information if they are preserved in the graph. As such, we hypothesize that edges with small weights/scores assigned by the learning process in  $A$  are mostly noisy edges and should have minimal contribution to the induced representation vectors. To this end, we propose to obtain a sparser version  $\mathcal{G}'$  of  $\mathcal{G}$  where edges with small weights
are completely eliminated. In particular, we employ a threshold  $\alpha$  ( $0 < \alpha < 1$ ) and compute the adjacency matrix  $A' = \{a_{ij}'\}_{i,j=1..|\mathcal{N}|}$  for  $\mathcal{G}'$  via:  $a_{ij}' = a_{ij}$  if  $a_{ij} > \alpha$ ; and 0 otherwise.
To explicitly encourage the minimal contribution of small-weight edges, our goal is to enforce that the representation vectors learned by the sparse graph  $\mathcal{G}'$  are still close to those learned by the full graph  $\mathcal{G}$  (i.e., the removal of smallweight edges in  $\mathcal{G}'$  does not have much effect on representation learning). To implement this idea, we first apply our GCN model over the sparse graph  $\mathcal{G}'$  to learn another version of GCN-based representation vectors for the nodes in  $\mathcal{N}$ :  $H_G' = [h_1',\dots,h_{|\mathcal{N}|}'] = GCN(H_0,A',\mathcal{G}')$ . Afterward, we seek to minimize the difference  $L_{reg}$  between representation vectors of corresponding nodes in  $H_G$  and  $H_G'$  in the overall loss function:  $L_{reg} = 1 / |\mathcal{N}|\sum_{i = 1..|\mathcal{N}|}||h_i - h_i'||_2^2$ .