SlowGuess/NAACL_2021_OCR huggingface dataset:
Finally, let  $n_{s'}$  and  $n_{t'}$  be the two nodes in  $\mathcal{N}$  that correspond to the two event mentions of interest  $e_s$  and  $e_t$  for DECI. An overall representation vector  $V = [h_{s'}, h_{t'}, h_{s'}, h_{t'}']$  is formed (from both  $H_L$  and  $H_L'$ ) and fed into a two-layer feed-forward network with softmax in the end to produce the distribution  $P(\cdot | D, e_s, e_t)$  over the two possible types for our DECI problem (whether there is a causal relation between  $e_s$  and  $e_t$  or not). The negative log-likelihood function  $L_{pred}$  is then computed by:  $L_{pred} = -\log P(y^* | D, e_s, e_t)$  ( $y^*$  is the golden type for DECI). The overall loss function to train our model is thus:  $L = L_{pred} + \gamma L_{reg}$  where  $\gamma$  is a trade-off parameter.
3.1 Datasets and Hyperparameters
Following prior work (Gao et al., 2019; Liu et al., 2020), we evaluate our models on two benchmark datasets for ECI, i.e., EventStoryLine and Causal-TimeBank. In particular, EventStoryLine (i.e., version 0.9) is introduced in (Caselli and Vossen, 2017), involving 258 documents, 22 topics, 4316 sentences, 5334 event mentions, 7805 intrasentence and 46521 inter-sentence event mention pairs (1770 and 3855 of them are annotated with a causal relation respectively). Following (Gao et al., 2018), we group documents according to their topics and put the topics in the order based on their topic IDs. The documents in the last two topics are used for the development data while the remaining 20 documents are employed for a 5-fold
Model	Intra-sentence	Inter-sentence	Intra + Inter
P	R	F1	P	R	F1	P	R	F1
OP (Caselli and Vossen, 2017)	22.5	98.6	36.6	8.4	99.5	15.6	10.5	99.2	19.0
LSTM (Cheng and Miyao, 2017)	34.0	41.5	37.4	13.5	30.3	18.7	17.6	33.9	23.2
Seq (Choubey and Huang, 2017)	32.7	44.9	37.8	11.3	29.5	16.4	15.5	34.3	21.4
KnowDis* (Zuo et al., 2020)	39.7	66.5	49.7	-	-	-	-	-	-
LR+ (Gao et al., 2019)	37.0	45.2	40.7	25.2	48.1	33.1	27.9	47.2	35.1
LIP (Gao et al., 2019)	38.8	52.4	44.6	35.1	48.2	40.6	36.2	49.5	41.9
BERT* (our implementation)	39.2	49.3	43.7	22.3	29.2	25.3	27.3	35.3	30.8
Know* (Liu et al., 2020)	41.9	62.5	50.1	-	-	-	-	-	-
RichGCN* (proposed)	49.2	63.0	55.2	39.2	45.7	42.2	42.6	51.3	46.6
cross-validation evaluation, using the same data split in (Gao et al., 2019; Liu et al., 2020). For Causal-TimeBank (Mirza, 2014a), this dataset consists of 184 documents, 6813 events, and 318 of 7608 event mention pairs annotated with a causal relations. Following (Liu et al., 2020), we do a 10-fold cross-validation evaluation using the same data split for this dataset. Note that as in (Liu et al., 2020), we only evaluate the ECI performance for intra-sentence events in Causal-TimeBank as the number of inter-sentence event mention pairs with the causal relation is very small (i.e., only 18 pairs).
We tune the hyperparameters for our model on the development data of EventStoryLine and use the chosen parameters to train the models for both EventStoryLine and Causal-TimeBank. The selected values from the tuning process include:  $1e-5$  for the learning rate of the Adam optimizer; 8 for the mini-batch size; 128 hidden units for all the feed-forward network and GCN layers; 2 layers for the GCN model  $(G = 2)$ ,  $\alpha = 0.5$  for the weight threshold, and  $\gamma = 0.2$  for the trade-off parameter in the loss function  $L$ . Finally, as mentioned earlier, we use the BERT<sub>base</sub> model (of 768 dimensions) for the pre-trained word embeddings (updated during the training) in this work.
We compare our model (called RichGCN) with the state-of-the-art models for ECI in each benchmark dataset as follows.
Story: For this dataset, the following baselines are chosen for comparison: (i) OP: a dummy model used in (Caselli and Vossen, 2017) that assigns a causal relation to every pair of event mentions; (ii) LSTM (Gao et al., 2019): a dependency path based sequential model that is adopted from (Cheng and Miyao, 2017); (iii) Seq (Gao et al., 2019): another dependency path based se
quential model that is originally developed for temporal relation prediction from (Choubey and Huang, 2017) and applied to ECI; (iv) BERT: a baseline method that takes the embedding vectors from BERT and performs ECI in (Liu et al., 2020). Note that (Liu et al., 2020) only reports the performance on intra-sentence events of EventStoryLine for this model. We reimplement and fine-tune the model to obtain its performance for inter-sentence events. Our reimplemented model for BERT achieves higher performance on intra-sentence ECI than those in (Liu et al., 2020); (v) KnowDis (Zuo et al., 2020): a BERT-based model that leverages additional data from distant supervision; (vi) LR+ and LIP (Gao et al., 2019): document structure-based models that have the current state-of-the-art performance for inter-sentence ECI; and (vii) Know (Liu et al., 2020): a BERT-based model that exploits ConceptNet and achieves the state-of-the-art performance for intra-sentence ECI. Table 1 shows the performance of the models.
Table 1: Model's performance on EventStoryLine. The performance improvement of RichGCN over the baselines is significant with  $p < {0.01}$  . * designates models that use BERT embeddings.
Model	P	R	F1
RB (Mirza, 2014b)	36.8	12.3	18.4
ML (Mirza, 2014a)	67.3	22.6	33.9
BERT* (Liu et al., 2020)	30.3	41.1	34.9
Know* (Liu et al., 2020)	36.6	55.6	44.1
RichGCN* (proposed)	39.7	56.5	46.7
Table 2: Model's performance on Causal-TimeBank (for intra-sentence events). RichGCN is significantly better than the baselines with  $p < 0.01$ . * indicates BERT-based models.
Causal-TimeBank: We use the following baselines for this dataset: (i) RB: a rule-based system in (Mirza, 2014b); (ii) ML: a feature-based model for ECI in (Mirza, 2014a); and (iii) BERT and Know (Liu et al., 2020): These are the same models BERT and Know (respectively) for EventStoryLine (both are based on BERT). We use the reported performance for the two models in (Liu et al., 2020) for a fair comparison. Know has the
current state-of-the-art performance for this dataset in our 10-fold cross-validation setting. Note that the BERT model essentially corresponds to our RichGCN model when the interaction graphs  $\mathcal{G}$  and  $\mathcal{G}'$  (thus the GCN model) are completely excluded. Table 2 presents the performance of these models on Causal-TimeBank.
The most important observation from the tables is that the proposed model RichGCN significantly outperforms all the baselines for both intra- and inter-sentence events on both EventStoryLine and Causal-TimeBank  $(p < 0.01)$ , thus clearly demonstrating the effectiveness of the proposed model for DECI. In addition, we also see that BERT performs much worse than the document structure-based models LR+, LIP and RichGCN. The sequential modeling of the context in BERT is thus not effective for document-level ECI, necessitating better mechanisms to encode document context (e.g., via the interaction graph of relevant objects as we do). Finally, the significant better performance of RichGCN over Know for intra-sentence ECI in different datasets confirms our intuition in the introduction that capturing context beyond sentences (i.e., document context as in RichGCN) is helpful for causal prediction of intra-sentence event pairs.
This section analyzes the contribution of each component in the proposed model with an ablation study. In particular, we examine the following ablated models: (i) "RichGCN - x" where  $\mathbf{x}$  is one of the six interaction scores generated to compute the unified score  $a_{ij}$  (i.e.,  $a_{ij}^{sent}, a_{ij}^{coref}, a_{ij}^{span}, a_{ij}^{dep}, a_{ij}^{context}$  and  $a_{ij}^{struct}$ ). For instance, "RichGCN -  $a_{ij}^{coref}$ " refers to the RichGCN model where the coreference-based interaction score  $a_{ij}^{coref}$  is excluded in the computation of the overall score  $a_{ij}$  in Equation 2; (ii) "RichGCN - Entity Nodes": the entity mention nodes in  $M$  are not included in the construction of interaction graph  $\mathcal{G}$  in this model (i.e.,  $\mathcal{N} = D \cup E$  only); (iii) "RichGCN - Event Nodes": the event mention nodes in  $E$  do not appear in the node set  $\mathcal{N}$  of the interaction graph  $\mathcal{G}$  in RichGCN (i.e.,  $\mathcal{N} = D \cup M$ ). We directly use the representation vectors  $v_i$  for the event mentions in the overall representation vector  $V$  for prediction in this model. Note that the interaction matrix  $A$  is also adapted accordingly in the ablated models "RichGCN - Entity Nodes" and "RichGCN
- Event Nodes"; (iv) "RichGCN - GraphCombination": this model does not combine the six generated interaction scores to compute an overall score  $a_{ij}$  for  $A$  in Equation 2. Instead, it considers each of the six generated interaction scores as forming a separate interaction graph, thus generating six different graphs. The GCN model is then applied over these six graphs (using the same input representation vectors  $v_{i}$  for the nodes  $n_i$  in  $\mathcal{N}$ ). The outputs of the GCN model for the same node  $n_i$  (with different graphs) are then concatenated to produce the final representation vector for  $n_i$  (i.e., serving as  $h_i$  in the model). Note that we still employ the sparse graph idea (with  $\mathcal{G}'$  and the loss  $L_{reg}$ ) in this model; (v) "RichGCN -  $\mathcal{G}$ " and "RichGCN -  $\mathcal{G}''$ : these models exclude the full graphs  $\mathcal{G}$  or  $\mathcal{G}'$  from RichGCN (respectively). The regularization loss  $L_{reg}$  is thus not used and the vectors generated by the excluded graphs are not employed in the final vector  $V$  (i.e.,  $h_{s'}, h_{t'}, h_{s'}, h_{t'}'$ ) for prediction in these cases; and (vi) "RichGCN -  $L_{reg}$ : this model removes the regularization term  $L_{reg}$  from the overall loss function  $L$ .
Model	Intra	Inter	Intra +Inter
RichGCN (full)	59.5	43.3	48.3
RichGCN - a sent ij	55.8	40.3	44.9
RichGCN - a coref ij	57.2	37.6	43.0
RichGCN - a span ij	49.2	35.8	39.5
RichGCN - a dep ij	54.1	39.5	43.5
RichGCN - a context ij	57.7	42.2	46.9
RichGCN - a struct ij	57.5	41.5	46.3
RichGCN - GraphCombination	54.7	41.1	44.7
RichGCN - Entity Nodes	51.6	38.7	42.8
RichGCN - Event Nodes	52.0	38.4	42.5
RichGCN - G	53.7	39.9	44.3
RichGCN - G'	54.6	40.5	44.8
RichGCN - Lreg	56.8	41.3	46.3
Table 3: Performance of models (F1) on the development data of EventStoryLine.
Table 3 shows the performance of the models on the development data of EventStoryLine. As can be seen from the table, all the components are helpful for the proposed model RichGCN as eliminating any of them degrades the performance significantly for both intra- and inter-sentence ECI. Notably, the worse performance of "RichGCN -  $\mathcal{G}$ " suggests that only using the sparse graph  $\mathcal{G}$  for GCN to completely cancel small-weight edges in  $\mathcal{G}$  is suboptimal as it might unexpectedly remove some useful (though small-weight) edges. Instead, the sparse graph should be exploited in conjunction with the full graph to minimize the overall contribu
tion of small-weight edges as we do in RichGCN.
3.4 Cross-Topic Evaluation
To further demonstrate the benefits of document context modeling with GCN for intra-sentence ECI, we perform a cross-topic evaluation on EventStoryLine as in (Liu et al., 2020). In particular, as documents in different topics tend to mention different events in EventStoryLine, this section aims to train the models on a source topic, but evaluate them on other topics (i.e., the target topics) to reveal the topic generalization. Following (Liu et al., 2020), we choose topics T8, T13, and T18 in EventStoryLine as the source topics. For each of these source topics, the other topics are ranked according to their similarity with the source topic. As such, the similarity score between two topics  $t_1$  and  $t_2$  is based on  $\delta = \frac{E_{t_1} \cap E_{t_2}}{E_{t_1} \cup E_{t_2}}$ , where  $E_t$  is the set of lemmas of event trigger words in topic  $t$ . Afterward, topics with the lowest, medium and highest similarity scores with the source topic are chosen as the target topics for evaluation. Table 4 presents the intra-sentence ECI performance (i.e., F1 scores) of LIP, Know (Liu et al., 2020) and the proposed model RichGCN for this cross-topic experiment.
Setting	Source (Train)	Target (Test)	δ	LIP	Know	RichGCN (Proposed)
Low	T8	T35	0%	2.8	44.7	47.0
T13	T12	0%	-	25.1	42.7
T18	T30	0%	-	19.5	28.2
Medium	T8	T3	1.7%	6.7	30.9	38.0
T13	T41	0.1%	4.5	28.6	41.6
T18	T35	2.8%	17.1	44.5	50.0
High	T8	T19	12.4%	19.4	45.1	54.0
T13	T14	17.1%	27.4	46.0	50.5
T18	T33	27.2%	32.2	49.0	53.1
Table 4: Cross-topic performance (F1) for inter-sentence ECI.  $\delta = \frac{E_{t_1} \cap E_{t_2}}{E_{t_1} \cup E_{t_2}}$  is the topic similarity score.
It is clear from the table that RichGCN is significantly better than the baselines LIP and Know over different cross-topic settings, thereby further testifying to the generalization advantages of capturing document-level context via GCN for intra-sentence ECI in the proposed model.
To suggest potential directions for future research, we analyze the errors made by the proposed model. In particular, we sample 100 event mention pairs in the development data of EventStoryLine whose causal relation cannot be predicted correctly by
RichGCN. Afterward, we manually categorize these examples into different types that are described below:
(i) Implicit causal relation:  $33\%$  of the errors in our model is due to the implicit indication of the causal relation between two event mentions in the context, necessitating common-sense knowledge to make correct causality prediction. For instance, consider the following document:
"South Sudan warns of war after Sudan bombs refugee camp. Military aircraft from Sudan crossed the new international border with South Sudan and dropped bombs Thursday in and around a camp filled with refugees, officials said. A government official initially reported deaths, but an American activist who spoke to aid workers at the camp later said there were no casualties."
RichGCN cannot recognize the causal relation between two events "bombs" and "deaths" in this document. The reason is that there is no explicit context in the document to hint such a relation. The models need to rely on the common-sense causal order of "bombs" and "deaths" to correctly predict the label in this case.
(ii) Preprocessing toolkit: Our model leverages several toolkit to obtain information to construct the interaction graph  $\mathcal{G}$ , including the dependency parser, the entity mention detection and coreference (i.e., from Stanford CoreNLP), and the word sense disambiguation model.  $18\%$  errors in our model originate from the errors in such toolkit that introduce noise into our model. For instance, Stanford CoreNLP incorrectly considers "South Sudan" and "Sudan" as the same entity in some of the examples.
(iii) Lack of factuality modeling: Our model fails in this error type as it does not consider the factuality of the causal relation, treating hypothetical relations as the actual ones. This accounts for  $5\%$  of the errors. For instance, in the document above, the proposed model predicts the causal relation between "war" and "bombs"; however, this is incorrect (not factual) due to the appearance of the word "warns".
(iv) Lack of fine-grained distinction: The errors in this type (accounting for  $23\%$ ) are due to the failure to capture the fine-grained distinction of event mentions in the context, causing the confusion and incorrect predictions for the model. For instance, in the sentence "Updated : July 02 , 2013 15:50 IST A 6. 1-magnitude earthquake which hit the Indonesian province of Aceh on Tues-
day killed a child, injured dozens and destroyed buildings, sparking panic in a region devastated by the quake-triggered tsunami of 2004.", our model incorrectly predict "killed" and "injured" as having a causal relation with "quake" (underlined). This stems from the strong connection between the underlined "quake" and the "1-magnitude earthquake" in the same sentence (i.e., due to the sentence boundary- and semantic-based interaction scores). Such strong connection leads the model to believe that "killed" and "injured" are also caused by the underlined "quake" as the "1-magnitude earthquake". The model would need to better encode the fine-grained distinction between the underlined "quake" and the "1-magnitude earthquake" (i.e., of the year 2004 and 2013 respectively) to address this issue. Finally, our analysis shows that the other errors have to do with annotation errors  $(6\%)$  and more complicated issues that cannot be categorized clearly.
The early feature-based methods for ECI has explored different features and resources to improve the performance, including lexical and syntactic patterns (Hashimoto, 2019; Gao et al., 2019), causality cues/markers (e.g., "because") (Riaz and Girju, 2014a; Hidey and McKeown, 2016), statistical co-occurrence of events (Beamer and Girju, 2009; Do et al., 2011; Hu et al., 2017), temporal patterns (Mirza, 2014a; Ning et al., 2018), lexical semantics of events (Riaz and Girju, 2013, 2014b), and weakly supervised data (Hashimoto, 2019). Although we also apply related features and resource for ECI (e.g., syntax, WordNet), our model employs such resources to build interaction graphs for documents to induce more abstract representations with GCNs. Recently, deep learning has been applied to solve ECI, leveraging advanced language models (e.g., BERT) (Kadowaki et al., 2019; Zuo et al., 2020) and common-sense knowledge resources (i.e., ConceptNet) (Liu et al., 2020) to produce state-of-the-art performance. However, none of these deep learning models has explored document-context modeling with rich information for graph construction and GCNs as we do.
Recently, there have been much interest in designing task-specific graphs to learn representation vectors for different NLP tasks, including sentence-level graphs for event factuality identification (Pouran Ben Veyseh et al., 2019) and event ar
gument extraction (Pouran Ben Veyseh et al., 2020; Nguyen and Nguyen, 2021), and document-level graphs for relation extraction (Christopoulou et al., 2019; Nan et al., 2020; Tran et al., 2020) and event argument extraction (Veyseh et al., 2021). Our model is different from such related work in that we design document-level interaction graphs that are tailored to our ECI task. In addition, our model is also the first model that employs the inherent structure of external knowledge graphs (i.e., WordNet) to augment interaction graphs for documents in representation learning.
We present a novel deep learning model for document-level ECI to address the limitation of prior deep learning models that only focus on causal prediction for inter-sentence event mention pairs. Our model designs interaction graphs to capture important objects and connections for input documents, leveraging GCNs to induce representation vectors for causal prediction. We introduce several information sources to enrich the interaction graphs based on discourse, syntax, and semantic information. The experiments confirm the effectiveness of the proposed information sources and models for DECI. In the future, we plan to extend our model to other related tasks, e.g., event coreference resolution (Nguyen et al., 2016).
Brandon Beamer and Roxana Girju. 2009. Using a bigram event model to predict causal potential. In CI-CLing.
Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1499-1510, Doha, Qatar. Association for Computational Linguistics.
Terra Blevins and Luke Zettlemoyer. 2020. Moving down the long tail of word sense disambiguation with gloss informed bi-encoders. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1006–1017, Online. Association for Computational Linguistics.
Tommaso Caselli and Piek Vossen. 2017. The event StoryLine corpus: A new benchmark for causal and temporal relation extraction. In Proceedings of the Events and Stories in the News Workshop, pages 77-86, Vancouver, Canada. Association for Computational Linguistics.
Fei Cheng and Yusuke Miyao. 2017. Classifying temporal relations by bidirectional LSTM over dependency paths. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-6, Vancouver, Canada. Association for Computational Linguistics.
Prafulla Kumar Choubey and Ruihong Huang. 2017. A sequential model for classifying temporal relations between intra-sentence events. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1796-1802, Copenhagen, Denmark. Association for Computational Linguistics.
Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Connecting the dots: Document-level neural relation extraction with edge-oriented graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4925-4936, Hong Kong, China. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.