Enhancing Event Causality Identification with Event Causal Label and Event Pair Interaction Graph
Year: 2023

Abstract:
Most existing event causality identification (ECI) methods rarely consider the event causal label information and the interaction information between event pairs. In this paper, we propose a framework to enrich the representation of event pairs by introducing the event causal label information and the event pair interaction information. In particular, 1) we design an event-causal-label-aware module to model the event causal label information, in which we design the event causal label prediction task as an auxiliary task of ECI, aiming to predict which events are involved in the causal relationship (we call them causality-related events) by mining the dependencies between events. 2) We further design an event pair interaction graph module to model the interaction information between event pairs, in which we construct the interaction graph with event pairs as nodes and leverage graph attention mechanism to model the degree of dependency between event pairs. The experimental results show that our approach outperforms previous stateof-the-art methods on two benchmark datasets EventStoryLine and Causal-TimeBank.

heading: 1 Introduction
text: Event causality identification (ECI) aims to identify the causal relationship between pairs of events in the text. As shown in Figure 1(a), giving text and events as inputs, the ECI model needs to identify three causal relationships < e7, cause, e5 >, < e5, cause, e2 >, and < e7, cause, e2 >. As a semantic relationship, causality is important for semantic understanding and discourse analysis. Moreover, causal knowledge identified from a text can be useful for many natural language processing tasks (Fei et al., 2020a,b; Wang et al., 2020; Zhou et al., 2021; Dalal et al., 2021).
Various approaches have been proposed for ECI, from the early feature-based methods (Mirza, 2014;
*Corresponding author.
Caselli and Vossen, 2017; Gao et al., 2019) to the current neural-network-based methods (Liu et al., 2020; Zuo et al., 2020, 2021a,b; Cao et al., 2021; Phu and Nguyen, 2021). The existing methods have achieved impressive performance. However, as far as we know, they usually improve the performance of ECI models by introducing commonsense knowledge or generating additional training data through data augmentation, with less focus on the inherent characteristics of the data. Specifically, 1) Minimal use of event causal label information. In the ECI task, a sentence usually contains multiple events. However, not all events are involved in causality. For instance, the sentence shown in Figure 1(a) has seven events, among which e2, e5, and e7 are causality-related events, whereas e1, e3, e4, and e6 are causality-unrelated events. In general, there is a strong semantic correlation between events that have a causal relationship. If the model can first identify e2, e5, and e7 as causality-related events according to the dependencies between events and contextual semantics, it can limit the identification scope of causality to some extent, as shown in Figure 1(b), thereby reducing the interference of unrelated events. Furthermore, by performing statistical analysis on two benchmark datasets, EventStoryLine and Causal-TimeBank, we find that 55.13% and 38.63% of the sentences with events in the two
10314
datassets contained four or more events, where the average number of the different candidate event pairs is 13.12 and 10.64, and the average number of the event pairs with causal relationship is 1.99 and 0.30, respectively. If we can first predict which events are causality-related events according to the dependencies between them, the average number of candidate event pairs composed of causality-related events is 3.68 and 0.37, respectively, thereby narrowing the range of causality from 13.12 and 10.64 to 3.68 and 0.37, respectively. Therefore, it is necessary to introduce the event causal label information to assist the ECI tasks. 2) Little focus on the interaction between event pairs. The existing method (Phu and Nguyen, 2021) uses events as nodes and leverages toolkits to obtain information (such as dependency parsing tree and coreference) to construct event graph. However, they are modeled with events as nodes, cannot directly learn the interaction between event pairs. As shown in Figure 1(a), e2 and e7 are far from each other in the text. Thus, directly identifying the causal relationship between them is difficult for the model. If < e7, cause, e5 > and < e5, cause, e2 > are known, it is easier to infer the causal relationship between e2 and e7 according to the transitivity of the causal relationship (Hall, 2000). Therefore, it is essential to introduce the interaction information between event pairs to learn their dependencies.
To address the above limitations, we propose a framework called ECLEP, which introduces Event Causal Label information and Event Pair interaction information to enrich the representation of event pairs. In particular, 1) we design an eventcausal-label-aware module to model the event causal label information, in which we introduce the event causal label prediction task as an auxiliary task of ECI to mine the dependencies between events. 2) We further design an event pair interaction graph module to model the interaction information between event pairs. In particular, we construct the interaction graph with event pairs as nodes and adopt graph attention mechanism to model the degree of dependency between event pairs. The experimental results on two benchmark datasets show that our overall framework outperforms previous state-of-the-art methods.

heading: 2 Methodology
text: In ECI, the goal is to predict whether a causal relationship exists in each event pair (ei, ej)(i ̸= j)
by giving text S = (w1, w2, ..., wn) and events set E = (e1, e2, ..., em) contained in S as inputs, where wi denotes the i-th word in S, and ei denotes the i-th event in E. The overall framework of the proposed method is shown in Figure 2, which mainly includes four modules, we illustrate each component in detail in the following.

heading: 2.1 Encoding Layer
text: Given an input sentence S, we adopt a pre-trained language model BERT (Devlin et al., 2019) as the sentence encoder to extract hidden contextual representation HS = (hw1 ,hw2 , ...,hwn), where hwi denotes the hidden representation of the ith token in S. Then, the events representation HE = (he1 ,he2 , ...,hem) can be obtained by summing the token representation contained in the event.

heading: 2.2 Event-Causal-Label-Aware Module
text: A sentence usually contains multiple events, however, not all events are causality-related events. We design an event-causal-label-aware module to mine the dependencies between events and help the model to pay attention to the extraction of the causal relationship from the event pairs composed of causality-related events. In general, there is a strong semantic correlation between events with causal relationships, so we first adopt the Transformer mechanism (Vaswani et al., 2017) to capture the dependencies between events, and get the updated event representation via Eq. (1).
HLE = Transformer(HE) = (h L e1 ,h L e2 , ...,h L em).
(1) Then, we construct a binary classification to predict the probability pLei of each event ei in E as a causality-related event via Eq. (2).
pLei = softmax(W L eiH L ei + b L ei), (2)
where W Lei and b L ei are learnable parameters.
To incorporate the event causal label into the event pair, we introduce a learable label vector set L = {lij}(lij ∈ {cr, cu}), where cr denotes the event pairs composed of causality-related events, cu denotes the event pairs that have at least one causality-unrelated event. In particular, the embedding vector L is randomly initialized via sampling from a uniform distribution and is learned together with the model training process. For the event pair (ei, ej), if ei and ej are both causality-related events (i.e. pLei ≥ 0.5 and pLej ≥ 0.5), the label of (ei, ej) is lij = cr, else lij = cu.

heading: 2.3 Event Pair Interaction Graph Module
text: To capture the interaction information between event pairs, we construct the interaction graph with event pairs as nodes and adopt the graph attention mechanism to adaptively fuse the information of neighbor nodes.
• Event Pair Interaction Graph Construction. Given the event set E, the events are combined in pairs to form the set of candidate causal event pairs EP = {epij}(0 < i, j ≤ |E|, i ̸= j) as nodes. In EP , the epij = [hei ;hej ; rij ; lij ], where hei and hej ∈ HE and rij denotes the relative position embedding between event ei and event ej in the text.
For the edges of the interaction graph, consider that event pairs in the same row or column are strongly associated with the current event pair (Ding et al., 2020). As shown in Figure 1(b), the relationship of event pair < e7, e2 > can be transmitted through the same row event pair < e7, e5 > and the same column event pair < e5, e2 >. Thus, we connect the edges between the event pairs in the same row or column and add self-loop edges to fuse the information of their nodes.
• Event Pair Interaction Graph Update. Considering that different neighbor nodes have different importance for each event pair, we leverage Graph Attention Networks (GAT) (Veličković et al., 2018) to model the degree of dependency between event pairs. GAT propagates information among nodes by stacking multiple graph attention layers. At the t-th graph attention layer, the representation
of each node can be updated via Eq. (3).
eptij = ReLU( ∑
uv∈N (ij) αtij,uvW tept−1uv + b t),
(3) where N (ij) denotes the directly neighboring nodes of epij ; the attention weight αtij,uv is learned via Eq. (4), which reflects the strength of aggregation level between the nodes ept−1ij and ept−1uv ; W t, W tij , W t uv, w
t, and btare learnable parameters.
eptij,uv = w ttanh([W tijep t−1 ij ;W t uvep t−1 uv ]),
αtij,uv = exp(LeakyReLU(eptij,uv))∑
st∈N (ij) exp(LeakyReLU(eptij,st))
,
(4)
Therefore, the updated nodes representation EP I = {epIij} can be obtained by stacking T layers to model the inter-node relationships.

heading: 2.4 Prediction and Training
text: The predicted probability pij of the event pair (ei, ej) as a causal event pair can be obtained by performing a binary classification with epIij as input via Eq. (5).
pij = softmax(W I ijep I ij + b I ij), (5)
where W Iij and b I ij are learnable parameters.
For training, we utilize the cross-entropy function to supervise the causal event pair prediction via Eq. (6).
Lep = − ∑
s∈D
∑
ei,ej∈Es ei ̸=ej
yij log(pij), (6)
where D denotes the training set; s denotes the sentence in D; Es denotes the events set in s; yij denotes the ground truth label of event pair (ei, ej).
We also add an auxiliary supervision for the event causal label prediction task via Eq. (7).
Le = − ∑
s∈D
∑
ei∈Es yei log(p
L ei), (7)
where pLei denotes the predicted probability of event ei as a causality-related event; yei denotes the ground truth label of event ei, which can be automatically obtained according to the labels of event pairs without any manual labeling.
The final loss function is a weighted sum of the aforementioned terms L = Lep + λeLe, where λe ∈ (0,1).

heading: 3 Experiments

heading: 3.1 Datasets and Evaluation Metrics
text: We verify the effectiveness of the proposed model on two benchmark datasets, EventStoryLine (Caselli and Vossen, 2017) and Causal-TimeBank (Mirza, 2014), respectively. Following previous works (Cao et al., 2021; Zuo et al., 2021a), we perform the 5-fold and 10-fold cross-validation on the two datasets, respectively. In addition, we adopt Precision (P), Recall (R), and F1-score (F1) as evaluation metrics.

heading: 3.2 Parameter Settings
text: We use HuggingFace’s Transformers1 library to implement the uncased BERT base model. The Adam algorithm (Kingma and Ba, 2015) is used as an optimizer, the learning rate is initialized to 2e-5, the batch size is set to 5, the GAT layers is set to 2, the dropout of GAT is set to 0.3, the dimensions of the event causal label embedding and relative position embedding are set to 80 and 40, respectively, and the weight λe is set to 0.2. Moreover, since the positive samples in the dataset are sparse, we adopt a negative sampling rate of 0.5 for training.
1https://github.com/huggingface/transformers

heading: 3.3 Baselines
text: We compare our model with the following baseline methods. Feature-based methods: 1) DD (Mirza and Tonelli, 2014), a data-driven method. 2) VRC (Mirza, 2014), a model with data filtering and causal signal enhancement. 3) OP (Caselli and Vossen, 2017), a dummy model to assign causality for ECI. 4) Seq (Choubey and Huang, 2017), a sequence model for ECI. 5) ILP (Gao et al., 2019), a document-level ECI model. Neural-networkbased methods: 1) KMMG (Liu et al., 2020), which proposes a knowledge-aware reasoner and a mention masking reasoner for ECI. 2) KnowDis (Zuo et al., 2020), a distantly supervised method for ECI. 3) LSIN (Cao et al., 2021), a method that utilize the structural commonsense knowledge for ECI. 4) LearnDA (Zuo et al., 2021a), a learnable knowledge-guided data augmentation method for ECI. 5) CauSeRL (Zuo et al., 2021b), a selfsupervised representation learning enhanced ECI method. 6) RichGCN (Phu and Nguyen, 2021), a graph convolutional network with rich information.

heading: 3.4 Overall Results
text: The experimental results are shown in Table 1, we can observe that our proposed method ECLEP outperforms all the baselines on the two datasets. In particular, on the EventStoryLine, compared with the current best method RichGCN, our method ECLEP achieves 1.9% improvement in the F1score; on the Causal-TimeBank, compared with the current best method CauSeRL, our method ECLEP achieves 3.1% improvement in the F1-score. This
finding indicates that our proposed method is effective for the ECI task. In addition, we observe that baseline methods of ECI often require external knowledge resources or toolkits to improve the performance. Our approach achieves the best performance by mining the inherent characteristics of the data.

heading: 3.5 Ablation Study
text: This section analyzes the contribution of each part in our model through ablation experiments, as shown in Table 2. In particular, we examine the following ablated models: 1) -ECL and -EPI denote the removal of the event causal label information and the event pair interaction information, respectively. We note that removing any of them degrades the performance for ECI, indicating that the two information we introduced are effective. 2) -Le denote the removal of auxiliary supervision. We note that removing it degrades overall performance, because it can help the model more sufficiently learn the representation of the label vector. 3) -pos denote the removal of relative position embedding. The experimental results show that relative position information is helpful for ECI, because the probability of causal relationship between events that are closer is higher than those that are farther apart.

heading: 3.6 Visualization Analysis
text: We visualize the distribution of each module to explore the effectiveness of our model further. The following can be observed from Figure 3: 1) The event causal label information and the event pair interaction information focus on different aspects of features to identify the causal relationships and they share complementary effects. This finding also provides an explanation on the good performance of our full model ECLEP. 2) The event causal label can limit the identification scope of causality to some extent, and help the model to pay attention to the extraction of the causal relationship from the
event pairs composed of causality-related events.

heading: 4 Conclusion
text: In this paper, we propose a framework to enrich the representation of event pairs by introducing event causal label information and event pair interaction information. The experimental results on two widely used datasets indicate that our approach is effective for the ECI task. In the future, we aim to mine other potential causal features for this task and apply our model to other types of relation extraction tasks, such as temporal relation extraction.
Limitations
In this paper, we only focus on whether or not there is a causal relationship between the given events, does not discriminate the specific cause/effect event. In addition, we only conduct research on sentence-level ECI, whereas document-level ECI often present more challenges. These are the focus of our future research.

heading: Acknowledgements
text: The authors would like to thank all anonymous reviewers for their valuable comments and suggestions which have significantly improved the quality and presentation of this paper. The works described in this paper are supported by the National Natural Science Foundation of China (62106130, 62076158, 62072294, 62272286), Natural Science Foundation of Shanxi Province, China (20210302124084), Scientific and Technological Innovation Programs of Higher Education Institutions in Shanxi, China (2021L284), and CCFZhipu AI Large Model Foundation of China (CCFZhipu202310).

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
5
7 A2. Did you discuss any potential risks of your work? There is no risk involved in our work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we aim to mine other potential causal features for this task and apply our model to other types of relation extraction tasks, such as temporal relation extraction. Limitations
In this paper, we only focus on whether or not there is a causal relationship between the given events, does not discriminate the specific cause/effect event. In addition, we only conduct research on sentence-level ECI, whereas document-level ECI often present more challenges. These are the focus of our future research