Discriminative Reasoning with Sparse Event Representation for Document-level Event-Event Relation Extraction
Year: 2023

Abstract:
Document-level Event-Event Relation Extraction (DERE) aims to extract relations between events in a document. It challenges conventional sentence-level task (SERE) with difficult long-text understanding. In this paper, we propose a novel DERE model (SENDIR) for better document-level reasoning. Different from existing works that build an event graph via linguistic tools, SENDIR does not require any prior knowledge. The basic idea is to discriminate event pairs in the same sentence or span multiple sentences by assuming their different information density: 1) low density in the document suggests sparse attention to skip irrelevant information. Our module 1 designs various types of attention for event representation learning to capture long-distance dependence. 2) High density in a sentence makes SERE relatively easy. Module 2 uses different weights to highlight the roles and contributions of intraand intersentential reasoning, which introduces supportive event pairs for joint modeling. Extensive experiments demonstrate great improvements in SENDIR and the effectiveness of various sparse attention for document-level representations. Codes will be released later.

heading: 1 Introduction
text: Event-Event Relation Extraction (ERE) is the task of identifying the relation between two events in texts. As shown in Figure 1, for any pair of events1, e.g., (Services, downtime), it shall make the classifications for which relation type it holds. Clearly, event pairs may be in the same sentence (SERE) (Kadowaki et al., 2019a; Liu et al., 2020; Kadowaki et al., 2019b), or scattered across the entire document (DERE) (Phu and Nguyen, 2021). In practice, DERE can benefit a wider range of applications, such as knowledge graph construction (Chen et al., 2019) and future event forecasting
∗Corresponding author 1Event is defined as the trigger word in this area.
(Hashimoto, 2019) but it remains challenging due to the difficulty in long-text understanding.
In this paper, we propose to improve the reasoning ability among events spanning the entire document for DERE. Different from conventional methods, which build an event graph based on linguistic tools (Phu and Nguyen, 2021; Gao et al., 2019; Xu et al., 2023; Zeng et al., 2021), we focus more on the nature of document itself and do not rely on any prior knowledge. To do this, we highlight the following key questions:
• How to capture events’ dependence that may be far away?
• Should we treat all event pairs equally considering the essential difference between SERE and DERE?
To address them, we propose a novel DERE model that learns Sparse EveNt representations for Discriminating Intra- and intersentential Reasoning, namely SENDIR. Inspired by MAE (He et al., 2022), we observe a different information density between sentences and documents — for an event, most parts of the document are irrelevant, leading to a low information density. By contrast, the sentence has a high density and
16222
usually contains related words as causal indicators. As shown in Figure 1, problem3
causal−−−−→ damage3 in the third sentence has clear causal word (due to)2, making the prediction much easier. While, for problem3
causal−−−−→ damage4 across the third and fourth sentences, there is no such pattern. This motivates us the design two modules as follows.
The goal of module 1 is to shorten the dependence distance among events to learn high-quality local and global context representations. The basic idea is to learn event-specific sentence embeddings as local features. Based on that, we further utilize sparse self-attention globally to skip irrelevant information. We have defined various types of attention masks to reflect specific dependence among sentences. In addition to conventional random sparse attention (Tay et al., 2021), we have also explored Narrative, Flashback, Global→, Global←, and Banded attention, to reflect specific language bias according to human writing habits. We name this module sparse event representation learning, while these sparse attentions shall also benefit other long-text understanding tasks.
Module 2 aims to discriminate intra- and intersentential reasoning to help difficult cross-sentence events with relatively easy within-sentence-level events. As shown in Figure 1, it is easy to predict (problem3
causal−−−−→ damage3) with high confidence, which is part of the path problem3
causal−−−−→ damage3
causal−−−−→ affected4 causal−−−−→ damage4 for prediction of (problem3
causal−−−−→ damage4). Thus, for each event pair, we take the outputs of module 1 as intra-sentential features. We then enhance them with selected supportive event pairs to constitute possible reasoning chain, and utilize Gated Attention Unit (GAU) (Hua et al., 2022) to conduct inter-sentential reasoning. Finally, we combine these two types of features with varying weights to differentiate their confidence and roles in ERE. Thus, we can improve the prediction of event pairs across sentences without hurting the performance of the pairs within a sentence.
We summarize the contributions as follows:
• We propose to discriminate intra- and intersentential reasoning considering the essential difference between SERE and DERE.
• We propose a novel DERE model SENDIR without any prior knowledge or external tools.
2We use subscript to denote the sentence index.
• Experimental results on three public datasets demonstrate the effectiveness of SENDIR. Further studies also verify our proposed sparse attentions and discriminative reasoning in long-text understanding.

heading: 2 Related Work

heading: 2.1 Sentence-level ERE
text: Early ERE methods focus on SERE and exploit various textual features to represent the relation, such as syntactic features (Venkatachalam et al., 2021; Ning et al., 2019), causal patterns (Riaz and Girju, 2010; Hidey and McKeown, 2016), and statistical features of causal information (Mirza et al., 2014; Hu et al., 2017; Tan et al., 2022). Following the success of pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019), recent works tend to enhance PLMs with external knowledge, so that SERE models can obtain high-quality contextualized event representations for classification. Hashimoto (2019) exploited the cause and effect entities in Wikipedia and the multilingual inter-wiki links as weak supervision. Zuo et al. (2021a) introduced external causal statements and adapted a contrastive transfer strategy to incorporate them into a target model. Cao et al. (2021) utilized ConceptNet (Speer et al., 2017) to learn latent structure of event causal relation, and Zuo et al. (2021b, 2020) designed a knowledge-guided method to generate new samples based on several knowledge sources, such as WordNet (Miller, 1998) and VerbNet (Schuler, 2005). Although the SERE has achieved great success, events usually scatter the entire document in real scenarios. Therefore, DERE has attracted more and more research attention.

heading: 2.2 Document-level ERE
text: Compared with SERE, DERE has a wider range of applications but is more challenging due to the difficult long-text understanding. Thus, researchers tend to consider event-event structures for global reasoning. Gao et al. (2019) used integer linear programming (ILP) to model causal information by designing constraints and modifying the objective function to encourage causal structures and discourage the opposite. To build the connections among the events, Phu and Nguyen (2021) designed various document-level graphs and used graph convolutional networks to learn structure-preserved features. Chen et al. (2022) proposed to build an event pair relational graph and converted DERE to
the node classification, which captures the global interactions and alleviates the spurious correlation between events. Man et al. (2022) proposes to model the important context sentences to identify relations. However, to capture long-distance information, these approaches typically construct an additional document-level graph to assist global reasoning. The graph introduces unnecessary noise and decreases efficiency. Instead, we explore the information in documents without requiring prior knowledge or external tools. Certainly, our method can be further improved by incorporating structural knowledge. We leave it in the future.

heading: 3 Methodology
text: SENDIR aims to learn high-quality event representations to facilitate both intra- and inter-sentential reasoning. As shown in Figure 2, our framework has four main components: Encoder to encode a document into vector, Sparse Event Representation Learning (SER) that further learns event representations based on document embeddings, Discriminating Intra- and inter sentential Reasoning (DIR) that conducts joint inference based on each pair of event representations, and Classification to make final predictions.

heading: 3.1 Encoder
text: We utilize the BERT (Devlin et al., 2019) with BiLSTM to encode the document for long documents (more than 512 tokens). Given a document D with n sentences and N events, D = [X1, X2, . . . , Xn],
and the sentence (Xi = [x1, x2, . . . , xl]) contains l words, the encoder is expressed as follows: {
H = Bi-LSTM([s1, . . . , si, . . . , sn]) si = BERT(Xi) , (1)
where H = [h1,h2, . . . ,hn∗l] is the embedding of token, andhi ∈ Rd. For event ei,p, where i denotes ith event and p denotes the index of sentence, we define its embeddings as ei,p = hk, if the event mention word is xk, the position of the event in the document is k.

heading: 3.2 Sparse Event Representation Learning
text: SER explores various types of attention to capture long-distance dependence between sentences for high-quality document representation, which will be used to enhance the event representation. In specific, SER first learns event-specific sentence embedding as the local context. Based on them, we then apply sparse self-attention to skip irrelevant information as global contexts. Particularly, we introduce various types of long-distance dependency assumptions. Finally, we define event representations based on local and global contexts. We highlight the following differences and advantages of SER from the previous document representation: (1) The global contexts taking sentences as nodes shortens the token-level distance between events. (2) Well-designed sparse attention mechanism will bring useful language bias that further alleviates the difficulty of modeling long-distance dependence. Event-specific Sentence Embedding. Given event and its sentence embeddings (i.e., ei,p and sp), we
use the event as a query and compute event-specific sentence embedding ci as follows:
{ wi = Softmax(spei,p) ci = wisp , (2)
where ci is defined as local context of event ei,p, and wi is the trainable parameters. Sparse Attention Mask. We apply selfattention technique to fuse information from the above event-specific sentence embeddings C1 = [c1, c2, . . . , cN ] ∈ RN×d as the global context. Note that N is the number of events rather than sentences, because each event has its own sentence embedding, even if two events are located in the same sentence.
Inspired by Child et al. (2019), we design the sparse mask matrix to deal with the long-distance dependency issue. We have designed six types of masks: Global→, Global←, Random, Banded, Narrative, and Flashback. Note that the assumptions behind them are not always true and we will give a discussion later. Their intuitive impression is shown in Figure 2. Formally, we define the mask matrix as G ∈ RN×N , where each element Gi,j ∈ {0, 1} denotes if the information of the jth event in the qth sentence ej,q can be seen by the ith event in the pth sentence ei,p. We define the following masks:
• Global→. We assume the events in the first several sentences (e.g., the first two sentences) are core topics of the document and should see all of the other events:
Gi,j = { 1, if p < 3, or q < 3 0, otherwise . (3)
• Global←. We assume the events in the last several sentences (e.g., the last two sentences) are conclusion topics of the document and should see all of the other events:
Gi,j = { 1, if p > N − 3, or q > N − 3 0, otherwise .
(4)
• Random. Random sparse masks are usually used to increase the ability of non-local interactions—we randomly sample 20% matrix element as 0, and others are 1.
• Banded. We assume that the related information is narrowed down into neighbor sentences only. That is, each event can only see
the events in neighbor sentences:
Gi,j = { 1, if |p− q| < 3 0, otherwise . (5)
• Narrative. We assume that the events are mostly described in narrative order, so that the former event can see the latter one:
Gi,j = { 1, if q − p > 0 0, otherwise . (6)
• Flashback. We assume that events are sequentially written, and thus the latter one should see the former one:
Gi,j = { 1, if p− q > 0 0, otherwise . (7)
Discussion of sparsity assumptions. Due to the complexity of language, we have designed the above six types of attention masks to capture different linguistic biases. Although these masks have patterns suitable for specific settings, they also have unsuitable cases where the underlying assumptions shall fail. However, all of them are designed based on our core assumption — the information density in documents is lower than that in sentences. Thus, we capture long-distance dependence via the sparse attention mechanism. In the experiment (Section 4.7), we demonstrate that even the random attention mask can improve document-level performance, and other types of attention masks (e.g., Narrative) have achieved further improvements by introducing additional language bias. Event Representation. Given local context C and attention mask G, we now use selfattention (Vaswani et al., 2017) to obtain global context, which is further combined with local context as event representations. The global context C′ = [c
′ 1, c ′ 2, . . . , c ′ N ] ∈ RN×d can be computed
as follows: C′ = αC (8)
whereα is the sparse self-attention matrix and each element is defined as follows:
   atti,j = (ciWi)(cjWj) T √ d ∗Gi,j αi,j =
exp(atti,j)∑ z∈N−1 exp(atti,z)
, (9)
where d is the dimension of hidden states for scaling, Wi and Wj are the trainable parameters. To capture different representation subspaces, we also use multi-head attention (Vaswani et al., 2017).
Now, we define the sparse contextualized event representation as follows:
e ′ i = ReLU([ei, ci, c ′ i]We), (10)
where We ∈ R3d×d is the trainable parameters, and e
′ i ∈ Rd. Given each pair of events (ei, ej), we
define their representation as follows3:
vi,j = ReLU([e ′ i + e ′ j , |e ′ i − e ′ j |]Wc), (11)
where Wc ∈ R2d×d is the trained parameters.

heading: 3.3 Discriminating Intra- and inter-sentential Reasoning
text: Section 3.2 defines event pair representations based on local and global contexts. In this section, DIR takes them as intra-sentential features, indicating that they have not considered the event pairs in other sentences to form a reasoning chain. To further obtain inter-sentential features for each pair of events, we first select supportive event pairs for each event pair and use GAU (Hua et al., 2022) for information fusion. Then, we combine two types of features with different weights to differentiate two types of reasoning — the relatively easy intrasentential tasks can help cross-sentence relation identification without the loss of performance.
First, instead of using all event pairs as supports, we assume that only the pairs sharing at least one common event can contribute to the reasoning chain. For example, given four events (a, b, c, d) to predict the relation between (a, c), the event pairs (a, b) and (b, c) are clearly related, but (b, d) is clearly irrelevant. By stacking more layers, our proposed model can implicitly deal with longer reasoning chains, as any length of chains can be decomposed into several shorter chains. Take the chain a −→ b −→ c −→ d for predicting event pair (a, d) as an example, we indeed will use a −→ c and c −→ d as supportive evidence, while a −→ c shall be supported as illustrated above. This is, the longer chain has been decomposed into two sub-chains, modeled using multiple layers, and optimized jointly. Based on the assumption, we build a set of supportive event pairs for query (ei, ej), T1 = [vi,j ,vi,1, . . . ,vN,j ]. Note that we include the representations of query event pair vi,j . Next, we utilize GAU to conduct reasoning over the sup-
3For clarity, we delete the index of the sentence.
portive set as follows:    T2 = (U AV)Wo U = T1Wu,V = T 1Wv,Z = T 1Wz
A = (ReLU((ZWq)(ZWk)T + b))2 ,
(12) where Wo,Wu,Wv,Wz,Wq,Wk, and b are the trainable parameters, and T2 = [v
′ i,j ,v ′ i,1, . . . ,v ′ N,j ] are the output
event pair representations enhanced by reasoning chains. We take v
′ i,j as the inter-sentential
reasoning features of query (ei, ej). Now, we are to combine two types of features with different weights. The basic idea is that event pairs within the same sentence is relatively easy to predict with high confidence, e.g., causal indicator words (e.g., due to or lead to) in Figure 1 provide clear patterns. We thus leverage intra-sentential features to facilitate the event pairs spanning different sentences. To avoid the harm of easier predictions, we assign higher weights to intra-sentential features if the event pair is within the same sentence. By contrast, we assign higher weights to inter-sentential features for events from different sentences to highlight the inter-sentential reasoning. Finally, the query event pair representations for relation between (ei, ej) are defined as follows:
o =    vi,j + β1v ′ i,j , if p− q < 0( −−−→ inter) vi,j + β2v ′ i,j , if p− q = 0(intra)
vi,j + β3v ′ i,j , if p− q > 0(
←−−− inter)
,
(13) where β1, β2, β3 are the weights that highlight different type of features for different event distribution. We determine these hyper-parameters by the heuristic experiments. Note that we separate the two cases of events in different sentences: −−−→ inter and ←−−− inter. −−−→ inter indicates that the event pairs are located in narrative order, while ←−−− inter denotes a flashback order. The separation considers the different attention assumptions in Section 3.2, which captures various language biases. In experiments, we indeed found such a bias that ←−−− inter has a high probability of negative relation, we thus set β3 = 0 and β1 = 0.8, β2 = 0.2 heuristically, but this may be varying in different scenarios.

heading: 3.4 Classification
text: Given the final representation of an event pair, we use the linear function to predict relations as follows:
ŷ = σ(oW + b), (14)
where W and b are the trainable parameters, ŷ is the probability of being positive, and σ is an activation function. For training, we adopt crossentropy as the loss function: L = −∑ei,ej (1 − y) log(1− ŷ) + y log(ŷ) (y is the golden). We use dropout to prevent overfitting.

heading: 4 Experiments

heading: 4.1 Datasets and Metrics
text: To demonstrate the performance of our model, we evaluate the model on two domains three datasets. EventStoryLine4 (Mostafazadeh et al., 2016) and Causal-TimeBank5 (Mirza and Tonelli, 2014) are event causal relation extraction (RE) dataset, and MATRES6 (Ning et al., 2018) is event temporal RE dataset. And we use Precision (P), Recall (R), and F1-score (F1) as evaluation metrics.
EventStoryLine annotates 258 documents, 22 topics, 4, 316 sentences, 5, 334 event mentions, 7, 805 intra-sentential event pairs, and 46, 521 intersentential event pairs. Following (Gao et al., 2019), we put them in order based on their topic IDs. Causal-TimeBank (Causal-TB) annotates 184 documents, 6, 813 events, and 7, 608 event pairs. MATRES annotates 275 documents for four temporal relations, i.e., BEFORE, AFTER, EQUAL, and VAGUE.
4https://github.com/tommasoc80/EventStoryLine 5https://github.com/paramitamirza/CATENA/tree/master/data 6https://github.com/qiangning/MATRES

heading: 4.2 Parameter Settings
text: We choose the most widely used BERT-base as basic PLMs, to avoid exhaustive parameter tuning. The learning rate is set to 1e−5 for pretraining and 1e−4 for others. We optimize our model with AdamW. We conduct the grid search to tune hyperparameters: the size of embedding is in {64; 128; 256; 512;768}, where bold font denotes the best setup. The batch size for pre-trained model is set to 2. The dropout rate is set to 0.4.

heading: 4.3 Baseline
text: We compare SENDIR with state-of-the-art methods for Event Causal RE and Event Temporal RE. Event Causal RE. (1) KMMG (Liu et al., 2020) that proposes a mention masking generalization and use external knowledge databases; (2) KnowDis (Zuo et al., 2020) that investigates a data augmentation to solve the data lacking; (3) LSIN (Cao et al., 2021) that employ ConceptNet to capture the latent causal relational structure; (4) LearnDA (Zuo et al., 2021b) that augments data to solve the data lacking; (5) LR+ and LIP (Gao et al., 2019) that models rich causal structures via designing constraints and objection function; (6) RichGCN (Phu and Nguyen, 2021) that builds multi-level graphs to capture structure-preserved features; (6) ERGO (Chen et al., 2022) that designs an event relational graph and converts the event causal identify to a node classification framework. Event Temporal RE. (1) DEER (Han et al., 2020) that constructs many training samples to simulate the machine reading comprehension for event temporal understanding; (2) SMTL (Ballesteros et al.,
2020) that extract the important text based on event pairs to identify relations. (3) TIMERS (Mathur et al., 2021) that uses rhetorical discourse features, temporal arguments, and syntactic features to extract the relation information. (4) SCS-EERE (Man et al., 2022) that seeks to identify the most important context sentences to identify the temporal relation.

heading: 4.4 Overall Performance
text: Table 1 and 2 show the overall performance on EventStoryLine, Causal-TB, and MATRES, respectively. We can see that: (1) SENDIR achieves better F1 scores on EventStoryLine and CausalTB, it also has a competitive result on MATRES, which demonstrates the effectiveness and generalization ability of our model. (2) On the MATRES, SENDIR is slightly lower than SCS-EERE. Because event temporal RE is particularly sensitive to direction between events. (3) All models perform better on intra-sentence than inter-sentence in Table 1. This is consistent with our claim that intra-sentence is easier to identify. (4) Particularly, SENDIR has much higher precision on intrasentence. Because the discriminative reasoning scheme alleviates the negative impacts of more difficult cross-sentence reasoning. (5) On intersentence setting, the improvements are mainly from higher recall. We attribute this to the enhanced long-distance modeling ability and the supportive query set — it tends to find relation clues from broader contexts and other event pairs.

heading: 4.5 Ablation Study
text: To further analyze SENDIR, we also conduct an ablation analysis to illustrate the effectiveness of our main modules. We show the results of the
ablation study in Table 37. SER. We examined the impacts of SER in Table 3. w/o Sparse Att, w/o Event Repre, and w/o SER denote that we gradually remove the key designs in Section 3.2: remove sparse attention mask, use local context only as event representations, and remove the entire SER module. (1) w/o SER. The performance becomes sharply poor without SER, especially on intra-sentence. Specifically, the experimental results are reduced by 5.3%/2.2%/2.4% F1 on intra-sentence, intersentence, and intra+inter. This demonstrates that SER can capture high-quality document representation, and intra-sentence event pairs rely more on high quality event representation, so intra-sentence drops more. (2) w/o Sparse Att. The experimental results are reduced. Specifically, the sparse attention mask has a more significant impact on intrasentence, and this sparsity is the key to improving the quality of the document representation. We will do further analysis of multiple attention masks in Section 4.7. (3) w/o Event Repre. The main performance is the decline in inter-sentence, while intra-sentence is even slightly up. We attribute the reason that the inter-sentence relies more on global context and the intra-sentence relies more on local context. Moreover, the number of inter-sentence is much more than intra-sentence. Therefore, when inter-sentence drops, results of inter+inter drops, even though intra-sentence arises.
DIR. In Table 3, we can also observe the following insights. Note that w/o DIR, w/o Selection, and w/o Weight denote the deletion of our entire DIR module, taking all other event pairs as supports, and regarding intra- and inter-sentential reasoning the same by using the same weight 1. (1) w/o DIR. Compared with SER, removed DIR mainly brings a
7Intra and inter indicate intra-sentence and inter-sentence setting, respectively.
0.0 0.2 0.4 0.6 0.8 1.0 Weight(Inter)
0.0
0.2
0.4
0.6
0.8 1.0 W ei gh t(I nt ra )
Intra+Inter (F1)
5 × 101
Figure 3: Impacts of the weight (β) on the EventStoryLine. Weight (Inter) is the weight of −−−→
inter.
decrease on inter-sentence, which is consistent with the method’s motivation and validates the DIR’s effectiveness. (2) w/o Selection. The experimental results are reduced. The major reason is that considering all event pairs as candidates will increase the noisy information. (3) w/o Weight. The drop on intra-sentence is very significant. Because of the lack of distinction in cross-sentence reasoning, the over-reasoning of simple event pairs leads to poor results. And using different weights facilitates simple tasks and global enhancements. We will follow up with a detailed discussion in Section 4.6.

heading: 4.6 Effects of Weights
text: As shown in Figure 3, we show the performance of Intra+Inter with different values for β1 ( −−−→ inter) and β2 (intra). The darker the color, the higher the F1 value. We did not include β3 due to the serious bias in datasets (β3 = 0 always leads to better results), which makes further investigation trivial. The darker the color, the higher the F1 value. We can find that: (1) The scores in the bottom right part is better than in other parts, where the weight of inter-sentence increases and the weight of intra-sentence decreases. This agrees with our assumption that inter-sentential event pairs rely more on cross-sentence reasoning than intra-sentential event pairs. (2) In general, the performance of the right part is better than the left. There are two reasons: first, as the weight of inter-sentence increases, we highlight more cross-sentence reasoning, which improves the performance of intersentence; second, there are far more inter-sentence than intra-sentence, and the improvements of intersentential results will significantly improve the
66.2 67
60.9
66 66.5 66.9
48.3 46.9 46.5 47.2 46.5 46.4
51.9 51.3 49.3
50.8 50.5 50.4
45
50
55
60
65
70
Order Reverse Global--> Global<-- Random Band
F1 (
%)
Sparse Graphs
Intra Inter Intra+Inter 66.2 67
60.9
66 66.5 66.9
48.3 46.9 46.5 47.2 46.5 46.4
51.9 51.3 49.3
50.8 50.5 50.4
45
50
55
60
65
70
Narrative Flashback Global--> Global<-- Random Banded
F1 (%
)
Sparse Attention Masks
Intra Inter Intra+Inter
Figure 4: Impacts of Sparse Attention Mask on the EventStoryLine.
overall experimental performance. (3) The subdiagonal elements indicate that intra-sentence and inter-sentence have the same weight. Clearly, the performance is unsatisfactory. The reason is that inter-sentential event pairs are usually difficult to predict and have lower confidence than intrasentential event pairs. Thus, treating them equally negatively impacts the easier sentence-level tasks.

heading: 4.7 Effects of Sparse Attention Mask
text: To investigate the impacts of various sparse attention masks on the SER, we report the results using different sparse attention masks: Narrative, Flashback, Global→, Global←, Random, and Banded. From the Figure 4, we can find that: (1) On intrasentence, these sparse attention masks have similar results except for Global→. This result is consistent with the previous results that event pairs rely more on local contexts rather than long-distance dependence on global contexts. (2) Random is unexpectedly good, indicating there is numerous redundant information in the document, and the sparse mask matrix can mitigate the effect of noise. (3) Narrative has achieved the best performance, which reflects a language bias from human writing habits — always talk about the main topic at first.

heading: 4.8 Case Study
text: As shown in Figure 5, we present a case study to better understand the effect of SENDIR compared to the baseline model (i.e., BERT). We can notice that: BERT and SENDIR can identify the intra-sentential event pair (dead causal−−−−→6.1 magnitude quake) and some easy inter-sentential event pairs (dead causal−−−−→6.1 magnitude earthquake and 6.1magnitude quake
causal−−−−→ rescue). However, for some complex event pairs that span multiple sentences, SENDIR can employ DIR to infer
6-0-dead 35-0-6.1. 13-3-quake 22-6-quake 7-2-6.1
6-35 6-13 6-22 6-7
35-0-6.1. 15-4-killed 33-2-rescue 17-5-killed 29-6-lanslides 8-2-dead 18-5-injured 23-6-evacuated 19-5-damage 12-2-injured
[0] Indonesia earthquake: 24 dead and over 200 injured as 6.1
magnitude quake hits Aceh province … [2] Soldiers and police are
leading rescue operations in Indonesia after a 6.1 magnitude
earthquake hit Aceh province leaving 24 dead and over 200 injured.
[3] The quake struck yesterday afternoon …[4] Twelve people were
killed and 70 others were injured … [5] … 10 people were killed,
140 were injured and about 1500 houses and buildings …
Event Pair
dead 0, 6 . 1 magnitude quake 0
dead 0, quake 3
dead 0, 6 . 1 magnitude earthquake 2
6 . 1 magnitude quake 0, rescue 2
6 . 1 magnitude quake 0, killed 4
6 . 1 magnitude quake 0, injured 5
…
GT
YES
YES
YES
YES
YES
YES
…
BERT
…
SENDIR
…
Figure 5: The case study of our proposed SENDIR and BERT models on EventStoryLine, where GT denotes ground truth. Red numbers are the sentence numbers.
causal relations, but BERT fails, such as dead causal−−−−→ quake and 6.1 magnitude quake causal−−−−→ injured. These demonstrate the significant advantages of SENDIR in dealing with inter-sentential event pairs to identify hard-to-identify causal relations by cross-sentence reasoning.

heading: 5 Conclusion
text: In this paper, we exploit a novel Discriminative Reasoning with Sparse Event Representation for DERE. It can learn high-quality event representation and facilitate inter-sentential reasoning for document-level understanding. Experimental results show that our method is effective and significantly better than competitive baselines, improving inter-sentence cases without harming intrasentence event pairs. The extensive analysis also provides interesting insights about various language biases for sparse long-text representation learning. In the future, we will combine various sparse assumptions for high-quality document representations and incorporate graph reasoning.

heading: 6 Limitation
text: The limitations of SENDIR include the following two points: (1) It has not extended to documentlevel entity-centric relations tasks. Our work is event-centric, and future work extends it with entity-centric cases. Document-level entity-centric RE needs to consider multiple mentions of an entity and different relations in different directions of the
same entity pair. (2) It does not bring in external commonsense knowledge. Knowledge can be used to enrich events and improve the accurate ERE.

heading: Acknowledgements
text: We appreciate the comments from anonymous reviewers which will help further improve our work.This study is supported by National Natural Science Foundation of China (No.U19B2020), Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant, and Beijing Institute of Technology Southeast Academy of Information Technology.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6
3 A2. Did you discuss any potential risks of your work? Section 5 and Section 6
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Our work is event-centric, and future work extends it with entity-centric cases. Document-level entity-centric RE needs to consider multiple mentions of an entity and different relations in different directions of the
same entity pair. (2) It does not bring in external commonsense knowledge. Knowledge can be used to enrich events and improve the accurate ERE