EDeR: Towards Understanding Dependency Relations Between Events
Authors: Ruiqi Li
Patrik Haslum
Leyang Cui
Venue: EMNLP 2023 Main (2023)
Keywords: dataset
event dependency relation
TL;DR: The paper proposes a dataset for exploring and understanding the dependency relations between events.

Abstract: Relation extraction is a crucial task in natural language processing (NLP) and information retrieval (IR). Previous work on event relation extraction mainly focuses on hierarchical, temporal and causal relations. Such relationships consider two events to be independent in terms of syntax and semantics, but they fail to recognize the interdependence between events. To bridge this gap, we introduce a human-annotated Event Dependency Relation dataset (EDeR). The annotation is done on a sample of documents from the OntoNotes dataset, which has the additional benefit that it integrates with existing, orthogonal, annotations of this dataset. We investigate baseline approaches for EDeR's event dependency relation prediction. We show that recognizing such event dependency relations can further benefit critical NLP tasks, including semantic role labelling and co-reference resolution.

EDeR: Towards Understanding Dependency Relations Between Events
Ruiqi Li Patrik Haslum
Leyang Cui
Australian National University
o Tencent Al lab
(ruiqi.li, hat.tummm@auu.u.....c leyangcui tencent.com
Abstract
[He] e1:tried [to e2:forgive [his father) [one time]]. [His father] e3:worked [in Guyuan, about 40 to 50 kilomaters away from home]. [If [nathing special] e4:happens]. [he] [will] [nat] e5:come [back]. e6: Learned [that [he] was [very sick]], [he] e8:ran [back to the villagee [right away]. [After [he] e9:arrived [at the village]], [he] did [not] e 010:say [anything]. [He] e11:put [him] [on his back] and o 012:carried [him] [to the town] [to e13:see (a doctor]).
Relation extraction is crucial task in natu- ral language processing (NLP) and informa- tion retrieval (IR). Previous work event re- lation extraction mainly focuses on hierarchi- cal, temporal and causal relations. Such rela- tionships consider two events to be indepen- dent in terms of syntax and semantics, but they fail to recognize the interdependence between events. To bridge this gap, we introduce annattted Event Dependency Relation dataset (EDeR). The annotation done a sample of documents from the OntoNotes dataset, which has the additional benefit that it integrates with existing, orthogonal, annota- tions of this dataset. We investigate baseline approaches for EDeR's event dependency rela- tion prediction. We show that recognizing such event dependency relations can further bene- fit critical NLP tasks, including semantic role labelling and co-reference resolution.
e1 required
required
e2
e6
o7
argument:
optional
e13
condition
Independent
non-argument: e5
e4 e10
e9
refined event: e5: [he] [will] [not] come [back].
e10: [he] [not] say [anything).
Figure I: Examples of the event dependency relations.
Above: Source text. (Event predicates marked in boldface and argument spans are marked with brackets.) Event dependency relations refined events.
Below:
1 Introduction
to be independent in terms of syntax and semantics but overlook the inter-dependence between them. This disregard for interdependence can potentially yield incomplete or erroneous narrative interpreta- tions. Take the sentence "He tried to forgive his father one time." from Figure 1, c2 with the predi- cate "forgive" is the action that he "tried" (el) to take. However, there is no temporal or causal or hierarchical relation between e2 and el; rather, e2 is the object (syntactically depending) and patient (semantically depending) of el. In fact, e2 didn't 't actually happen. Recognizing such relations, that reveal both semantic and syntactic dependencies, is an essential, yet challenging task for AI researchers. Correctly identifying these dependency relations can enhance language understanding and provide valuable information for various related NLP tasks.
Events play critical role in enabling AI agents to understand and perceive the world, as they provide information on what happened and the entities in- volved. An event is composed of a predicate (i.e., verb) and arguments, where the predicate indicates the event's action and the arguments represent the subject, object and so on of the predicate (Levin et al., 1999; Rappaport Hovav et al., 2010).
Previous work on relation extraction mainly fo- cuses investigating the relationships between entities (Miwa and Sasaki, 2014; Huguet Cabot and Navigli, 2021; Chen et al., 2020; Ma et al., 2022) rather than events. In the relatively lim- ited research work that studies the relations be- tween events, the types of relations considered are causal (Mariko et al., 2020, 2022; Tan et al., 2022), temporal (Bethard, 2013; Laokulrat et al., 2013), and hierarchical (Hovy al., 2013; Glavaš and Šna- jder, 2014). Such relationships consider two events
Motivated by these, investigate how event may be an argument of another as representing the event dependency relations, rather than being regarded as independent. Additionally, we dis- tinguish argument into required argument and
Corresponding author.

---

optional argument events. Required arguments, like e2 ("forgive") in el ("tried")) from Figure are present for an event to be complete and mean- ingful (to "try to", one has to try to do something). Optional arguments, such as e13 ("see") in c12 ("carried")), enhance or clarify the event that it is argument of (for example indicating the pur- pose), and their occurrence isn't explicitly stated. Non-argument events, which conjunctive or conditional, are often mislabelled as arguments in the event extraction task, like e9 ("arrived"), tem- poral modifier for e10 ("did not [say] anything"), but is independent.
datasets
reps.
relations
ver Perbagg.cm span
ECB (Bejan and Harabagiu, 2008)
X
H
Hieve (Glavaž ct al., 2014)
X
X
H
IC (Araki et al., 2014)
X
X
H
TimeBank (Pustejovsky et al., 2003)
X
X
T
TB- TB-Dense (Cassidy et al., 2014)
X
T T
MATRES (Ning et al., 2018b)
X
X
ECD (Do et al., 2011)
X
CA
CiRA (Fischbach al., 2021)
X
X
J
CA
CNC (Tan al., 2022)
X
V
CA
FCR (Yang al., 2022)
X
V
CA
CATENA (Mirza and Tonelli, 2016)
X
CA+T
ESC (Caselli and Vossen, 2017)
X
CA+T
TCR (Ning al., 2018a)
X
CA+T
EDeR (Ours)
CO+D
Table I: Comparisons of event representations (reps.) and relations between our EDeR dataset and other public event relation datasets. "arg." is the abbreviation of argument labels. In the column "relations", "H", "T", "CA". "CO" and "D" represent hierarchical, temporal, causal, conditional and dependency, respectively.
Argument-event dependency relations are fre- quent in natural language texts, especially in nar- rative texts such as news articles, conversations, and similar. Hence, we present human-annotated Event Dependency Relation dataset (EDeR) that (1) extracts event dependency information based on a sample of 275 documents spanning seven genres from OntoNotes (Pradhan et al.. 2013) and integrates with orthogonal annotations of this dataset, and (2) provides refined semantic role- labelled event representations based on this infor- mation. We build the EDeR dataset with 1 1,852 high-quality annotations, according to the proposed event relation taxonomy. can serve as the basis for an effective automatic classification process.
semantic role labelling (SRL) which marks roles (i.e., predicates and arguments) of words or word spans in sentences. The OntoNotes dataset uses the PropBank annotation schema (Bonial al., 2012) for SRL, categorizing arguments as text spans, as- signing numbered arguments (ARGO-ARG5), and labelling verb modifiers like purpose (PRP) and location (LOC). Examples are shown in Figure 2. However, as Table shows, most of the public event relation datasets, unlike EDeR, solely repre- sent events text verbs spans without identify- ing arguments and argument labels. This could re- sult in considerable information loss and increased ambiguity in the process of understanding an event.
We apply both EDeR -uman-annotated and moeecepreddcted event dependency relations to enhance a state-of-the-art semantic role labeling (SRL) model (Zhang al., 2022)'s event repre- sentation generation. Experimental results prove these relations improve event extraction, yield- ing updated representations from EDeR. These re- fined event representations were further applied to the co-reference resolution (CR) task. A per- formance comparison taking the original and our updated event representations as inputs a CR model (Jiang and Cohn, 2021) affirms the effec- tiveness and validity of EDeR's 's refined represen- tations. We release EDeR (e.g., dataset, base- line code, and model) at https://githhi com/ Ri RichieLeeg3/EDeR.
We find that events can be arguments of other events, though not all events within another's span are arguments. This discrepancy can lead to shared subjects objects without signifying an argument relationship. For instance, in Figure 2, the event "The man [who] works here" is contained in the event "The man who works here tells me to get out" but not an argument of "tells". These annotations in the SRL datasets such as OntoNotes overlook the dependency relationship between events, but clarifying this relationship allows for more fo- cused argument span selection, such as "The man" for ARGO (agent) of the event with the predicate "tells" in the example above.
2 Background and Related Work
2.1 Event and Event Representation
Event mentions involve a predicate (usually verb or phrasal verb) and a set of labelled argu- ments (Levin et al., 1999; Rappaport Hovav et al., 2010). Events can be identified from texts using
2.2 Relations Between Events
Several event-event relations have been proposed in recent. The summary of varied relations the

---

existing datasets is shown in Table I's last col- umn. The TimeBank (Pustejovsky al., 2003), TB-Dense (Cassidy et al.. 2014) and MATRES (Ning et al., 201 8b) datasets focus on the tempo- ral relationship which reveals one event happens BEFORE, AFTER, etc. another. Logical relation- ships like causality are also explored in datasets like ECD (Do et al., 2011), 1), ESC (Caselli and Vossen, 2017), CiRA (Fischbach et al., 2021) and CNC (Tan et al., 2022). Datasets like CATENA (Mirza and Tonelli, 2016), ESC (Caselli and Vossen, 2017), and TCR (Ning et al., 2018a) individually detect temporal and causal relations based on identical events. Such relationships regard two events as syntactically and semantically independent the occurrence and actuality of each event are isolated and not affected by others. Besides, several datasets propose hierarchical relationships with sub-events satisfying temporal and spatial conditions within a super-event (Hovy et al., 2013; Glavaš al., 2014; Glavaš and Snajder, 2014). Bejan and Harabagiu (2008) and Araki et al. (2014) explore event co- reference relations whether two event mentions refer to the same event. Such hierarchical relations cannot deal with cases when event (especially the verb of the event) requires clausal comple- ment, i.e., an argument of the event verb is itself an event.
OntoNotes SRL
The man who
works here talls
me
get
the
hell out
(ARG)"
(AR62")
JARG1*
(ARG1")
(V) (ARG2
(ARGU (R-RDO"")
4,0c%
Annotation
Sentence:
The man who worts hore talls me to get the hell out.
Event 1: The man who works here (V: talss me to get the hall out Event 2: (V: gel) hell out
Question: Event
required argument. xan optionel argument condition independent
Sentence: man works hare tals to get out
Event 1: The man who works here (Vitals) me to get the hell out
Event 2: The man who (V. warks) here
Question: Evant
xa required argument x an optional argument X condition of independent Please update
(V:
het
Figure 2: Above:
A sample sentence with semantic role labels from the OntoNotes dataset. Below: Correspond- ing event pairs presented for human annotation with event dependency relations.
in A.1 Table 6. The number of sampled documents and the separation of them into training, develop- ment and test sets under each genre follows their initial distributions in the OntoNotes dataset.
3.1.1 Candidate Event Pair Extraction
Because arguments are spans of text, part or all of an extracted event may lie within the argument of another event. If the verb (i.e., predicate) of ej is within an argument of 6;, we say Gj is contained in e,, This can be nested. Contained events are candidates for being argument events, but are not necessarily so. For example, in the top part of Figure 2, the event in green (e3 (ARGO: The man {R-ARGO: who {V: works } {ARGM- LOC: here }) is contained in the span of the event in blue (el = {ARGO: The man who works here } {V: tells } {ARG2: me } {ARGI: to get the hell out }), but e3 is not an argument of el: however, the event in orange (e2 {ARGI: me } (V: get) {ARG2: the hell out }) is argument of el.
3 Dataset
We utilize subset of OntoNotes documents with its human-annotated and medicate-argument for- matted events to extract candidate event pairs, where one event's predicate is contained in the span of another. These pairs, denoted as Eventl and Event2, are labelled by human annotators to in- dicate whether Event2 is a required argument of, optional argument of, condition of, inde- pendent from Event1. We next detail the way collect and pre-process the candidate event pairs and the human annotation process and construction the datasct.
We select event pairs (e,, eÿ) where ej 's verb is contained within the span of an argument e as candidate event pairs for annotation.
3.1.2 Preprocessing
We apply three filters to the selected candidate event pairs. First, we filter out candidate event pairs (o,, Cj) in which ej does not have any argu- ments. This typically occurs when ej is modal verb indicating the tense of the verb in e;. Second, events in the OntoNotes dataset sometimes mistake adjectives for verbs (e.g., "given" in "at any given time"), so we use the words' POS tags to filter these out. The POS tags of the event predicates are
3.1 Data Collection
OntoNotes contains semantic molefformatted event representations, the OntoNotes example in Fig- ure 2 shows. We randomly sampled 275 documents from seven genres: broadcast news (bn), magazine (mz), newswire (nw), pivot corpus (pt), telephone conversation (tc), broadcast conversation (bc), and web data (wb). Detailed data statistics are shown

---

Candidate Annotators
Annotators
3.2.2 Annotation Procedure
ScRORS
We adopt multi-level qualification- based annota- tion procedure in which annotators' work is sam- pled and inspected/correeted in several stages, and feedback from the inspections is passed back to the annotators. A schematic of the procedure shown in Figure 3. Compared with the commonly-usee crowd-sourcing and voting strategy, this procedure makes annotators learn to improve throughout the task. Although in the end not all event pairs have been annotated by multiple participants, the strict qualification tests used along the way ensure anno- tators give their best answers.
Quality inspectors
Annotation
40% sampled
Screening
results
Qualifeed?
Samples annotations
block
15-25% sompled
Annotation
results
Linguistic Expert
Authors
Samples annotations
Samples annotations
Over 80 candidate annotators were recruited from colleges in China through data annotation agency. They are English linguistic- graduate students and fluent in English (passed the TEM-8 test!). After reading the instructions, candidates took screening test, requiring them to annotate 50 examples, also annotated by authors, and cross- checked with the linguistic expert. The linguistic expert who has rich NLP-related annotation experi- ence was hired via the same data annotation agency. 16 candidates who answered at least 85% of these questions correctly in the test were selected as an- notators. Among the 16 annotators, the three with the highest accuracy were selected quality in- spectors (QIs). After further training, in which the linguistic expert explained the instructions and their mis-annotated cases from the screening test, the annotators started the annotation.
Final annotation results
Figure Illustration of our multi- qualification- based annotation procedure.
obtained using the Stanford CoreNLP toolkit (Man- ning et al., 2014). Third, remove transitively contained event pairs, i.e., pairs (e,, eÿ) such that there exists another event 6 such that the verb of Gg is contained in 6 and the verb of ey is contained in Eg. In such cases, (e,, ex) and (ek, Eÿ) may be candidate pairs, but (es, eÿ) is not. For example. in "[She] (V: try } [to (V: stop [[them] from {V: ruining } [...]]]"), "try" and "stop", and "stop" and "ruin" are two candidate pairs, but "try" and "ruin" are not.
The event pairs for annotation were evenly split into subsets and sequentially released to the an- notators in stages, each one week apart. each stage, 40% of the annotators' submitted cases were randomly chosen and annotated by QIs. The expert reviewed 15-25% of QIs' inspected cases, provided feedback on errors, and offered explanations to pre- vent repetition in subsequent stages. This expert su- pervision ensures QIs' quality and prevents unreli- able labels that could mislead annotators. After the linguistic expert's 's review of the Qls' annotations, if the agreement between an annotator's answers and the corresponding QI's lower than 95, the an- notator must re-annotate inconsistencies until this agreement ratio is met.
3.2 Annotation
3.2.1
Annotation Instruction
We present each candidate pair with the whole span of both events and highlight the predicate of each event with "(V: }", shown in the lower part of Figure 2. Annotators can choose one of four options answer the relation between the two events. For the annotation task, we separate the non-argument case into two: condition and inde- pendent ("neither an argument nor a condition"). This made the definitions of the labels easier to understand, since condition is also, like argument events, a hypothetical, rather than actual, event in the text. Identifying conditional statements in texts is itself an interesting and active topic of research (Fischbach et al., 2021; Tan et al., 2022), motivated in particular by their indication of causality. The full annotation instructions can be found in A.4.
The authors also monitor the quality of the an- notation. In each stage, we sampled and labelled
'Test for English Majors-hand 8, the highest level test in China for English major students that measures the overall English proficiency.

---

5% of the once-only annotated results from each annotator. If the accuracy achieved by an annota- tor is below 85%, the work of that annotator was discarded, and the annotator was dismissed from subsequent stages. Only one annotator failed this test and was removed after the first stage. Not counting the author-inspected cases, 4,771 (40.2% of all) cases received at least four times annotation.
pairs labelled as non-argument (condition or inde- pendent), 2,490 distinct events' representations are refined by the annotators and by our automated method.
argument required optional
non-argument
overall
cond.
indep.
train dev
4,096
2,837
335
1,861
9129 (77%)
635
421
41
355
1,452 (12.3%)
test
594
368
70
2399
1271 (10.7%)
3.2.3 Event Representation Refinement
overall
5,325
3,626
446
2,455
11,852
(44.9%)
(30.6%)
(3.8%)
(20.7%)
As described in the second annotation example in Figure 2, one event (predicate) can be contained in another event but actually not an argument of it. For the event pairs that annotators label as inde- pendent, further ask them refine containing event ("Event 1") by removing the span of the con- tained event ("Event 2") from it. However, the contained event often shares the subject or object with the containing event, so annotators will keep any shared parts in Event and remove the only remaining Event spans. This manual event repre- sentation refinement proceeds simultaneously with the amnotation of the event dependency relation and is inspected by the QIs and the linguistic expert as well.
Table Distribution event pairs in the annotated EDeR dataset across labels and across the training, de- velopment and test subsets.
For aanan--ndntttedd datasets, there is always a trade-off between the number of instances being annotated and the quality of annotations (Krysein- ski et al., 2019; Cui et al., 2020). The size of our dataset is constrained by the annotation method. But it is comparable with or larger than many other cumman-annotatdd event relations rea- soning datasets, e.g., Glavaš et al. (2014); Cassidy et al. (2014); Mirza and Tonelli (2016); Ning et al. (201 8b); Tan et al. (2022). We conducted various kinds of inspections to ensure the quality of EDeR. The statistics are as follows: (1) Among the 4,771 samples (40.2% of the total 11,852 cases) from EDeR that have received the three Quality Inspec- tors' annotations, only 31 (0.65%) of them received three different labels from the three QIs; (2) we ran- domly sampled 417 cases that received at least four annotations; the accuracy of the annotated labels is 90.65% (378/417) 7); (3) a 5.2% sample of the over- all annotations was double-checkee by the authors, with larger than 88% accuracy. These results show that the multi-level iterative annotation procedure produces high-quality annotation results.
Events tagged as conditions typically don't 't share subjects or objects with their enclosing events. We found automatic revision of the enclosing event feasible, hence didn't 't request annotators for it. If the conditional event ej and its containing event e; appear in the sentence as one of the subsequences (1) & ej (2) ej & or (3) Cj. Ci, where is one of the signal words/phrases "if", "wheneeer".. "as long as", "on [the] condition that", "unless", or "provided that", then we first remove all words in e; that are within the span of ej. and, second, remove any signal word/phrase that is antecedent and adjacent to ej. In the few cases when no signal word or phrase was detected in the second step, we manually checked and revised the containing event.
Experiment: Event Dependency Relation Prediction
3.3 Analysis
Table 2 shows the distribution of event pairs clas- sified under each label in the annotated dataset, as well as the number of event pairs in the training. development and test document subsets.
Given two events & and ey from a sentence X, the basic task is to predict whether éj is an argu- ment of e; or not. In this section, we evaluate the performance of several baseline methods on this task. The best achieves accuracy of just over 82%. In Section 5, we show that even this binary classification of events into dependent and indepen- dent sufficient improve performance in two further NLP tasks: event representation extraction and co-reference resolution.
It is not surprising that the distribution is bi- ased, with majority (just over 75%) of event pairs labelled "argument". The predicate-within- tetaiiing--vvnt-speee relation between events in the pairs we selected for annotation is a necessary condition for them to be dependent. Of the 2,901

---

The task can be refined into 3-way classifica- tion task, by distinguishing required and optional arguments, and into a 4-way classification task by distinguishing also the two types of con-argument events: conditions and independent events. The 3- and 4-way classification tasks are significantly harder. A summary of results is presented in Sec- tion 6, and full details in Appendix A.2.
label of the argument where the contained event predicate in the containing event. We also add a special token "[SRL]" for model recognition. E.g., "The man who works here V: tells me to get the hell out. [SEP] me V: get the hell out [SRL.] ARGI." Crent-Event-SRL--EE Based on the Lvvent- Event-SRL Input, we further add the syntactic de- pendency relation information of the two event predicates. A special token "IDEPI" is added. For instance, "The man who works here V: tells me to get the hell out. [SEP] me V: get the hell out [SRL] ARGI [DEP] xcomp."
4.1 Baseline Models
We evaluate several pre-trained language models, and rule-based heuristic method.
The language models include discriminative models BERT (Devlin et al., 2019), Distil- BERT (Sanh et al., 2019), XLNet (Yang et al., 2019), RoBERTa (Zhuang et al., 2021); as well as two auto-regressive generative models GPT-2 (Rad- ford et al., 2019) and ChatGPT (i.e., GPT-3.5- turbo). Specifically, we feed ChatGPT four exam- ples from the annotation instruction, representing each of the four relations used in the annotation as the prompt, along with the ground truth dependency labels, employing "fow-shot learning" approach. The implementation settings can be found in A.3.
Marked-Predicat Sentence Inspired by the success of MarkedBERT (Boualili et al., 2020), we add special marks to emphasize the predicate to- kens in the sentence. For example, the sentence "The man who works here tells me to get the hell out." with event predicates "tells" and "get" be- comes "The who works here [V1] tells [VVI] me [V2] get [VV2] the hell out."'
4.3 Results and Analysis
Table 3 shows different models' performance based on different inputs. The heuristic rules-based method achieves the highest precision of 97.67% %, yet the recall is considerably low at only 34.93%. We observe that adding the event predicate's 's syn- tactic dependency and the (semantic) argument label information improves the performance of most of the models. However, using the marked- predicate single sentence as input is more effective than these event-event style inputs. Taking the marked-predicate sentence as input, the BERT and RoBERTa models outperform others.
We also design an unsupervised heuristic rules- based method method. The rules are: If any of them is satisfied, contained event éj is an argu- ment of the containing event e;: (i) The syntactic dependency relation from the predicate of e, the predicate of ej is the clausal complement (ccomp or xcomp) or clausal subject (csubj)- (ii) The syntactic dependency relation from the predicate of ej to the predicate of e; is copula (cop). (iii) All of ej is contained in an argument of e; that is labelled with either ARGM-PRP ("purpose") or ARGM-PNC ("purpose not cause"). The syntactic dependency relations are obtained by the Stanford CoreNLP dependency parsing (Manning al., 2014).
GPT-2 underperforms compared to discrimina- tive models, indicating its generative architecture struggling with classification tasks such as detect- ing event dependencies. Similarly, ChatGPT's per- formance is also unsatisfactory, suggesting that its few-shot learning approach, despite leveraging powerful pre-trained model, is inadequate for com- prehending the dataset. The highest baseline ac- curacy reaches 82.61% %, revealing that EDeR is sufficient for training a good predictor for recog- nizing the event dependency relations based on the language model's 's pre-train&fine train&&fin--unee tune mechanism. Impact of Sentence Length and Predicate Dis- tance. Figure 4 (left) illustrates the performance of the baseline models across sentences with different lengths, given the input as the marked predicate sentences. Consistent with human intuition, the
4.2 Input Variations
Besides various baseline models, we also explore the influence of different types of input.
Event-Event Span We also create the two events' span style input. For the mentioned ex- ample, the Event-Event Span style input becomes "'The man who works here ( V: tells} me to get the hell out. [SEP] me VV: get} the hell o a special token "[SEP]" for separation.
Event-Event-SSe For further checking the in- fluence of the SRL labels, we add the argument
hhtps://chat. #haps://chat.opeaai.co///m

---

Input
Model
P (())
(%)
(1(%)
Ace! %
Majority
75.50
100.00
86.04
75.50
Sentence-prodicates
Rulc-based
97.67 (1)
34.93
51.46
50.08
DistilBERT
82.75
90.23
86.33
78.35
BERT
87.53
85.34
86.42
79.69
RoBERTa
84.71
87.53
86. 10
78.58
Event-Event Span
XI.Net
83.61
89.60
86.5
78.82
GPT-2
85,91
86.80
86.35
79.21
ChatGPT
76,33
85,14
80.49
68.76
DistilBERT
84.75
87.84
86.27
78.82
BERT
85.96
87.21
86.58
79.53
RoBERTa
82.89
91.16 (3)
86.83
80.06
Event-Event-SEe
XI.Net
82.00
93.76 (1)
87.49 (3)
79.69
GPT-2
86.60
85,97
86.28
79.29
ChatGPT
80.19
89.60
84.63
75.37
DistiBBERT
85.48
87.53
86,49
79.29
BERT
85.26
89.60
87.38
80.39
RoBERTa
83.21
91.16 (3)
87.00
80.37
XI.Net
83.11
91.06
86.90
79.21
GPT-2
85.89
87,94
86,90
79,92
ChatGPT
82.01
52.60
64.09
55,39
DistilBERT
82.13
93.14 (2)
87.29
79.45
BERT
91.30 (2)
85.14
88.11 (1)
82.61 (1)
RoBERTa
89.87 (3)
85,85
87.81 (2)
81.97 (2)
Marked-prediee predicate Sentence
XLNet
88.22
86.38
87.29
80.94 (3)
GPT-2
85.26
89.60
87.38
80.39
ChatGPT 75.83
80.87
78.27
66.01
Table 3: Comparison of the performance the test set based varying method and input combinations for the event dependency relation extraction task. The top-3 best results in each column are highlighted. Note: The majority classifier predicts all cases as argument.
100
100
Benefit to Related NLP Tasks
In this section, we show how EDeR data can be used to improve performance two related NLP tasks.
Acsuraty 180
70
70
60
RoSERTI
40
30
OPF/2
5.1 Event Representation Extraction
-10 19-20 20-10
30-40 40:00
35 510
10:15
15:00 >20
As mentioned in Section 3.2.3, in addition the classification of event dependencies, in the in- stances where the contained event is not an argu- ment, our data set includes more precise argument spans, which omit parts unique to the contained non-argument event, for containing events. We evaluate the performance of state-of-the--ar Se- mantic Role Labelling (SRL) system the result- ing refined event extraction task. Our results show that the refined task harder, and that giving the system access to the dependency classification (an- notated or predicted) improves it.
Sentence langth
Predicales' distance
Figure 4: Model performances (accuracy) across differ- ent ranges of sentence length (Lefft) and different ranges of predicates' distance (Right).
performance of all the models (except for Chat- GPT) decreases when the sentence length increases. Longer sentences may contain more complex struc- tures, which are challenging for models to capture. Similarly, Figure (right) shows decrease in ac- curacy with an increasing distance (i.e., the number of words) between two predicates, illustrating the models' struggle with distant dependencies. Specif- ically, few-shot-learning ChatGPT often performs better with moderate sentence length and predicate distance due to challenges with limited information from short sentences and complexity understand- ing complex event structures.
We selected SOTA SRL system CRFSRL (Zhang et al., 2022) as the baseline. CRFSRL pro- cesses sentence with marked event predicate (p) by predicting a labelled tree of syntactic depen- dencies between words, and extracting sub-trees argument spans. This is illustrated in the top part of Figure 5. Our revised system, CRESRL with Event Dependency information (w/ ED), prunes sub-trees

---

INPUT: The man who (PRD: warks] here (PRC: tnils) me (PRD: gel) the hell out
Sentence: When she saw them, the girl cried.
PRD
CRFSRL
Original event input:
ARGM-IMP cried:
40 the girl
A2
When she saw them
"",, to gei the hell out
The mán who works hene
talla
me
Xcoref-HGAT prediction: [[she, the girl], [them]]
EDeR refined event input:
cried: V the girl
XOUTPUT:
The man who works here
bolls
mc
get the hall out
/coref-HGAT prediction: [[she], [them]. [the girl]]
Event Dependoncy Predictor
PRD
CRFSRL ED
GGround truth: [[she], [ihem], [the girl]
Co -............................... t...eeeed
tollo-W worka
AI
AZ
Figure 6: An example shows how EDeR-refined event representations help the coref-HGAT model get the correct output, unlike when using OntoNotes' original representations.
The man
who works here
tells
me
t0 get
out
V VOUTPUT:
The man
Tolls
me
to
get the hell out
Figure 5: An example of the event representation output by the CRFSRL system (Zhang et al., 2022) (above) and this system equipped with our event dependency relation information (below).
ing subset of the unmodified OntoNotes data set. The evaluation was done using the scripts provided for the CoNLL-2012 shared task². Table 4 sum- marises the result. CRESRLs performance on the refined extraction task, compared to the original task. suggests that the system finds it more chal- lenging to model the refined event representations in the EDeR dataset. However, results also show that giving the system access to the event depen- dency information allows some of that drop be recovered. The performance gap much more no- ticeable on the subset of events in EDeR-test with at least one refined argument span (shown the bottom part of Table 4). In these cases, using either the baseline model- predicted or
rooted at non-argument predicates, as shown in the bottom part of Figure 5. In the following, w/ ED (G) uses the gold-standard event dependency information from the EDeR annotation, while w/ ED (P) uses predicted predicted event dependency relations, from the baseline model with the highest accuracy.
System
Precision Recall Fi-score
Test set: subset of OntoNotes corresponding to EDeR-tes(())
CRESRL (OntoNotessttranee))
77.07
81.03
79.00
Test set: EDeR-test.
CRFSRL (EDeR-traine)
74.19
78.45
76.26
event dependency feature, CRFSRL w/ ED (P) and
w/ HD (P)
74.35
78.53
76.38
(G) both improve significantly (increases of 13.8% and 20.77% in precision, and 7. 7.44% and 11.18% in F1-score) over the CRESKL system without event dependency information.
w/ ED (G)
76.09
78.81
77.43
Test set: events with refined argument only
CRESRL (EDeR-traine
67.34
65.73
66.53
w/ ED (P)
81.14
67.97
73.97
w/ ED (G)
88.11
69.51
77.71
These findings underline the importance of event dependency relation information in enhancing the SRL system for event representation extraction, especially for refined events.
Table 4: Performance of the CRFSRL system (Zhang et al., 2022), and our CRFSRL systems with predicted (w/ ED (P)) and annotated (w/ ED (G)) event depen- dency information on the refined event extraction task. (a) For comparison, performance of the CRFSRL sys- tem on the original (unrefined) event extraction task. This system is trained on the subset of OntoNotes corre- sponding to EDeR-train, i.e., with unrefined argument spans, and tested on the subset corresponding to EDeR- test. (b) This is the subset of E.DeR-test containing only events with at least one refined argument span (i.e., that differ from the original OntoNotes annotation).
5.2 Co-reference Resolution
To further investigate whether the refined event representations from EDeR are more reasonable than OntoNotes' original version, we apply them to a downstream task: co-reference resolution (CR). Given text, CR aims to identify all men- tions that refer to the same entity. We use coref- HGAT (Jiang and Cohn, 2021), SOTA CR model, initially trained on OntoNotes CR annotations. This model incorporates both event semantic role and word syntactic dependency information in het- erogeneous graphs, creating contextualized embed-
We train both the CRESRL and CRFSRL w/ ED systems on the training portion of the EDeR data set. Thus, both systems are trained to predict the refined argument spans. For comparison, we also trained a version of CRESRL on the correspond-
Phtts:/wwww.cssupcccuu sriconll/

---

dings identify co-reference links. However, the model can misinterpret unrelated entities as co- referents. Figure show an example, where "the girl" is wrongly linked to "she". this example, there is a sentence, "She went to visit them", prior to the input sentence, and "she" in the input sen- tence actually refers to "She" in this prior sentence. In this case, EDeR's 's refined event representation removes the distracting potential referent, leading the model to accurately identify "she" and "the girl" as separate entities. While it is also possible that the reduced event representation prunes informa- tion that is important for identifying co-referents, our results show that the cases that benefit from it are at least more frequent.
Conclusion
In this paper, we introduced EDeR, a high- quality human-annotated event dependency rela- tion dataset. We presented thorough description of the dataset construction process followed by detailed analysis. We implement various language models and one unsupervised rulc-based method, to establish benchmark performance against the dataset. The experimental results of these competi- tive baselines, along with the evident improvements in some related NLP tasks such as semantic role labelling and cor-reference resolution, demonstrate the utility and benefit of the EDeR dataset. We hope that EDeR facilitates future research in un- covering potential relations among events, thereby enriching the understanding of our linguistic world.
Input
Precision Recall
Fi-score
OntoNotes Event Reps. (G)
67.73
65.41
66.50
EDeR Event Reps. (G)
69.14
65.69
67.40
Limitations
OntoNotes Event Reps. (P)
64,82
63.79
24
EDeR Event Reps. (P)
67.79
64.42
65.96
As described, for our EDeR dataset, the argu- ment label can be further categorized into two fine- grained classes: required argument and optional argument. We apply the introduced baseline mod- els and inputs to this three-way classification task. The experimental results and some case study de- tails are in A.2. Overall, compared with the binary classification
Table 5:
Comparison of coref-HGAT model's co-
reference resolution performance using annotated (G) and predicted (P) event representations from OntoNotes and EDeR.
We conduct this CR task using the annotations from the same 275 OntoNotes documents and use the CoNLL-2012 official scripts for evaluation.
paapeeet/ooo-aarumeeee the per- formance is significantly dropped the highest ac- curacy is 70.79% and the ChaiGPT few-shot learn- ing results are much worse, It demonstrates that the three-way classification is more challenging for our current baseline models. Such significant per- formance gap will direct our further investigation.
As Table 5 shows, The EDeR annotated updated event representations (i.e., EDeR Event Reps. (G)) boost the coref- model's precision, recall, and F1-score by 1.41%, 0.28%, and 0.9%, respec- tively, when compared with the original event rep- resentations (i.e., OntoNotes Event Reps. (G)). This proves the refined event representations en- hance co-reference resolution, demonstrating their validity.
Ethical Considerations
We annotate the proposed EDeR dataset based on OntoNotes 5.0, without copyright constraints for academic uses. During the human annotation pro- cedure (ef. Section 3), the annotators, quality in- spectors, and the linguistic expert are only required to label factual information (i.e., the dependency relations between events and refined event represen- tations). The annotators and quality inspectors are all anonymous and the annotation does not involve any personally sensitive information.
Furthermore, the coref-HGAT model with CRF- SRL w/ ED (P) predicted updated events (i.c.. EDeR Event Reps. (P)) also reveals enhanced precision, recall, and Fl-score by 2.97%, 0.63%, and 1.68%, compared with the model with CRF- SRL predicted OntoNotes original events (i.e., OntoNotes Event Reps. (P)), as the last two lines in Table 5) shows. Despite model prediction errors, these results even comparable to the perfor- mance of using OntoNotes Event Reps. (G). This highlights the feasibility of applying the CR model more widely, by substituting the original annotated event inputs with our predicted refined event repre- sentations.
References
Jun Araki, Zhengzhong Liu, Eduaad Hovy, and Teruko Mitamura. 2014. Detecting subevent structure for event coreference resolution. In Proceedings of the Ninth International Conference on Language

---

Resources and Evaluation (LREC"14), pages 4553- 4558, Reykjavik, Iceland. European Language Re- sources Association (ELRA).
the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Cosmin Bejan and Sanda Harabagiu. 2008. A linguistic resource for discovering event structures and resolv- ing event coreference. In Proceedings of the Sixth International Conference on Language Resourees and Evaluation (LREC 08), Marrakech, Morocco. European Language Resources Association (ELRA).
Quang Do, Yee Seng Chan, and Dan Roth. 2011. Min- imally supervised event causality identification. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 294- 303, Edinburgh, Scotland, UK. Association for Com- putational Linguistics.
Steven Bethard. 2013. -rieeKK--MMM:AAAA A minimal- ist approach to TempEval 2013. In Second Joint Conference on Lexical and Computational Seman- tics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (Se- mEval 2013), pages 10-14, Atlanta, Georgia, USA. Association for Computational Linguistics.
Jannik Fischbach, Julian Frattini, Arjen Spaans, Maxim- ilian Kummeth, Andreas Vogelsang, Daniel Mendez, and Michael Unterkalmsteiner. 2021. Automatic de- tection of causality in requirement artifacts: the cira approach. In International Working Conference on Requirements Engineering: Foundation for Software Quality, pages 19-36, Cham. Springer International Publishing.
Claire Bonial, Jena Hwang, Julia Bonn, Kathryn Con- ger, Olga Babko-Malaya, and Martha Palmer. 2012. English propbank annotation guidelines. Center for Computational Language and Education Research Institute of Cognitive Science University of Colorado at Boulder, 48.
Goran Glavaš and Jan Snajder. 2014. Constructing co- herent event hierarchies from news stories. In Pro- ceedings of Tex:Graphs-92 the workshop on Graph- based Methods for Natural Language Processing, pages 34-38, Doha, Qatar. Association for Computa- tional Linguisties.
Lila Boualili, Jose G. Moreno, and Mohand Boughanem. 2020. Markedbert: Integrating traditional ir cues in pre-trained language models for passage retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Infor- mution Retrieval, SIGIR '20, page 1977-1980, New York, NY, USA. Association for Computing Machin- ery.
Goran Glavaš, Jan Snajder, Parisa Kordjamshidi. 2014. HiEve: A corpus for extracting event hicrarchies from news stories. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (IRREC 14), pages 3678-3683, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA). Eduard Hovy, Teruko Mitamura, Felisa Verdejo, Jun Araki, and Andrew Philpot. 2013. Events are not simple: Identity, non-identity, and quasi-identity. In Workshop on events: Definition, detection, corefer- ence, and representation, pages 21 21-28, Atlanta, Geor- gia. Association for Computational Linguistics.
Moens, and
Tommaso Caselli and Piek Vossen. 2017. The event
StoryLine corpus: A new benchmark for causal and temporal relation extraction. In Proceedings of the Events and Stories in the News Workshop, pages 77- 86, Vancouver, Canada. Association for Computa- tional Linguistics.
Taylor Cassidy, Bill McDowell, Nathanel Chambers, and Steven Bethard. 2014, An annotation framework for dense event ordering. Technical report, Carnegie- Mellon Univ Pittsburgh PA.
Pere-Lluis Huguet Cabot and Roberto Navigli. 2021. REBEL: Relation extraction by end-to-end language generation. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 2370- 2381, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Miao Chen, Ganhui Lan, Fang Du and Victor Lobanov. 2020. Joint learning with pre-trained transformer on named entity recognition and relation extraction tasks for clinical analytics. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 234-242, Online. Association for Computa- tional Linguistics.
Fan Jiang and Trevor Cohn. 2021. Incorporating syntax and semantics in coreference resolution with hetero- geneous graph attention network. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1584-1591, Online. Association for Computational Linguistics.
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. MuTual: A dataset for multi-turn dia- logue reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics. pages 1406-1416, Online. Association for Computational Linguistics.
Wojeiech Kryscinski. Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of

---

Processing (EMNLP-JJCLLP), pages 540-551, Hong Kong, China. Association for Computational Linguis- tics.
Long Papers), pages 2278-2288 Melbourne, Aus- tralia. Association for Computational Linguistics.
Qiang Ning, Hao Wu, and Dan Roth. 2018b. A multi- axis annotation scheme for event temporal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1318-1328, Melbourne, Aus- tralia. Association for Computational Linguistics.
Natsuda Laokulrat, Makoto Miwa, Yoshimasa Tsuruoka, and Takashi Chikayama. 2013. UTTime: Temporal relation classification using deep syntactic features. In Second Joint Conference on Lexical and Compu- tational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 88-92, Atlanta, Georgia, USA. Association for Computational Lin- guistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng. Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards ro- bust linguistic analysis using OntoNotes. In Proceed- ings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria, Association for Computational Linguistics.
Beth Levin et al. 1999. Objecthood: An event structure perspective. Proceedings of CLS. 35(1):223-24)
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization.
James Pustejovsky, Patrick Hanks, Roser Saurí, An- drew See, Rob Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, and Marcia Lazo. 2003. The timehank corpus. Proceed- ings of Corpus Linguistics, 2003:40.
Youmi Ma. Tatsuya Hiraoka, and Naoaki Okazaki. 2022. Joint entity and relation extraction based on table labeling using convolutional neural networks. In Proceedings of the Sixth Workshop Structured Prediction for NLP, pages 11-21, Dublin, Ireland. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Christopher Manning, Mihai Surdeamu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language pro- cessing toolkit. In Proceedings of 52nd Annual Meet- ing of the Association for Computational Linguis- tics: System Demonstrations, pages 55-60. Balti- more, Maryland. Association for Computational Lin- guistics.
Malka Rappaport Hovav, Edit Doron, and Ivy Sichel. 2010. Lexical Semantics, Syntax, and Event Struc- ture. Oxford University Press, UK.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
Dominique Mariko, Hanna Abi-Akl, Kim Trottier, and Mahmoud El-Haj 2022. The financial causality extraction shared task (FinCausal 2022). In Pro- ceedings of the 4th Financial Narrative Processing Workshop @ LREC2022, pages 105-107, Marseille, France. European Language Resources Association.
Fiona Anting Tan, Ali Hürriyetoglu, Tommaso Caselli, Nelleke Oostdijk, Tadashi Nomoto, Hansi Het- tiarachchi, Iqra Ameer, Onur Uca, Farhana Ferdousi Liza, and Tiancheng Hu. 2022. The causal news cor- pus: Annotating causal relations in event sentences from news. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 2298-22 2310, Marseille, France. European Language Resources Association.
Dominique Mariko, Estelle Labidurie, Yagmur Ozturk, Hanna Abi Akl, and Hugues de Mazancourt. 2020. Data processing and annotation schemes for fincausal shared task.
Paramita Mirza and Sara Tonelli. 2016. CATENA:
CAusal and T'mppora relation extraction from NAtu- ral language texts. In Proceedings of COLING 2016. the 26th International Conference on Computational Linguistics: Technical Papers. pages 64-75, Osaka, Japan. The COLING 2016 Organizing Committee.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame. Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table represen- tation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1858-1869, Doha Qatar. Associa- tion for Computational Linguistics.
Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth. 201 8a. Joint reasoning for temporal and causal relations. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1:
Linyi Yang, Zhen Wang, Yuxiang Wu, Jie Yang, and Yue Zhang. 2022. Towards fine -ne-ggained causal reasoning and qa.

---

Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Ruslan Salakhutdinov, and Quoe Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding.
Therefore, I can feel that um. um, parents who (V2: work) in the city
also (V1: have) no alternative.
Gold: independent
Syntactic dependency parser error: work have
XModel prediction: required argument
As DPP legisiator Shen Fu hilung (V2: pointed) out the Group (V1: hadj to draft something
Yu Zhang, Qingrong Xia, Shilin Zhou, Yong Jiang, Guo- hong Fu, and Min Zhang. 2022. Semantic role la- beling as dependency parsing: Exploring latent tree structures inside arguments. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 4212-495 4222, Gyeongju. Republic of Korea. International Committee on Computational Linguistics.
Gold: independent
Event argument labelling error:
(ARGM-ADVV As DPP legislator Shen Fu hsiung point out) |V1 had)
XModel prediction:
optional argument
Figure 7: Case study for fine- aie-ggrined event dependency relation extraction.
Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218-1221 1217, Huhhot, China. Chinese Informa- tion Processing Society of China.
events labelled as required arguments are located within the numbered arguments (ARGO-ARG5) of their containing events. Second, like in the binary classification task, the BERT and RoBERTa models outperform the others. RoBERTa achieves the best overall precision, recall, Fl Fl-score and accuracy re- sults, which reveals its better relation reasoning capacity. Third, the overall precision and recall results for the "required argument" label tend to be higher in comparison other labels, likely as a result of its preponderance in the dataset.
A Appendix
A.1 Statistics of Documents for Data Collection
OntoNotes contains semantic role-formatted event representations, the OntoNotes example in Fig- ure shows. We randomly sampled 275 docu- ments from seven genres: broadcast news (bn). magazine (mz), newswire (nw), pivot corpus (pt), telephone conversation (tc), broadcast conversation (bc), and web data (wb). Data statistics of the 275 raw documents are shown in Table 6. The num- ber of sampled documents and the separation of them into training, development and test sets under each genre follows their initial distributions in the OntoNotes dataset.
Finally, the overall performance on both the 3- way and 4-way classification tasks is lower than for binary classification - the highest accuracy achieved are 70.79% and 69.69% respectively indicating that these tasks are harder.
A.2.1 Case Study
We identify two phenomena from the Event-Event SRL-DEP based RoBERTa model for the fine- grained event dependency relation extraction:
A.2 Fine-grained Event Dependency Relation Extraction
Syntactic Dependency Parsing Errors Some incorrect (three-way) classification results may be because of the misleading syntactic dependency parsing results. As shown in the first case of Fig- ure 7, the off-the-sheelf dependency parser wrongly assigns syntactic dependency relation between the two event predicates "work" and "have", which could affect the model prediction.
As described, for our EDeR dataset, the argument label can be further categorized into two more fine- grained classes: required argument and optional argument. Likewise, the non-argument label can be further separated into the two classes condition and independent. We apply the baseline models and inputs also to these 3-way and 4-way classification tasks.
The 3-way classification results of the baseline language models taking various types of inputs are shown in Table 7. First, the input achieves overall better performance than oth- ers in this task. Besides the useful information pro- vided by the syntactic dependencies as discussed in Section 4.4, another possible reason is that the EDeR dataset, the predicates of over 95.79% of the
Event Argument Labelling Errors Further- more, the second case of Figure shows, the mantic labels provided by OntoNotes for the event predicate "had" is incorrect (e.g.. "the group" is missing as the ARGO). It may prevent the model from accurately predicting the event dependency relation types.

---

bn
mz
nw
pt
tc
be
wb
overall
# documents
104
7
107
31
7
3
16
275
# documents-train
90
5
89
24
5
1
13
227
# documents-dev
7
1
9
4
1
1
25
# documents-test
7
9
3
1
1
23
avg # sentences per doc
6.2
48.9
14.4
40.7
71.7
203.8
59.3
24.9
avg # words per doc avg # events per doc
139.7
1599.7
418.7
657.0
1089.4
4045.8
1509.3
543.4
23.6
202.1
58.3
148.7
362.1
1058.6
268.5
93.5
Table 6: Statistics the documents (under each genre and all) that we sampled from the OntoNotes dataset for annotation: number of documents and the number of documents split into different (train, development and test), and average number of sentences, words and events per document.
Input Model
Precision (%)
Recall (%)
FI(%)
Accuracy(%
required
optional
non-arg.
overall
required
optional
non-arg.
overall
DistilBERT
76.06
52.04
55.06
61.05
81.31
45.11
56.49
60.97
60.90
64.80
BERT
82.90
58.48
54.62
65.33
85.69
44.02
67.21
65.64
64.92
69.13
RoBERTa
83.47
54.01
59.41
65.63
83.33
54.89
58.44
65.56
69.06
XLNet
80.46
50.71
56.39
62.52
83.16
48.37
55.84
62.46
62.47
66.46
GPT-2
76.41
$2.01
56.68
61.70
45.65
$6.49
61.49
65.43
ChatGPT
47.63
32.08
52.53
47.55
37.59
35.19
39.65
DistilBERT
75.04
51.50
64.68
63.74
84.51
55.98
42.21
60.90
61.41
65.98
BERT
84.59
54.55
$7.89
65.68
83.16
53.80
60.71
65.89
65.77
69.21
RoBERTa
83.75
58.13
56.37
66.08
84.18
50.54
64.61
66.44
66.08
69.69
XLNet
81.66
59.27
56.20
65.71
84.68
44.29
69.16
66.04
65.28
69.21
GPT-2
76.12
$4.75
58.25
63.04
82.66
47.01
58.44
62.70
62.73
66.46
ChatGPT
54.55
32.45
26.92
37.97
65.66
46.74
2.27
38.22
34.02
44.77
DistiBBERT
77.27
53.93
60.23
63.81
81.82
55.98
50.65
62.82
63.15
66.77
BERT
81.51
56.42
60.70
66.21
85.35
51.36
61.69
66.13
66.12
69.76
RoBERTa
83.20
59.38
59.04
67.21
85.86
52.45
63.64
67.31
67.15
70.79
XLNet
86.27
53.68
58.07
66.01
82.49
55.43
60.71
66.21
66.08
69.37
GPT-2
78.58
53.42
56.57
62.86
82.15
46.74
60.06
62.99
62.82
66.54
ChatGPT
55.07
26.18
27.39
36.21
19.19
37.77
47.25
34.74
31.36
31.39
DistilBERT
73.07
$1.91
61.57
62.18
81.31
48.10
53.57
60.99
61.40
64.96
BERT
85.26
$3.80
58.66
65.91
8..82
50.00
68.18
66.67
66.13
69.29
RoBERTa
84.32
56,93
56.23
65.83
84.18
51.36
62.99
66.17
65.89
69.53
XLNet
84.75
55.34
54.41
64.83
80.47
46.47
70.13
65.69
64.78
68.11
GPT-2
77.00
50.00
55.11
60.70
45.38
57.79
79.46
60.88
60.74
64.33
ChatGPT 37.68
28.42
25.00
32.39
17.51
66.03
11.33
33.22
27.54
31.86
Table 7: Comparison of the performance for the 3-way event dependency relation classification task. Input data and ( Marked-predicatee Sentence. The precision and recall results for particular labels are presented columns under "required", "optional" and "non-argument". The best results each column are highlighted. The overall precision, recall and Fi-score results are macro-averaged.
types are: ! Event-Event Span; @ E-vent-Event-SRI: 3

---

A.3 Implementation Settings
V: says.
Event 2: The stock price V; unstable.
We employ the distilbert-base-cased bert- large-cased, roberta-large, xlnee-large- cased and gpt2-large HuggingFace's Trans- former Library (Wolf et al., 2020) as represent- ing the DistilBERT, BERT, RoBERTa, XLNNe and GPT-2 baseline models in the experiment. For the fairness of comparison, all the models use the same optimization method as AdamW (Loshchilov and Hutter, 2017) 7) which is adopted with an initial learn- ing rate of 1e-5 and batch size of 4 and a number of epochs of during the finetuning process. The model with the highest Fl-score on the develop- ment set is selected.
Explanation: Event 2 is a required argument. If is removed, Event would become "Mr. Jones says", which is incomplete and not meaningful (one must say something, "something" is missing if Event 2 is omitted).
Optional argument: Event 2 is an argument of the verb of Event 1, but Event 2 only provides additional or clarifying information about Event (for example, Event may indicate the purpose or goal of Event 1). Event is still syntactically complete and meaningful, although it may not keep the exact same meaning, if Event 2 is removed. For example.
Sentence: A power failure with the docking door forces Brodski to go EVA to fix it. Event 1: Brodski V: go EVA to fix it.
A.4 Annotation Instructions
An event is composed of a verb and arguments, where the verb indicates the event's action and the arguments represent the subject, object and so on of the verb. This task requires you to determine whether, in pair of events (Event 1, Event 2) from the same sentence, Event 2 is an argument condition of Event 1, and if it an argument, whether is required or optional.
Event 2: Brodski V: fix it.
Explanation: Event 2 an optional argument, indicating the purpose of Event 1. Removing it leaves Event "Brodski go(es) EVA", which is still syntactically complete and meaningful clause. Another example of the optional argument relation: Sentence: It is good opportunity, as eventually turned out to be.
Required argument: Event 2 is required for the verb of Event to be complete and meaningful. For example,
Event 1: It V: is good opportunity, as eventually turned out to be. Event 2: eventually V: turned out to be.
Sentence: Jenny tries to stop them from ruining her family.
Explanation: Event 2 is an optional argument, adding information about the timing of Event 1. Removing it leaves Event 1 "It is good opportu- nity". which is still syntactically complete and meaningful clause.
Event 1: Jenny V: tries stop them from ruining her family.
Event 2: Jenny V: stop them from ruining her family. ("V: marks the verb of each event.) Explanation: Event a required argument. it is removed, Event becomes just "Jenny tries (to)", which is incomplete and not meaningful (to "try to", one has to try to do something).
Condition: Event 2 describes condition for the occurrence of Event For example,
Sentence:
Joe will hate her if she tells the truth.
Another example of the required argument:
Event 1: Joe will V: hate her if she tells the truth.
Sentence: Krishna suggests Ramesh to send his liquor to town.
Event 2: she V: tells the truth. Explanation: Event 2 is condition of Event 1; it is not an argument.
Event 1: Krishna V: suggests Ramesh to send his liquor to town.
Another example,
Event 2: Ramesh V: send his liquor to town.
Sentence: Should problems arise, we will seek professional help.
Explanation: Event 2 is a required argument. If is removed, Event becomes "Krishna suggests Ramesh", which is not mcaningful (to suggest to someone, one must should suggest something).
Event 1: Should problems arise, we will seek professional help.
Event 2: Should problems V: arise.
The third example of the required argument: Sentence: "The stock price is unstable." Mr. Jones says.
Explanation: Event 2 is condition of Event 1; it is not an argument.
Neither argument nor condition: Event
Event 1: "The stock price is unstable." Mr. Jones

---

an optional required argument, or condition of Event It is an independent/separut event, which happens to be mentioned in the same sentence. Like an optional argument or condition, removing Event 2 still leaves Event complete and meaningful. The difference to Event 2 being optional argument is that when Event 2 is not argument it does not add any information//llrificalli
2
two events does not make Event 2 argument of Event
about the verb of Event it may still add some information about one of the arguments of Event 1.
For example,
Sentence: Julia, who has fallen in love with Barnabas, discovers his dalliance with Maggie, Event 1: Julia, who has fallen in love with Barnabas, V: discovers his dalliance with Maggie. Event 2: Julia, who has V: fallen in love with Barnabas
Explanation: Event is unrelated to Event 1; is a separate event that happens to share the same subject. Whether Julia falls in love with Barnabas or not does not affect whether, how, or when, she discovers his dalliance with Maggie. Another example:
Sentence: It increases the discount rate it offers to known customers.
Event 1: It V: increases the discount rate it offers known customers.
Event 2: the discount rate V: offers to known customers.
Explanation: Event is unrelated to Event 1; is a separate event that only describes the object (the discount rate) of Event 1. Event 2 does not affect the verb of Event "increases".
The third example of neither argument condition:
Sentence: After an opening-credits montage of the major players demonstrating their abilities, the story begins.
Event I: After an opening-credits montage of the major players demonstrating their abilities, the story V: begins.
Event 2: an opening-credits montage of the major
players V: demonstrating their abilities.
Event 2 separate/independent event that happens before Event 1. Event only adds some description of the opening-credits mon- tage; it does not add information about Event ("The story begins"). The ordering between the
Explanation:


Reviews:
[{"review_id": "hzHnoq6tjb", "reviewer": "Reviewer_HYy2", "summary": "", "questions": "1. Table 3. Why the recall score of ChatGPT is only 52.6% under Event-Event-SRL-DEP setting?\n2. Table 4. What is the meaning of “All update events”? Is it “Refined events only” plus “unchanged events”? If so, why does “Refined events only” achieve better results in F1 scores, since the training set is smaller than “All update events”?\n3. Line 554-556. Is it more likely that “she” and “the girl” refer to the same entity?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback. In response to the \"Reasons To Reject\" and the \"Questions For The Authors\" presented by the reviewer, we would like to address them as follows:\n\n### Reasons To Reject:\n\nExperimental results do not show the effectiveness of fine-grained classes.\n\nPherpas lacks Inter-annotator Agreement to better demonstrate the quality of EDeR.\n\n**Author response:** Current fine-tuned language models exhibit a 10% lower accuracy for fine-grained classes compared to binary ones. Given that binary classes already sufficiently cater to downstream tasks such as co-reference resolution, our immediate focus is on enhancing the predictor's performance. Nevertheless, evaluating the effectiveness of fine-grained classes is firmly on our future work agenda.\n\nRegarding the demonstration of the quality of EDeR, we have some statistics that are not presented in the paper due to the page limit:\n\n1. Among the 4,771 samples (40.2% of the total 11852 cases) from EDeR that have received the three Quality Inspectors' annotations, only 31 (0.65%) of them received three different labels from the three QIs.\n   \n2. The authors randomly sampled 417 cases that received at least four annotations; the accuracy of the annotated labels is 90.65% (378/417).\n\n3. The authors also randomly sampled 205 cases from the cases that received a single annotation (from one annotator), the accuracy is 83.41% (171/205).\n\nWe will include these statistics in the revision.\n\n### Questions For The Authors:\n\n* **Question:** Table 3. Why the recall score of ChatGPT is only 52.6% under Event-Event-SRL-DEP setting?\n  \n  **Answer:** The potential reasons:\n  \n   1. **Distribution of DEP features:** A significant portion of the test set data, specifically over 70% labeled as \"argument\", has the DEP feature classified as \"none\". More than half of the \"non-argument\" data also has the feature as \"none\". Given our application of few-shot learning on ChatGPT, this overlap might confuse the model, complicating its inference abilities.\n\n   2. **Utility of specific dependency relations:** While dependency relations like \"ccomp\" and \"xcomp\" assist in detecting required and optional argument relations, they primarily enhance precision. For instance, event pairs with \"ccomp\" or \"xcomp\" relations are likely to exhibit event dependency relations, but the reverse isn't necessarily true. This observation is evident from our proposed rule-based approach. For unsupervised methods, or models like few-shot learning-based ChatGPT, while precision is satisfied, the recall might be significantly lower.\n\n   3. **Comparison of SRL and DEP features:** The SRL feature operates differently from the DEP feature. Primarily, the semantic role of argument events tends to be numbered arguments. For instance, over 78% of cases labeled as \"argument\" in the test set possess the SRL feature categorized as either \"ARG1\" or \"ARG2\". In contrast, this ratio drops to 39% for \"non-argument\" cases. This distinction might explain why the SRL feature, when used solely, yields superior performance.\n\n\n* **Question:** Table 4. What is the meaning of “All update events”? Is it “Refined events only” plus “unchanged events”? If so, why does “Refined events only” achieve better results in F1 scores, since the training set is smaller than “All update events”?\n  \n  **Answer:** Yes, \"all update events\" consist of both \"refined events only\" and \"unchanged events\". We utilized the EDeR training set events to train the CRFSRL model, aiming to predict \"all updated events\" and \"refined events only\" separately. The results in Table 4 reflect comparisons made by the CRFSRL model using the same training set, but differing test sets. Specifically, \"all updated events\" refers to the complete EDeR test set events, while \"refined events only\" pertains to just the refined events from the EDeR test set. We apologize for any confusion and will ensure clarity in the subsequent edition.\n  \n   \n* **Question:** Line 554-556. Is it more likely that “she” and “the girl” refer to the same entity?\n  \n  **Answer:** \"she\" in the example sentence does not refer to \"the girl\". There is a sentence prior to the example sentence in the document: \"She went to visit them.\" \"she\" in the example sentence actually refers to \"She\" in this prior sentence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The focus on addressing independence between two events is quite interesting and makes a lot of sense.\n2. This paper explores the conditional and dependency relationships between events, which is helpful in some related NLP tasks.\n3. The dataset has high-quality annotation results under a reasonable annotation procedure.\n4. The experiments are well set and show positive results.\n", "reasons_to_reject": "1. Experimental results do not show the effectiveness of fine-grained classes.\n2. Pherpas lacks Inter-annotator Agreement to better demonstrate the quality of EDeR.", "questions_to_authors": "1. Table 3. Why the recall score of ChatGPT is only 52.6% under Event-Event-SRL-DEP setting?\n2. Table 4. What is the meaning of “All update events”? Is it “Refined events only” plus “unchanged events”? If so, why does “Refined events only” achieve better results in F1 scores, since the training set is smaller than “All update events”?\n3. Line 554-556. Is it more likely that “she” and “the girl” refer to the same entity?", "paper_topic_and_main_contributions": "This paper introduces a human-annotated Event Dependency Relation dataset (EDeR) which explores the interdependency between events. Annotators annotate argument relationships between the event pairs. The quality of the dataset is assured by a multi-level qualification-based annotation procedure. The experiment results demonstrate the utility and benefit of the EDeR dataset.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dfPhvqZ2e3", "reviewer": "Reviewer_9FEu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's perceptive feedback. Although the reviewer hasn't posed any specific questions to us, we would like to address the points raised under the \"Reasons To Reject\" section.\n\n### Reasons To Reject:\n\nWhile the paper introduces new dataset along with the solid explanation of data construction procedure, the novelty of the paper is not enough.\n\n**Author response:** We'd like to highlight several aspects that underscore the novelty of our paper:\n\n1. **Shift in focus from entities to events:** For decades, much of the previous literature in relation extraction domain has been concentrated on relationships between entities rather than events. Our paper shifts significantly from this norm by bringing to the forefront the importance of events and their intricate interrelationships.\n\n2. **Broadening the understanding of event relations:** While existing research on event relations has primarily explored causal, temporal, and hierarchical relations, our study embarks on a previously underexplored domain: the syntactic and semantic dependencies between events. Such an exploration addresses the gaps left by current event-relation systems, thereby broadening our overall comprehension of event dynamics.\n\n3. **Highlighting overlooked dependencies:** Our work is pioneering in that it emphasizes the often-overlooked interdependence of events. The prevailing research has mostly treated events as syntactically and semantically independent, potentially leading to incomplete or even erroneous interpretations of narratives. By exploring these deeper dependencies, our study proposes a more holistic approach to event relation extraction.\n\n4. **Real-world application and impact:** By understanding the dependencies between events, we can not only interpret textual nuances more accurately but also reduce potential misinterpretations. This paper, in particular, shows how recognizing event dependency can innovatively alter the interpretation of the narrative. Also, the experimental results demonstrate the improvement of such relations in some popular NLP tasks such as semantic role labeling and co-reference resolution. Its impact on the development of the NLP community is significant.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation is very clear. Paper presents a thorough description of annotation process followed by well-designed experiments and analysis.", "reasons_to_reject": "While the paper introduces new dataset along with the solid explanation of data construction procedure, the novelty of the paper is not enough.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces human-annotated dataset EDeR. The dataset extracts event dependency information based on a sample set of documents and provides refined semantic role labeled event representations.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aXBXyh0CEJ", "reviewer": "Reviewer_R3vw", "summary": "", "questions": "A. How was the total number of documents for annotation (275) determined? Was this just based on resource constraints or was there some other principle behind it?\n\nB. It is counterintuitive to me that the modified event structures would boost performance on entity coreference resolution. If I understand correctly, by pruning dependency parses based on the event relations, isn't it possible that you are dropping spans that participate in a gold coreference chain? Or does the model still have access to these spans, albeit not as part of the dependency parse? I feel like I must be misunderstanding something here.\n\nC. The authors note that \"For the analysis of the dataset and experiments in the remainder of this paper, we treat the two non-argument labels as one\" (L253-5). Why was this done? Among non-arguments, the distinction between a condition and a wholly independent event is semantically very salient.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful feedback. In response to the concerns highlighted in \"Reasons To Reject\", \"Questions For The Authors\", and the suggestions for \"Presentation Improvement\", we present the following explanations and adjustments:\n\n### Reasons To Reject:\n\nI do not in general have significant methodological concerns with the paper; most of my concerns relate to presentation and certain points of clarity (see Questions for The Authors and Typos, Grammar, Style, and Presentation Improvements), though I'll raise one point about annotator agreement here. While the authors' description of the data collection procedure inspires confidence with respect to the dataset's quality (notably, the fact that the 5% sampled annotations had 95% accuracy with respect to the authors' judgments), no standard agreement metrics are presented on the full subset of the data that is redundantly annotated. This does make it somewhat difficult to get a full sense for its quality.\n\n**Author response:** Regarding the evaluation of the quality of EDeR, we have some statistics that are not presented in the paper due to the page limit:\n\n1. Among the 4771 samples (40.2% of the total 11852 cases) from EDeR that have received the three Quality Inspectors' annotations, only 31 (0.65%) of them received three different labels from the three QIs.\n   \n2. The authors randomly sampled 417 cases that received at least four annotations; the accuracy of the annotated labels is 90.65% (378/417).\n\n3. The authors also randomly sampled 205 cases from the cases that received a single annotation (from one annotator), the accuracy is 83.41% (171/205).\n\nWe will include these statistics in the revision.\n\n### Questions For The Authors:\n\n* **Question:** How was the total number of documents for annotation (275) determined? Was this just based on resource constraints or was there some other principle behind it?\n  \n  **Answer:** The number of documents we sampled corresponds to the number of event pairs we aim to annotate. Based on existing research in event relation extraction, we believe a minimum of 10k samples is essential for fine-tuning a pre-trained language model. For reference, the BERT baseline model already attains an accuracy of 82.61%. This sampling helps us gauge the potential benefits of these event dependency relations. We're considering annotating more documents in the future to further enhance the dataset. \n\n* **Question:** It is counterintuitive to me that the modified event structures would boost performance on entity coreference resolution. If I understand correctly, by pruning dependency parses based on the event relations, isn't it possible that you are dropping spans that participate in a gold coreference chain? Or does the model still have access to these spans, albeit not as part of the dependency parse? I feel like I must be misunderstanding something here.\n  \n  **Answer:** It is possible that our refined event representations drop spans that participate in a gold coreference chain, but such instances are fairly rare. As illustrated in Figure 6, \"she\" in the example sentence does not refer to \"the girl\". There is a sentence prior to the example sentence in the document: \"She went to visit them.\" \"she\" in the example sentence actually refers to \"She\" in this prior sentence. We shorten the span of the event with the verb \"cried\" by removing \"when she saw them\", which breaks the dependency path between \"she\" and \"the girl\" then successfully corrects the incorrect co-referring chain between them. \n\n* **Question:** The authors note that \"For the analysis of the dataset and experiments in the remainder of this paper, we treat the two non-argument labels as one\" (L253-5). Why was this done? Among non-arguments, the distinction between a condition and a wholly independent event is semantically very salient.\n\n  **Answer:** Thank you for pointing it out. The mentioned sentence is indeed misplaced and will be removed in the revision. We've presented data set statistics for the two non-argument labels separately, in table 2. As for experiments, we've also executed both three-way (required argument, optional argument, and non-argument) and four-way (required argument, optional argument, condition, and neither an argument nor a condition) classifications utilizing the baseline models. The results of the three-way classification and case studies can be found in Appendix A.2. In our revision, we'll incorporate the four-way classification results. The classification accuracy has seen a decline of about 15% in comparison to the binary classification models.\n\n### Missing References, Typos, and Presentation Improvement:\n\n**Overall response:** We are grateful for your literature recommendations. Especially in the third paper, where the proposed \"parthood event-event relation\" aligns closely with our event dependency relations and offers significant inspiration. We also thank you for identifying typos and suggesting improvements in the representation. We will cite the missing references, provide more technical details, and conduct more rounds of proofreading to ensure that grammar issues are fixed.\n\nSpecifically, addressing the representation concerns you've highlighted:\n\n* Majority class baseline accuracy really should be explicitly presented (even if only in text) for comparison with the results in Table 3 (though I believe it's inferrable from Table 2).\n  \n  **Author response:** We'll incorporate the baseline accuracy details you mentioned into Table 3 for a more explicit comparison in the revision.\n  \n* Table 3 should also include F1 scores.\n  \n  **Author response:** We will include the F1-score into Table 3 in the next edition.\n\n* For clarity, it would be good to have explicit examples of the \"Event-Event-SRL\" and \"Event-Event-SRL-DEP\" input variations presented in 4.2. (This is already done for the other two variants.)\n  \n  **Author response:** Due to page constraints, we omitted examples for both \"Event-Event-SRL\" and \"Event-Event-SRL-DEP\". Here's an illustrative example for each:\n\n  * **Event-Event-SRL:** \"The man who works here {V: tells} me to get the hell out. [SEP] me {V: get} the hell out [SRL] ARG1.\"\n  * **Event-Event-SRL-DEP:** \"The man who works here {V: tells} me to get the hell out. [SEP] me {V: get} the hell out [SRL] ARG1 [DEP] xcomp.\"\n    \n   We aim to include such examples in the revision for clarity.\n\n* An analysis of model performance broken down by label, possibly including some qualitative evaluations, would (for my money) be more interesting and informative than the present analysis based on predicate distance and sentence length (though this analysis is still worthwhile).\n \n  **Author response:** We will include the P/R/F1 results for particular labels, respectively, for the 3-way and 4-way classifications in the revision. Similar to our approach with the 3-way classification results (can be found in Appendix A.2.1), we will also do some qualitative evaluations, like case studies, for binary classification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The dataset seems to be fairly carefully constructed and likely to be of interest to a significant subset of the NLP community, including folks working on event semantics and on information extraction. The fact that it builds on an existing widely used benchmark (OntoNotes) is also a plus.\n- The experiments in general seem to accomplish most of what one expects of experiments in a resource paper — namely, establishing an accessible set of baselines and demonstrating the utility of the resource for existing tasks.", "reasons_to_reject": "I do not in general have significant methodological concerns with the paper; most of my concerns relate to presentation and certain points of clarity (see **Questions for The Authors** and **Typos, Grammar, Style, and Presentation Improvements**), though I'll raise one point about annotator agreement here. While the authors' description of the data collection procedure inspires confidence with respect to the dataset's quality (notably, the fact that the 5% sampled annotations had 95% accuracy with respect to the authors' judgments), no standard agreement metrics are presented on the full subset of the data that is redundantly annotated. This does make it somewhat difficult to get a full sense for its quality.", "questions_to_authors": "A. How was the total number of documents for annotation (275) determined? Was this just based on resource constraints or was there some other principle behind it?\n\nB. It is counterintuitive to me that the modified event structures would boost performance on entity coreference resolution. If I understand correctly, by pruning dependency parses based on the event relations, isn't it possible that you are dropping spans that participate in a gold coreference chain? Or does the model still have access to these spans, albeit not as part of the dependency parse? I feel like I must be misunderstanding something here.\n\nC. The authors note that \"For the analysis of the dataset and experiments in the remainder of this paper, we treat the two non-argument labels as one\" (L253-5). Why was this done? Among non-arguments, the distinction between a condition and a wholly independent event is semantically very salient.", "paper_topic_and_main_contributions": "This paper presents the Event Dependency Relation (EDeR) dataset, which annotates binary argument relations on close to 12,000 pairs of predicates from a subset of documents from the OntoNotes dataset. Relations between predicates are assigned one of four labels: for eventive arguments, one predicate may be semantically **required** by the other or else may be **optional**; and for eventive *non-arguments*, one predicate may nonetheless express a **condition** on the other predicate or else may be fully **independent** of it. The paper also presents baseline models for relation classification on EDeR in both fine-tuned (using standard pretrained LMs) and few-shot settings (using ChatGPT), with experiments showing performance variation across different input representations. Results from additional experiments argue that incorporating event dependency information into training for related tasks (semantic role labeling and coreference resolution) may boost performance on those tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "I'm aware of several datasets that are distinct in their aims from the present work but that are worth citing as relevant general background:\n- The Basic dataset from the IARPA BETTER program (see [here](https://ir.nist.gov/better/)) contain event structures with eventive arguments (so-called \"referred events\"). NB: To my knowledge, there is no publication associated with the Basic datasets specifically, but see [this paper](https://aclanthology.org/2022.lrec-1.384/) for a description of the program and the related Abstract datasets.\n- \"Richer Event Description: Integrating event coreference with temporal, causal and bridging annotation\" (O'Gorman et al., 2016) has received a good amount of attention and contains several different types of event-event relations.\n- \"Decomposing and Recomposing Event Structure\" (Gantt et al., 2022) annotates the English Web Treebank for event parthood relations as part of its event structure dataset.", "Typos_Grammar_Style_and_Presentation_Improvements": "Typos / Grammar:\n\nIn general, the paper would benefit from a careful proofread. Below are a sampling of suggested edits:\n- 142-3: \"Several event-event relations have been proposed for decades\" -> (e.g.) \"Various event relation ontologies have been proposed over the years.\"\n- 179: \"an optional argument of, or a condition of\" -> \"an optional argument of, a condition of\"\n- 332: hence -> and so\n- 394-6: this description of the rule-based classification method is rather awkward. Please rephrase.\n- 529: feature -> features\n- figure 6 caption: help -> helps\n\nPresentation:\n- Majority class baseline accuracy really should be explicitly presented (even if only in text) for comparison with the results in Table 3 (though I believe it's inferrable from Table 2).\n- Table 3 should also include F1 scores\n- For clarity, it would be good to have explicit examples of the \"Event-Event-SRL\" and \"Event-Event-SRL-DEP\" input variations presented in 4.2. (This is already done for the other two variants.)\n- An analysis of model performance broken down by label, possibly including some qualitative evaluations, would (for my money) be more interesting and informative than the present analysis based on predicate distance and sentence length (though this analysis is still worthwhile)."}}]

Author rebuttal:

Meta-review: The paper introduces a human-annotated Event Dependency Relation dataset.

The three reviewers agree on the strong soundness of the work (4) and moderate to strong excitement (4-3-4). The reviewers have provided several suggestions for improvement and questions that should be incorporated in the final version of the manuscript if accepted. Among these, I find the following particularly relevant: 
- Inter-annotator Agreement to better demonstrate the quality of EDeR - and any other quality statistics (such as as the one described in the rebuttal).
- Typos, grammar, style, and presentation improvements

Although I agree that a statement such as " the novelty of the paper is not enough" is unsupported, I believe that including in the paper the novelty highlights summarized in the rebuttal could improve the final version.
Decision: Accept-Main