GLEN: General-Purpose Event Detection for Thousands of Types
Authors: Sha Li
Qiusi Zhan
Kathryn Conger
Martha Palmer
Heng Ji
Jiawei Han
Venue: EMNLP 2023 Main (2023)
Keywords: event extraction; event detection; dataset
TL;DR: We introduce a new large-scale event detection dataset GLEN which covers over 3k event types.

Abstract: The progress of event extraction research has been hindered by the absence of wide-coverage, large-scale datasets. 
To make event extraction systems more accessible, we build a general-purpose event detection dataset GLEN, which covers 205K event mentions with 3,465 different types, making it more than 20x larger in ontology than today's largest event dataset. GLEN is created by utilizing the DWD Overlay, which provides a mapping between Wikidata Qnodes and PropBank rolesets. This enables us to use the abundant existing annotation for PropBank as distant supervision.
In addition, we also propose a new multi-stage event detection model specifically designed to handle the large ontology size in GLEN. We show that our model exhibits superior performance compared to a range of baselines including InstructGPT.
Finally, we perform error analysis and show that label noise is still the largest challenge for improving performance for this new dataset.

GLEN: General-Purpose Event Detection for Thousands of Types
Qiusi Zhan't Sha Li Kathryn Conger", Martha Palmer", Heng Ji', Jiawei Han'
UUniversity of Illinois Urbana-Champaign
University of Colorado Boulder
{qiusiz2, shal2, hengji, hanjj@@illiois.edd (kathryn.conger, martha, palmmer)@coloordooooc edu
Abstract
Sentence
clinical tasting services holding company basad san juan capistrano, Calif.
PropBank Annolation
The progress of event extraction research has been hindered by the absence of wide-coverage, large-scale datasets. To make event extraction build a general-
Propßank Rolesets
1eat.1
serve.01
systems more accessible, purpose event detection dataset GLEN which covers 205K event mentions with 3,465 ferent types, making it more than 20x larger in ontology than today's 's largest event dataset. GLEN is created by utilizing the DWD Over- lay, which provides mapping between Wiki- data Qnodes and PropBank rolesets. This en- ables
Overlay
medical _est
business_sene
phychological.co
national_servies
Wikibata Gnodes
test quality)
public_eervioe
(merft assessment)
testing (collective)
The GLEN event detection dataset is con- structed by using the DWD Overlay, which provides mapping between PropBank rolesets and WikiData Qnodes. This allows us to create a large distantly- supervised training dataset with partial labels.
Figure I:
the abundant existing annotation for PropBank as distant supervision. In addi- tion, we also propose a new multi stagg event detection model CEDAR specifically designed to handle the large ontology size in GLEN. We show that our model exhibits superior per- formance compared
facto standard evaluation benchmark. Key limi- tations of ACE include its small event ontology of 33 types, small dataset size of around 600 doc- uments and restricted domain (with a significant portion concentrated on military conflicts). The largest effort towards event extraction annotation is MAVEN (Wang et al., 2020), which expands the ontology to 168 types, but still exhibits limited do- main diversity (32.5% of the documents are about military confficts).
range of baselines in- cluding InstructGPT. Finally, we perform error analysis and show that label noise is still the largest challenge for improving performance for this new dataset.'
1 Introduction
As one of the core IE tasks, event extraction in- volves event detection (identifying event trigger mentions and classifying them into event types), and argument extraction (extracting the participat- ing arguments). Event extraction serves the basis for the analysis of complex procedures and news stories which involve multiple entities and events scattered across a period of time, and can also be used assist question-answertne answering and dialog sys- tems.
The focus of current benchmarks on restricted ontologies in limited domains is harmful to both developers and users of the system: it distracts researchers from building scalable general-purpose models for the sake of achieving higher scores said benchmarks and it discourages users as they are left with the burden of defining the ontology and collecting data with little certainty as to how well the models will adapt to their domain. We believe that event extraction can, and should be made accessible to more users.
The development and application of event ex- traction techniques have long been stymied by the limited availability of datasets. Despite the fact that it is 18 years old, ACE 2005 is still the de
To develop general- purppsse event extraction system, we first seek to efficiently build high- quality open-domain event detection datasct. The difficulty of annotation has been a long-standing
OOr dataset, code, and models are released at https: //gi sigithub.coom cotuu...om//qsii// 943/GLEN, git. 'https: //www. http//wwll....o 1dc. upenn. upenn.edu/col. latoraaionoo past-projects/ace

---

Sentence a: A police officer taking part in recovering the bodies said about 17 people were killed.
Trained with cleen annotation
Trainsd aed with noney annotation
Predicted triggers
Trigger identification
recovering
Type Claseification
killed
P(e 4, is a trigge)) Pryee" ) P(
Event Type Ranking
@ Interiation
Predicted results
Top k events
recovery
MaskodLM
killed killing
recovering recovery
killing
8:
violent death
defined (defnnton), (sentence). (ropper) indicatte (lype) event? [MASK]
massearre
Event Node eş: recovery return lost stolen item to its owners
}
Thousands of events
Figure 2: Overview of our framework. We first identify the potential triggers from the sentence (trigger identifica- tion) and then find the best matching type for the triggers through coarse-grained sentence-level type ranking followed by a fine-grained trigger-specific type classification.
issue for event extraction as the task is defined with lengthy annotation guidelines, which require training expert annotators. Thus, instead of aim- ing for perfect annotation from scratch, start with the curated DWD Overlay (Spaulding al., 2023) which defines a mapping between Wikidata" and PropBank rolesets (Palmer et al., 2005) as displayed in Figure 1. While Wikidata Qnodes can serve as our general-purpose event ontology, the PropBank roleset information allows us to build large-scale distantly supervised dataset by link- ing our event types existing expert- annotated resources. In this process, reuse the span-level annotations from experts, while dramatically ex- panding the size of the target ontology. Our result- ing dataset, GLEN(The GeneraL-purpose EveNt Benchmark), covers 3,465 event types over 208k sentences, which is 20x increase in the num- ber of event types and 4x increase in dataset size compared to the previous largest event extraction dataset MAVEN. We also show that dataset has better type diversity and the label distribution is more natural (Figures and 4).
BERT (Khattab and Zaharia, 2020), which is a very efficient ranking model based on separate encoding of the event type definition and the sentence. This stage allows reduce the number of type candi- dates for each sentence from thousands to dozens, retaining ~90 recall @ 10. In the last stage, for each trigger detected, we perform type classifica- tion to connect with one of the top-ranked event types from the first stage. We use joint encoder over both the sentence and the event type definition for higher accuracy.
In summary, our paper makes contributions in (1) introducing new event detection benchmark GLEN which covers 3,465 event types over 208k sentences that can serve as the basis for developing general- purpose event detection tools; (2) design- ing multi-stage event detection model CEDAR for our large ontology and annotation via distant supervision which shows large improvement over range of single-stage models and few-shot Instruct- GPT.
Related Work
We design multi-stage cascaded event detec- tion model CEDAR address the challenges of large ontology size and distant-superrisee data shown in Figure 2. In the first stage, we per- form trigger identification to find the possible trigger spans from each sentence. We can reuse the span-level annotations, circumventing the noise brought by our distant supervision. In the second stage, we perform type ranking between sentences and all event types. This model is based on Col-
As the major contribution of our work is introduc- ing new large-scale event detection dataset, we review the existing datasets available for event ex- traction and the various ways they were created.
Event Extraction Datasets ACE05 set the event extraction paradigm which consists of event detec- tion and argument extraction. It is also the most widely used event extraction dataset to date. The more recent MAVEN (Wang al., 2020) dataset has a larger ontology selected from a subset of FrameNet (Baker et al., 1998). FewEvent (Deng et al., 2020) is a compilation of ACE, KBP, and
"wikidata.o oog
'A roleset is a set of roles that correspond to the distinct usage of predicate.

---

Dataset
#Documents
#Tokens
#Sentences
#Event Types
#Event Mentions
ACE2005
599
303k
15,789
33
5,349
MAVEN
4480
1276k
49,873
168
118,732
Train
5607
3641k
187,468
3,464
184,806
Dev
311
204k
10,359
1720
9878
GLEN
Test
306
201k
10,627
1334
8290
Total
6224
4045k
208,454
3,465
205,0445
Table Statisties our GLEN dataset comparison to the widely used ACE05 dataset the previously largest event detection dataset MAVEN.
Wikipedia data designed for few-shot event de- tection. A number of datasets (DCFFE (Yang et al., 2018), ChFinAnn (Zheng al., 2019), RAMS(Ebner et al., 2020), WikiEvents (Li al.. 2021), DocEE (Tong et al., 2022)) have also been proposed for document-level event extraction, with a focus on argument extraction, especially when arguments are scattered across multiple sentences. Within this spectrum, GLEN falls into the category of event detection dataset with a heavy focus on wider coverage of event types.
3.1 Event Ontology and Data
The DWD Overlay is an effort to align WikiData Qnodes to PropBank rolesets, their argument struc- tures, and LDC tagsets. This mapping ensures that our ontology is superset of the ontology used in ACE and ERE (Song et al., 2015). (See Sec- tion 3.3 for a detailed comparison of the ontology coverage.)
To make this ontology more suitable for the event extraction task, remove the Qnodes lated cognitive events that do not involve any physical state change such as belief and doubt. We also discovered that many rolesets such as ill. 01 were heavily reused across the ontology (mainly due to the inclusion of very fine-grained types), therefore we manually cleaned up the event types that were associated with these rolesets. We show some examples of removed Qnodes in Ap- pendix Table 8.
Weak Supervision for Event Extraction Due to the small size of existing event extraction datasets and the difficulty of unnotation, prior work has at- tempted to leverage distant supervision from knowl- edge bases such as Freebase(Chen et al., 2017; Zeng et al., 2017) and WordNet (Araki and Mi- tamura, 2018; Tong et al.. 2020). The former is limited by the number of compound value types (only 20 event types are used in both works) and the latter does not perform any typing. Weak su- pervision has also been used to augment training data for an existing ontology with the help of a masked language model (Yang et al., 2019) or ad- versarial training (Wang et al., 2019). Our dataset is constructed with the help of the DWD Overlay mapping which is defined between event types (Qn- odes) and PropBank rolesets instead of on the level of concrete event instances (as in prior work that utilizes knowledge bases).
Since the DWD Overlay is aligned with Prop- Bank, we propose to reuse the existing PropBank annotations After the automatic mapping, each event mention in the dataset would be associated with one or more Qnodes, which leads to the partial label challenge when using this distantly- supervised data. We then perform another round of data filtering based on the frequency of rolesets (details in Appendix C). After these cleaning ef- forts, we used the annotation for 1,804 PropBank rolesets, which are mapped to total of 3,465 event types.
To make our data split more realistic and preserve the document-level context, we split the dataset into train, development, and test sets based on documents using a ratio of 90/5/5. Note that although our test set is only 5% of the full data, it is already similar in scale to the entire ACE05 dataset. For datasets such as OntoNotes and AMR that con-
3 The GLEN Benchmark
To build the GLEN benchmark, we first build general-purpose ontology based on the curated DWD Overlay and then create distantly-supervised training data based on the refined ontology (Section 3.1). In order to evaluate our model, we also cre- ate labeled development set and test set through crowdsourcing (Section 3.2).
'This line with scope of events as defined in previous ACE ERE datasets.
htttps: / thub. com/ httm///opbak//pooon/ prapppaa--abaalpeeeeee

---

come
3000
use
Types
125
2000
100
1500
taking office
75
1000
50
valuation
defcet
500
drag
75-
injection
dating
impicty
10"
10.
# Instances (log scale)
10"
0
300
1000
Sorted Index
1500
2000
2500
3000
2500
(a)
(b)
Figure 3: The event type distribution of GLEN. In the training set, instances associated with N labels are weighted as v. Figure (a) illustrates the distribution event types based on the number of instances. figure (b), we show the top three most popular event types as well as randomly sampled types from the middle and tail of the distribution curve.
tain documents from multiple genres (newswire, broadcast, web blogs etc.), we perform stratified sampling to preserve the ratio between genres. The statistics of our dataset are listed in Table 1. Com- pared with ACE05 and MAVEN, GLEN utilizes 20x larger ontology and 4x larger corpus.
social phenomena, chemicals, and other topics, in- dicating its novelty and potential usefulness.
We show the event type distribution in our dataset in Figure 3. Our type distribution closely mirrors real-world event distributions. The distri- bution exhibits frequent events such as come and use, along with long tail of rare events such as defect and impiety. In terms type diversity, for ACE, the single most popular event type at tack accounts for 28.8% of the instances and the top- 10 event types account for 79.4% % of the data. Our type distribution is much less skewed with the top-10 events composing 8.3% of the data.
3.2 Data Annotation
Instead of performing annotation from scratch, we formulate the annotation task as a multiple-choice question: the annotators are presented with the trigger word in context and asked to choose from the Qnodes that are mapped to the roleset.
Figure illustrates the part-of-speech distribu- tion of trigger words in our dataset?. Over 96% of trigger words are verbs or nouns, which similar to that 94% in MAVEN and 90% in ACE2005. In addition, 0.6% of the triggers are multi-word phrases.
For the test set, we hired graduate students with linguistic knowledge to perform annotation. For the development set, we screened Mechanical Turk workers that had high agreement with our in- in-house annotators and asked those who passed the screen- ing to participate in the annotation. The weighted average kappa value for exact match on the test set (27 annotators), is 0.60, while for the dev set (981 annotators) 0.37 over 5.2 options. If we allow for soft match for event types of different granu- larity, such as trade and international_trae the kappa value is 0.90 for the test set and 0.69 for the dev set. For more details on the annotation interface, see Appendix C.
4 Method
In the event detection task, the goal is to find the trigger word offset and the corresponding event type for every event. The main challenge for our dataset is the large ontology size and partial labels from distant supervision. To mitigate label noise, first separate the trigger identification step from the event typing step, since trigger identification can be learned with clean data. Then, handle the large ontology, we break the event typing task into two stages of type ranking and type classifica- tion to progressively narrow down the search space. Finally, in the type classification model, we adopt
3.3 Data Analysis
We first examine event ontology by visualiz- ing hierarchy (as shown in Figure 4) with the parent-child relations taken from Wikidata (the wverlay_parent field in DWD Overlay). Our on- tology offers a wider range of diverse events, in- cluding those related to military, disaster, sports,
We use universal POS tagging tools from https://www. nltk. nltk.org/. org/.

---

competiion
chemical process
meeting
milltary operation
disesteri
damage
compotiive debato
slection
congress
workshop
logal hearing
major trauma
property damage)
Injury
re-election
primary election
bone fracture
off
cold injuyy
1001: the entire branch is not covered by ACE2005
least one event type in this branch covered by ACE200S.
Figure 4: comparison between our ontology and that ACE. Our dataset offers broader coverage of events compared to ACE05, with diverse branches ranging from sports_compett tion to millitary.operation.
ADJ
The model is trained using binary cross entropy loss on all of the candidate spans in the sentence.
3%
VERB
NOUN
ADJ
NOUN 34%
VERB-PRT
4.2 Event Type Ranking
ADV
In the next stage, we perform event type ranking the entire event ontology for each sentence. Since our ontology is quite large, improve effi- ciency we make two design decisions: (1) ranking is done for the whole sentence and not for every single trigger; (2) the sentence and the event type definitions are encoded separately.
VERB-NOUN
VERO 62%
ADV-VERR
VERB-VERB
Other
Figure 5: The Part-of-Speech distribution of trigger words for events in GLEN. Multiple POS tags mean that the trigger has multiple words.
We use the same model architecture Col- BERT (Khattab and Zaharia, 2020), which is efficient ranking model that matches the perfor- mance of joint encoders while being an order of magnitude faster.
a self-labeling procedure to mitigate the effect of partial labels.
4.1 Trigger Identification
In the trigger identification stage, the goal is to identify the location of event trigger words in the sentence. This step only involves the sentence and not the event types. We formulate the problem as span-level classification: compute the probabil- ity of each span in sentence being an event trigger, we first obtain sentence token representations based on a pre- trained language model:
We first encode the sentence as:
PLM[[CLSJJSENT]S] ... (n[SEP) E [(++1) h h." Norm(ldConv(5)) E Ro
(1)
[SENT] is a special token to indicate the input type. The dieeddmenniioall convolution operator serves a pooling operation and the normalization oper- ation ensures that each vector in the bag of embed- dings h2 has an L2 norm of
[s] S.. s,,'
(I ... (,(SEP)) E m
The event type definition is encoded similarly, only using a different special token [EVENT].
where s; a h-diimensionl representation vector of token
Then the similarity score between a sentence and an event type is computed by the sum of the maxi- mum similarity between the sentence embeddings and event embeddings:
Then we compute the scores for each token being the start, end, and part of an event trigger individu- ally
fu(s:) wIs:, D E (start, end, part),
P(e.s)
=Σ max
(2)
he
where wD h is learnable vector. We then compute the probability of span [s) to be an event trigger as the sum of its parts:
Our event type ranking model is trained using the distant supervision data using margin loss since each instance has multiple candidate labels. This margin loss ensures that the best candidate is scored higher than negative samples.
p[[si.... .......... s:])
= =fffann fena(s8) + Σ
ΣΣ max (0, max Pie.e) + P(e-a))) (3)
eEGs
k=

---

e denotes negative samples, Cy the set of candi- date labels and is a hyperparameter representing the margin size.
the top-K ranked event types for the event trigger (the event trigger span needs to be correct). This can be seen as more lenient version of trigger classification FI.
4.3 Event Type Classification
Baselines We compare with three categories of models: (1) classification models including DM- BERT (Wang et al., 2019), token-level classifica- tion and span-level classification; (2) a definition- based model ZED (Zhang et al., 2022) and (3) InstructGPT(Ouyang et al., 2022) with few-shot prompting. We attempted to compare with CRF models such as OnelE model (Lin et al., 2020) but were unable do so due to the memory cost of training CRF with the large label space. For de- tailed bascline descriptions and hyperparameters. see Appendix B.
Given the top-ranked event types from the previous stage and the detected event triggers, final step is to classify each event trigger into one of the event types. Similar to (Lyu et al., 2021), we formulate this task as a Yes/No QA task to take advantage of pre-trained model knowledge, The input to the model is formatted as "(type) is defined as ini tion). (sentence). Does (trigger indicate a (type) event? [MASK]". We directly use pretrained masked language model and take the probability of the model predicting "yes" or "no" for the [MASK] token, denoted as (M)M(W), where w E {"yes", "nno".. From these probabilities, we calculate the probability of the event type as follows:
5.2 Results
We show the evaluation results in Table 2 and some example predictions in Table 3. The first group of baselines (DMBERTTTO&CCISSpaCC)) are all classification-baase which means that they treat event types as indexes and do not utilize any seman- tic representation of the labels. We observe that DMBERT will have a substantially lower trigger identification score if we allow spans of more than token to be predicted due to its max pooling mecha- nism (it will produce overlapping predictions such as "served". "served in", "served in congress"), ) TokCls's performance hints on the limit for learn- ing only with partial labels. As shown in the ex- ample, TokCls usually predicts event types that are within the candidate set for the roleset, but since has no extra information to tell the candidates apart, the prediction is often wrong.
exp( (MLL("yes")) P(e) uxp(PMIM("yess) + exp( exp(pMMMMMee (MMLM""O""))
To train the model, we employ binary cross-entropy loss across pairs of event triggers and event types. As mentioned in Section 3.1, our data contains label noise due to the many-to-one mapping from Qnodes PropBank rolesets. We adopt incre- mental self-labeling procedure to handle the partial labels. We start by training base classifier on clean data labeled with PropBank rolesets that map to only one candidate event type. Despite being trained on only a subset of event types, the base model exhibits good generalization to rolesets with multiple candidate event types. We then use the base classifier to predict pseudo-labels for the noisy portion of the training data, selecting data with high confidence to train another classifier in conjunction with the clean data.
Although ZED utilizes the event type definitions, it only achieves minor improvement in perfor- mance compared to TokCls. ZED employs mean pooling to compress the definition embedding into a single vector, which more restricted compared to the joint encoding used by our event type classi- fication module.
5 Experiments
5.1 Experiment Setting
Evaluation Metrics Previous work mainly uses trigger identification F1 and trigger classifica- tion F1 to evaluate event detection performance. An event trigger is correctly identified if the span matches one of the ground truth trigger spans and it is correctly classified if the event type is also correct. In addition to these two metries, due to the size our ontology, we also report Hit@k K which measures if the ground truth event type is within
We observe that InstructGPT with in-context learning does not perform well on our task. The low trigger identification scores might be attributed to the lack of fine-tuning and under-utilization of the training set. The low classification scores are mainly caused by the restriction in input length which makes it impossible to let the model have full
"The maximum input length was 2049 tokens at the time of our experiments.

---

Trigger Identification
Trigger Classification
Hit@k
Model
Prec
Recall
F1
Prec
Recall
F1
Ili@
Hii@2
Ili@5
DMBERT (Wang et al., 2019)
56.87
84,32
67.93
32,93
48.84
39.34
Token Classification Span Classification
68.04
82.19
74,46
41.48
50.10
45.38
62.36
78.71
69.58
37.36
47.16
41.69
TokCls ZED (Zhang et al., 2022)
40.01
56.25
46.76
56.84
60.35
60.80
InstructGPT (32 shot)
28.41
42.30
33,99
11.76
17.52
14.08
CEDAR w/ self-labeliig CEDAR
49.21
61.62
54.72
68.21
80.21
89.18
71.05
88.96
79.00
50.91
63.74
56.60
71.30
80.84
88.63
Table Quantitative evaluation results (%) for event detection on GLEN.
Context
Predictions
CEDAR: payment(transfer of an item of value)
You do pay income tax your paycheck and the sales tax consumable goods?
TokCls: disbursement (payment from public fund)
ZED: income (consumption and savings opportunity)
GPT3: None
CEDAR: discovery (detecting something new)
... situation hrought on mostly by the police where information is discovered way that perpetuates the story
TokCIs: medical finding
7ED: physpcal_findeeeeecppc (from physical examination of patient)
GPT3: discovery (detecting something new)
CEDAR reporting (producing a oral or written report)
40% of the female students Georgetown law
TokCls: scoop (jouralissm term for a new interesting story)
reported to that they struggle financially result of this policy.
ZED: reporting (producing a oral or written report)
GPT3: None
Table 3: Comparison across different systems. Correct predictions are shown in green. Predictions that map to the same PropBank roleset are shown in orange. For ZED, we show the TokCls+ZED variant which has better performance.
knowledge of the ontology. As result, only 57.8% of the event type names generated by InstructGPT could be matched in the ontology. With a larger input window and possibly fine-tuning, we believe that large LMS could achieve better performance.
Component
Hit@k
Hit@ 10 89.86
Hit@ 20 93.52
Hit 50
Type Ranking
95.44
Type Classification (ground truth in top-10)
Hit@ 78.70
Hit@2 89.37
Hit@5
97.76
For our own model, we show that decoupling trigger identification from classification improves TI performance, and performing joint encoding of the definition and the context improves TC perfor- mance. Furthermore, using self-labeling can help improve top-1 classification performance by con- verting partial labels into clean labels.
Table 4: Evaluation for type ranking and classification separately. The scores for type classification component are computed over the subset of data where the ground truth event is among the top-10 10 ranked results.
type classification. Table shows precision, re- call, and Fl scores for trigger identification, while Table 4 presents the performance of event type ranking and classification. We assess per-stage performance using Hit@k metrics for event type ranking on ground truth trigger spans and event type classification ground truth event type appears in the ranked sults by the type ranker. The scores indicate that the primary bottleneck exists in the precision of trigger identification and the Hit@ score of type classification.
6 Analysis
this section, we investigate the following ques- tions: (1) Which component is the main bottleneck for performance? (Section 6.1) and (2) Does our model suffer from label imbalance between types? (Section 6.2)
event mentions where the
6.1 Per-Stage Performance
Our model CEDAR comprises three components: trigger identification, event type ranking, and event

---

Plastunces
Trigger Classification F1
Most of our errors come from the noisy anno- tation (Candidate Set): our model can predict an event type that falls within the set of candidate types associated with the ground truth PropBank roleset but fails to find the correct one. Extended Roleset refers to the predicted event being associ- ated with roleset that shares the same predicate as the ground truth. The uncategorized errors are often due to the imperfect recall of our event rank- ing module (as in the second example Table 5 where the context is long and "fund" fails to be included in the top-10 ranked event types), or cases where our model prediction is related semantically to the ground truth but the event types have no con- nections in the hierarchy (as in the first example in Table 5 where we predicted "quantification" in- stead of "measurement"). On the other hand, in another 22.9% of the cases, we predict an event that close to the ground truth on the XPO hicrarchy, with the ground truth either being the child (Child), parent (Parent), or sibling node (Sibling) of our predicted type. This suggests that better modeling of the hierarchical relations within the ontology might be useful for performance improvement.
100000
100
52500
75
66.10
$2.35
S5000
47,50
50
27800
25
0
3
Group
Figure 6: Relationship between the number of instances of different event types in the dataset and our model's s performance. The event types are divided into four groups based on their frequencies.
6.2 Type Imbalance
Figure 3 illustrates the long-tailed label distribution in our dataset. To investigate whether our model is affected by this imbalance, we divided the event types into four groups separated at the quartiles based on their frequency and calculated the perfor- mance per group. The resulting figure shown in Figure While we do see that the most popu- lar group has the highest Fl score, the remaining groups have comparable scores. In fact, we see two factors at play in defining the dataset difficulty: the ambiguity of the event type and the frequency of the event type. Event types that more popular are also often associated with rolesets that have a high level of ambiguity, which balances out the gains from frequency.
7 Conclusions and Future Work
We introduce general-purpose event detec- tion dataset GLEN that covers over 3k event types for all domains. GLEN can be seen as our first attempt towards making event extraction systems more accessible to the general audience, removing the burden on users of designing their own ontol- ogy and collecting annotation. We also introduce multi-stage model designed to handle the large ontology size and partial labels in GLEN and show that our tailored model achieves much better per- formance than a range of baselines.
6.3 Error Categories
Candidate Set
70.2%
Our model CEDAR can be used as an off-the- shelf event detection tool for various downstream tasks, and we also encourage members of the com- munity to improve upon our results, (e.g. tackle the noise brought by partial labels) or extend upon our benchmark (e.g. include other languages).
Unrelated
77.0%
Sibling
12.7%
19.6%
Child
10.2%
Other
Parent
Extended Candidate
Figure 7: Categorization of errors based on the relation between the predicted event type and the candidate set produced by the mapping (left) and the predicted event type and the ground truth event type on the event ontol- ogy hierarchy (right).
8 Limitations
We perceive the following limitations for our work:
Lack of argument annotation. Event argument extraction is a critical component of event ex- traction. At the time of publication, GLEEN is only an event detection dataset without argu- ment annotation. This is an issue that we are
We categorize the type classification errors from the CEDAR model based on the relationship be- tween our predicted event type and the ground truth event type as shown in Figure 7 and Table 5.

---

Error Category
Context
Predicted
Gold
3 workers off critical reaction in 990000 when they poured too much uranium into precipitation lank.
reaction (response
Candidate
chemical
stimulus)
France is urged to increase research funding and support for innovations a deal with the problem.
research (systematic study)
research_metlee
settlement(operaiion cetteementtiistortioonn of building)
Extended Can-
Regulators and research firms promised that the $1.5 bil- lion settlement would be finalized months ago.
relating to the pay- ment)
didate
work (activities per- formed as means support)
work (activity done by a person for cco- nomic gain)
People working for minimum wage producing large number of products services.
Child
the middle conflict, you think the United States should Israel's side, take the Palestinians' side, or not either side?
social conffic (struggle ageney power in society)
armed_comee conflict flict including vio- lence)
Sibling
poysicll_xaminnte (process by medical professional)
Doctors will examine him for signs that the cancer may have come back while he awaiting trial a Russian jail.
Parent
inspection
wear (damaging. gradual removal or deformation)
Other
Ironically, in worn slogan on right.
"character matters" became well-
wears(clothing accessory)
or
In thenry, one could argue that the computer models are accurate and that the real measurements have some prob-
quantification
measurement
lems.
Though funds have already been allocated and voted on for the project, Blair himself insists that things are still "very much open"...
voting
fund
Table 5: Examples of erroneous type predictions. The trigger word (phrase) is shown in bold. In some cases, the error falls into multiple categories. We prioritize the XPO hierarchy-related categories since they are rarer.
actively working towards for future releases of the dataset.
glish. One idea would be to utilize resources such as the Universal PropBank (Jindal et al., 2022)which can help us align corpora in other languages to PropBank and then further to our ontology.
Timeliness of documents. The source doc- uments of the PropBank annotated datasets are not very new. In particular, the OntoNotes dataset contains news articles from 2000-2006 Hence, there is the possibility of a data dis- tribution shift between training set and any document that model is being applied, which might cause performance degrade, as shown in other tasks (Rijhwani and Preotiuc- Pictro, 2020).
A Acknowledgements
This research based on work supported by U.S. DARPA KAIROS Program No. FA8750-19-2. 1004. The views and conclusions contained herein are those of the authors and should not be inter- preted as necessarily representing the official poli- cies, either expressed implied, of DARPA, the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copy- right annotation therein.
Completeness of ontology. Although our dataset is the most comprehensive event de- tection dataset of date, we acknowledge that there might be event types that we have over- looked. We plan to keep GLEN as a living project and update our ontology and dataset over time.
References
Jun Araki and Teruko Mitamura. 2018. Open-domain event detection using distant supervision. Pro- ceedings of the 27th International Conference on Computational Linguistics, pages 878-89) Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Multilingual support. Currently, our docu- ments are from the English PropBank annota- tion dataset so our system only supports En-
"https://catalog. upennc.peeeeeeem ettemmeeu/LDD2813311

---

Collin F. Baker, Charles J. Fillmore, and John B. Lowe, 1998. The Berkeley FrameNet project. In 36th An- nual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume I. pages 86-90, Montreal, Quebec, Canada. Association for Compu- tational Linguistics.
Meeting of the Association for Computational Lin- guistics, ACL 2020, Online, July 5-10, 2020, pages 7999-8009. Association for Computational Linguis- tics.
Qing Lyu, Hongming Zhang, Elior Sulem, and Dan Roth. 2021. Zero-shot shot event extraction via transfer learning: Challenges and insights. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1ith International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 322 2323 Online. Association for Computational Linguisties.
Yubo Chen, Shulin Liu, Xiang Zhang. Kang Liu, and Jun Zhao. 2017. Automatically labeled data genera- tion for large scale event extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). Vancouver, Canada. Association for Computational Linguistics.
pages
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, S
Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, Abs/2203.02155
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi- pooling convolutional neural networks. In Proceed- ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Interna- tional Joint Conference on Natural Language Pro- cessing ( (Volume 1: Long Papers), pages 167-176, Beijing. China. Association for Computational Lin- guistics.
Martha Palmer, Paul R. Kingsbury, and Daniel Gildea. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31:71- 106.
Shumin Deng, Ningyu Zhang, Jiaojian Kang, Yichi Zhang, Wei Zhang, and Huajun Chen, 2020. Meta- learning with dynamic-memen memommm based prototypical network for few-shot event detection. In Proceedings of the 13th International Conference on Web Search and Data Mining. ACM.
Shruti Rijhwani and Daniel Preotiuc-Pietro 2020. Temporally-inforun analysis of named entity recog- nition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 7605-7617 Online. Association for Computational Linguisties.
Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence ar- gument linking. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 8057-8077, Online. Association for Computational Linguistics.
Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. 2015. From light to rich ERE: Annotation of entities, relations, and events. In Proceedings of the The 3rd Workshop EVENTS: Definition, Detection, Coreference, and Representation, pages 89-98, Denver, Colorado. As- sociation for Computational Linguistics.
Ishan Jindal, Alexandre Rademaker, Michat Ulewiez, Ha Linh, Huyen Nguyen, Khoi-Nguyen Tran, Huaiyu Zhu, and Yunyao Li. 2022. Universal proposition bank 2.0. In Proceedings of the Language Resources and Evaluation Conference, pages 1700-1711, Mar- seille, France. European Language Resources Asso- ciation.
Elizabeth Spaulding, Kathryn Conger, Anatole Gersh- man, Rosario Uceda-Sosa, Susan Windisch Brown, James Pustejovsky, Peter Anick, and Martha Palmer. 2023. The DARPA Wikidata Overlay: Wikidata as ontology for natural language processing. In Pro- ceedings of the 19th Joint ACL ISO Workshop on Interoperable Semantic Annotation (ISA-19), pages 1-10, Nancy, France.
Khattab and Matei A. Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
Meihan Tong, Bin Xu, Shuai Wang. Yixin Cao, Lei Hou, Juanzi Li, and Jun Xie. 2020. Improving event detec- tion via open- domain trigger knowledge. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5887-5897, On- line. Association for Computational Linguistics.
Sha Li, Heng Ji. and Jiawei Han. 2021. Document level event argument extraction by conditional generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional I. Linguistics: Human Language Technologies, pages 894-908, Online. Association for Computa- tional Linguistics.
MeiHan Tong, Bin Xu, Shuai Wang, Meihuan Han, Yixin Cao, Jiangqi Zhu, Siyu Chen, Lei Hou, and Juanzi Li. 2022. DocEB: A large-scale and fine- grained benchmark for document-level event extrac- tion. In Proceedings of the 2022 Conference of the
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. joint neural model for information extraction with global features. In Proceedings of the 58th Annual

---

North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- pages gles.pages 3970-3982, Seattle, United States. Asso- ciation for Computational Linguistics.
Xiaozhi Wang, Xu Han, Zhiyuan Liu, Maosong Sun, and Peng Li. 2019. Adversarial training for weakly supervised event detection. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pages 198-1008, Minncapolis, Min- nesota. Association for Computational Linguistics.
Xiaozhi Wang. Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan L Liu, Juanzi Li, Peng Li, Yankai Lin, and Zhou. 2020. MAVEN: Massive General Domain Event Detection Dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1652- 1671, Online. Association for Computational Linguis- tics.
Hang Yang. Yubo Chen, Kang Liu, Yang Xiao, and Jun Zhao. 2018. DCFEE: A document-level Chi- nese financial event extraction system based on au- tomatically labeled training data. In Proceedings of ACL 2018, System Demonstrations, pages 50-55, Melbourne, Australia. Association for Computational Linguistics.
Sen Yang. Dawei Feng, Linbo Qiao, Zhigang Kan, and Dongsheng Li. 2019. Exploring pre-trained language models for event extraction and generation. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5284- 5294, Florence, Italy. Association for Computational Linguistics.
Ying Zeng, Yansong Feng, Rong Ma. Zheng Wang. Rui Yan, Chongde Shi, and Dongyan Zhao. 2017. Scale up event extraction learning via automatic training data generation. In AAAI Conference Artificial Intelligence.
Hongming Zhang, Wenlin Yao, and Dong Yu. 2022. Efficient zero-shot event extraction with context- definition alignment. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7169-7179, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. 2019. Doc2EDAG: An end-to-end document-level frame- work for Chinese financial event extraction. In Pro- ceedings of the 2019 Conference Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (FMNIP-IICNNI.) pages 337-346, Hong Kong, China. Association for Computational Linguis- tics.

---

A Implementation Details
an instruction for the task and 32 training ex- amples from our training set for in-context learning.
Table presents the hyperparameters for three com- ponents in our model. During trigger identification, we restrict token spans to maximum length of 10 tokens. For event type ranking. we use de- signed loss function with = 1.0. The top 10 ranked events are selected candidates for event type classification. In event type classification, we do one round of self-labeling. Our model trained on a single Tesla P100 GPU with 16GB DRAM.
We list the hyperparameters for our baseline models in Table 7. For ZED, we follow the original paper and set the margin to 0.2. We set the thresh- old for predicting an event type to 0.3 (if the cosine similarity between the event type representation and the trigger representation is smaller than this value, we will refrain from predicting any event type).
Component
TI
ETR
ETC
For the InstructGPT baseline, we use the text-davinci-00 model with a temperature of 0.2 and top_p set to 0.95 for decoding, We show our detailed prompt in Figure 8. The first part of the prompt is the task instruction and then we in- clude 32 input-output examples. Due to the current input length limit of InstructGPT, we were unable to feed the ontology into the model as part of the input.
Training epachs
5
2
Batch
128
64
32
Max sequence length
128
128
512
Base Model
bert-base-uncased
learning rate
lc-5
Weight decay
0.01
Scheduler
Linear (with warmup steps)
Table 6: The training hyper-parametert of our model.
TI: Trigger Identification. ETR: Event Type Ranking. ETC: Event Type Classification.
C Data Filtering and Annotation Details
B Baseline Implementation Details
Table 8 shows some examples of removed Qnodes from DWD Overlay in our ontology with different kinds of reasons.
We list the details of our baselines below:
1. Token Classification: We use the 10 tagging scheme to classify tokens.
To improve dataset quality, we perform sentence de-duplication, remove sentences with less than 3 tokens and omit special tokens (marked by * or brackets). We ensure that every trigger a continu- ous token span and we remove events with overlap- ping triggers. For the AMR dataset, we additionally remove triggers with part-of-speect tag of MD (modal verbs) or TO (the word "to"") (such cases do not appear in other datasets).
2. Span Classification: We use the embedding of the first and last token to represent the span for classification.
3. DMBERT (Wang et al., 2019) is BERT- based model that applies dynamic pool- ing (Chen al., 2015) according to the can- didate trigger's location. We consider single tokens to be candidate triggers in our dataset during testing.
Based on this distant supervision dataset, we make further adjustments to the ontology. We man- ually inspected the most popular rolesets that have more than 1000 event mentions and removed role- sets that are too general or ambiguous (for instance, cause. 01 and see. 01). Finally, remove the rolesets that have less than event mentions across all datasets.
4. ZED (Zhang et al., 2022) is an event detec- tion model that utilizes definitions. Instead of pre-training on WordNet, we train the model on our noisy training data. As ZED only per- forms classification, report the results with the trigger spans predicted by the token classi- fication model (TokCIs+ZED).
Figure 9 displays our annotation interface. The left box features the context, highlighting single trigger, while the right box enumerates candidate event types, expressed as combination of name and description, with an extra choice labeled "None of the above options is correct." " Each Qnode is rep- resented by its name and description. The number of options varies from 2 to 9 based on the ontology.
5. InstrnetGPT(Ouyann al., 2022), also re- ferred to as GPT-3.5, the improved version of GPT3 trained with instruction-tuning. We use the text-davinci-003 model through the OpenAI API. We provide the model with

---

Model
DMBERT
SpanCIs
TokCls
ZED
Värgānia before noving Training epochs
10
10
10
10
Batch size
64
64
16
16
Negative samples
5
5
Leamingg rate
5o-5
5c-5
2c-5
2c-5
Max sequence length
64
Base Model
Bert- bantbbbs--
Weight decay
Scheduler
Lincar
Table 7: Hyperparameter settings for baseline models.
List the events that are mentioned in the given context and highlight the most related word or phrase using the <mark> tag for each event. There might be more than one event per context provided or no event the context.
For each tagged event, also generate its corresponding event type.
Context: average yield on
jumbas rose 8.80
two year average
fell the amount 7.89
Output: average yield threemonth jumbas <mark>roses/markk
year average <narkeeell</mark- by the same amount 7.89
rase: rose:clinb; :ese:nt::
Context: i also think its stupid that our european friends can innigrate to britian and seek benefits and denying them that would against human rights .... according the of rights
Output: 1 also think its stupid that our european friende can mark--immorrtte</mark britian and seek benefits and enarkodenying-/mare them that would be <markeagainsteemmar human rights according to the eu court of human rights
Evente:
immigrate thwmah. wwggatainnndnnynnnnnee :dnl aganstaatt discrimination;
Context: Nost of the 34 sailors hurt are being reunited with family in Norfolk on to naval hospital for further treatment
Output: Nost of the 34 sailors <nark>hurt</marks are being reunited with family in Norfolk Virginia
before smarkemovinge/marks on to naval hospital for further treatment
Events:
durt:damage:n :ww.ing:acttieemmooooooooo motion:
Figure 8: Truncated version of our prompt to InstructGPT.
The annotator's task is to select the option that most accurately represents the trigger word.
Each instance annotated by two annotators separately. For PropBank rolesets that were fre- quently labeled "None of the above". per- formed manual inspection determine the map- ping should be removed or revised. Finally, for the affected instances and the instances with disagree- ment, we asked our in-house annotators to perform a third pass as adjudication.

---

Quode
Name
Roleset
Description
Removed during cleaning the heavily used rolesets
Q2536390
abdominal distention
ilL01
Physical symptom
Q192989
acculturation
change.01
process of cultural and psychological change
Q422268
actinomycosis
ilL.11
Human disease
Q1319035
adult education
educate.()
form of learning adults engage in beyond traditional schooling metalworking
Q9363879
stamping
make.01
Q615857
stapedectomy
surgery.01
surgical procedure of the middle car performed to improve hearing surgical removal of the adrenal gland Medical procedure
Q366774
adrenalectomiy
remove.01
Q2035485
subcutancous, _injection
inject.01
Removing reason:
cognitive events that do not involve any physical state change
Q241625
wish want yearning intention
wish.0r
desire for a specific item or economic term for something that desired
Q26256512
want.0
Q26253999
yearn.01 01
deep and aching desire someone or something mental state representing commitment to perforn an action process by which two closely related linguistic varieties diverge from one another during their evolution
Q706622
intend.01
Q3027692
differentiation
differentiate.01
Q104776298
crosspatch suspicion trust
grouch.01
a person who easily annoyed emotion
Q516519
suspect.01
Q659974
trust.02
assumption and reliance on the honesty of another party
Removing reason: mapped too general or ambiguous rolesets
Q105606485
intellectual activity
think.01
human activity comprising of mental actions
Q2944236
photosensitivity religion
see.01
Light sensitivity in homo sapiens
Q9174
believe.01
set of beliefs, practices and traditions for a group community result of deliberation
Q16513426
decision
decide.01
Q2827815
international_n _id knowledge
give.01
voluntary transfer of resourees from one country to another experience or education by perceiving, discovering, learning agrcement between employer and employee terms of work and compensation
Q9081
know:01
Q1221208
employment_come contract
agrec.01
act of intentionally focusing visual perception someone or something
Q56274009
looking
look.01
Removing reason:
low frequency
Q379788
advection
advect.1
transport of substance by bulk motion
Q381105 105
acration
acrate.01
process circulating or mixing air with water
colloid of fine solid particles or liquid droplets, air or another gas
Q104541
acrosol
aerosolize.01
Q623179
state terrorism stenciling
terrorism.03
acts of terrorism against individuals conducted hy organs of a state
Q98394474
stencil.01
artistic technique for transfcrring images using stencils
Q844613
sintering
sinter.01
process of forming material by heat or pressure speech in praise a person, usually recently deceased boundary between different phases of matter
Q249697
eulogy
eulogize.00 01
Q901882
interface
interface.01
Table 8: Examples of removed Qnodes in XPO Overlay. Note that one node can be removed due to multiple reasons.

---

D Impact of Self-labeling
Roleset Category
Clean
Covered
Other
Total
#instances
3642
3308
1223
8173
@it@| I before
95.74
48.52
37.37
67.89
Hit@1 after
95.47
54.59
39.17
70.50
Table Hit@1 I scores before and after self-labeling different categories of PropBank rolesets. Clean: role- sets with only one candidate labels. Covered: rolesets covered in the self-labeled training data.
Figure 10 indicates that with a threshold of 0.9, the accuracy of selecting the correct label from a candidate set reaches 57.8% on the dev set. To investigate how self-labeling contributes to the improvements, we categorize test instances into three groups based their PropBank role- sets, shown Table 9. The 'Clean' rolesets map to only one event type DWD Overlay. We train the base classifier on 71,834 training instances corresponding to these 'Clean' rolesets, which nat- urally performs significantly well on this portion of data. The model after self-labeling is trained with additional 25,549 self-labeled data. The role- sets corresponding to these data categorized as "Covered". Table indicates that the main per- formance gain comes from the "Covered" data, which is boosted directly by including correspond- ing training data. The "Other" category also sees some improvement, at the cost of a slight drop in the "Clean" category.
1"The threshold is the margin between the probability of the top I event type and the other types in candidate set. higher threshold means a higher level of confidence.

---

Instructions
Shortcuts
matching meening highighted
Select an option
Reports that Indian Minister Chemicals, Fertilizer and Steel, Ram Paswan hopes to implement price control on more drugs Paswan hopes to make a one time price reduction on the 354 essential drugs designated by the WHO -LRB- including in them some hypertension drugs and antibiotics -RRB- first through negotiations and then connect future increase of drug prices with infiation. Currently India has total of 74 price controlled drugs, which will be increased to an amount of more than 40 % of the drugs sold by India. FICCI Vice President Habil Khorakiwala said that if price control measures promulgated. even large drug makers will find hard maintain their budget for research and development drugs Piramal, chairman of drug maker Nicholas Piramal, said. The continuously falling prices of common export drugs resulted thin profits, they ca n't impose limits on profits any
social, control: concept within disciplines the social sciences and within political science
damage canaaeecontrol:: in maritime context, emergency control of situations which may cause the sinking of a ship
riot control:
used by police, military, or other security forces during riot
risk control: part the
Submit
Figure 9: Annotation interface built with Amazon Mechanical Turk for labeling the development and test set.
Accuracy
IInttances
0.7
6000
0.625
4500
0.578
0.55
3000
0.475
1348
1500
0.4
Relationship between the threshold and two metrics: the accuracy of label selection dev and the number of selected instances. The x-axis represents the threshold.
Figure 10:


Reviews:
[{"review_id": "oMSJp8RDQc", "reviewer": "Reviewer_7vXC", "summary": "", "questions": "-  Is the manually cleaned up of events fime consuming? (line 128)\n- Do you have analysis on error propagation? The multi-stage approach compared with end-to-end approach would suffer from error propagation more\n- Have you tried with concatenate the sentence and event description? The current approach is using different embeddings for these two and then calculate similiarty, it might cause alignment issue", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our effort in creating GLEN: including the ontology, the dataset, and the corresponding system. \n\nWe would like to clarify the differences between GLEN and the existing DWD ontology. The DWD ontology is a human-curated subset of WikiData nodes, each entry in this ontology includes properties from WikiData (name, definition, overlay_parent and similar_nodes), properties from PropBank (argument names) and additional mappings to LDC event types (ldc_types). The DWD ontology can be seen as a dictionary,  but we believe it would be quite a stretch to say that the dictionary itself is an “annotation method”. We will add more description about the DWD ontology in Section 2.1 to make this more clear. \n\nWe used Figure 3 mainly to display the long-tail property of the dataset, showing the relationship between the frequency of the event type and the rank of the event type, as in Zipf’s law. \nA histogram would also be good for showing the dataset distribution and while we cannot attach figures here, we can add one in our Appendix during revision. \n\nRegarding the dataset split, since our dataset is much larger than ACE or ERE, having a higher proportion of test/dev instances would require much more human annotation. Although we simplified the annotation task by reusing the trigger spans, our experience with MTurk was still quite bad initially and we had to filter many annotators, redo part of the annotation, and then ask in-house annotators to adjudicate when there was disagreement. Such issues made it quite difficult to scale up the test/dev instances.\n\nTo address your questions: \n\n**For the cleanup procedure mentioned in Line 128**: \nin terms of heavily reused rolesets, we manually inspected 47 rolesets from Propbank corresponding to more than 10 DWD Overlay nodes. These nodes were either discarded or relabeled. One of our authors did an original pass of the labeling and then we asked some of our in-house annotators to check. A single pass took a few hours to complete. \n\n**Analysis of error propagation.** \nWe provided per-stage results in Table 2 and Table 4. The three stages of our method lead to three possible sources of errors: (1) trigger detection errors, which is shown in Table 2 Trigger Identification stage; (2) type ranking errors, as shown in Table 4 row 1; (3) type classification errors, as shown in Table 4 row 2. Among these three types of errors, trigger identification is the most prominent source. Although our pipelined model does have some error propagation issues, this does not put us at a disadvantage compared with the single stage classification baselines as they show even lower trigger identification scores.  (ZED is also a pipelined method.)\n\n**Concatenation of the sentence and the event description.**\n As mentioned in Section 3.3 Line 282-285, the third stage of our model, the type classification stage, takes the concatenated event description and sentence as input. “Using separate embeddings for these the event and the sentence and then calculate similarity” is only for our event ranking stage (Line 248-266). We agree with your intuition that separate encoding usually does not lead to the best performance, but it is more efficient, which is exactly the motivation behind our multi-stage approach (Line 143-147). We did not directly compare against \"skipping type ranking and only using type classification\" because it would require encoding each sentence 3500 times (for each trigger) to perform inference, which would be prohibitively slow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- contribute directly to the event detection system by releasing a new large scale dataset with 3465 fine-grained event types and 205k event instances\n- proposed new method of event detection\n- although annotation method is an already existing method, \"DWD Overlay\", the authors have their filtering of the event types and efforts on collecting and cleaning the data\n- ontology is a superset of ERE and ACE, which is an advantage for fair comparison", "reasons_to_reject": "- the annotation method is a previous work \"DWD Overlay\", but the authors have explained the difference \n- the quality of dataset is not explicit, if possible, a histogram of distributtion of data types will be more helpful than Figure 3", "questions_to_authors": "-  Is the manually cleaned up of events fime consuming? (line 128)\n- Do you have analysis on error propagation? The multi-stage approach compared with end-to-end approach would suffer from error propagation more\n- Have you tried with concatenate the sentence and event description? The current approach is using different embeddings for these two and then calculate similiarty, it might cause alignment issue", "paper_topic_and_main_contributions": "This paper build a new event detection dataset named as GLEN using distant supervision and based on the DWD Overlay method to automatically map Wikidata to PropBank rolesets so that the event triggers in sentences can be automatically annotated. Apart from building the new dataset, the authors also proposed a multi-stage event detection method.\n\nThe new dataset address the problem that there lack large-scale, diverse event detection dataset, the proposed approach is designed to address that difficulties met when working on this new dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 481, type \"toward\" should be \"towards\""}, "extra_text": {}}, {"review_id": "rqCF1BO4Pd", "reviewer": "Reviewer_rTom", "summary": "", "questions": "* A. How does the removal of cognitive Qnode events make the ontology \"more suitable for the event extraction task\" [§2.1]?\n\n* B. What's the motivation for removing general / ambiguous PropBank rolesets like cause.01 and see.01 [§C, §2.1 \"another round of data filtering\"]? Is that there would be too many options for annotators to choose among?\n\n* C. Does the error analysis for type classification [§5.3] refer exclusively to event triggers in ground truth data? If so, it would seem to me that error categories 'candidate set' (e.g., have trigger 'working' for roleset 'work.01', and predict wrong Qnode for 'work.01') and 'extended roleset' (e.g., for same example, predict a Qnode for 'work.XY' with XY != 01) should cover all possible error scenarios, as the trigger should constrain the possible Qnodes that can be picked for event classification (via 'trigger' -> possible rolesets for trigger -> possible Qnodes for such rolesets). Is it that Qnodes unrelated to trigger possible rolesets are considered as candidates following type ranking? Anyhow, it seems to me that error classes 'child', 'parent', 'sibling', and possibly 'unrelated', may form another orthogonal way to classify errors interesting fro practical applications (e.g., a user may still be satisfied if a child type is wrongly chosen since the parent type can be still inferred).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments. We are glad that you find our dataset useful, our experiments informative and our paper well-written. \n\nWe would like to provide our response to your mentioned weaknesses and questions as follows:\n\nW1: The GLEN dataset currently does not include argument roles and this is why we refer to it as a “General-Purpose Event Detection Dataset” instead of “Event Extraction Dataset” and we only state our contributions to event detection (Line 97-104). This is the same as MAVEN, which is meant to be a “A Massive General Domain Event Detection Dataset”. We do agree that argument extraction is an indispensable part of events and we are working on extending GLEN to cover arguments. We will make the distinction between event extraction and event detection more clear in our introduction and state the scope of our work more explicitly. \n\nQA: Following the convention of the most popular event extraction datasets ACE and ERE, we focus on events that are action-related and result in state changes in the physical world. Cognitive events only result in changes of the mental state and are more related to the opinion of the subject, which is related to other tasks such as stance detection. \n\nQB: This final round of filtering was done in a very restricted way, and on a case-by-case basis through discussion. We took the list of the most frequent rolesets and went over the examples to see if these rolesets were suitable to serve as event types. `see.01` was overloaded and it was used for both the physical sense of see ( I saw a tree out the window) and the metaphorical sense of see ( I see a bright future for him). Since the latter does not involve any physical state changes, we deemed that see.01 was not suitable to keep as an event type. For `cause.01`, this is because causal relations are usually a separate topic of research (such as in [1]) and usually not included as an event type. \n\nQC: Yes, Qnodes unrelated to trigger possible rolesets are still considered as candidates following type ranking, we did not use the ontology to do another round of filtering. In addition, some of the ground truth types did not appear in the top-10 ranked types (this accounts for ~10% of the errors as shown in Table 4).  \nThe mapping between Qnodes and rolesets is many-to-many, so the candidate rolesets cannot be directly determined by taking the lemma of the trigger. For example “form.01”, “create.01”, “make.01” and “craft.01” are all mapped to the Qnode “creation”. \nWe agree that the two ways of categorizing errors are a bit orthogonal (one is based on the mapping, another is based on the ontology hierarchy) and we can split them into two pie graphs for analysis. \n\n\n[1] Modeling Document-level Causal Structures for Event Causal Relation Identification (Gao et al., NAACL 2019)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* S1. The GLEN dataset and ontology resources contributed by the paper, which involve substantial manual annotation work and distinguish themselves from existing EE resources for the large number of events covered, filling this way a gap and supporting further work in this context.\n\n* S2. The proposed EE method which is shown to substantially improve over state-of-the-art methods in the considered EE context.\n\n* S3. Informative experiments and analysis reported in the paper (including appendix).\n\n* S4. Reproducibility facilitated by adequate description of the approach in the paper and by code and data provided as part of supplemental material.\n\n* S5. Overall well written paper.", "reasons_to_reject": "* W1. Event roles appear to be neglected in GLEN dataset and ontology, and arguments extraction and their assignment to roles is not part of the proposed extraction method.\n\n* W2. Some choices related to the construction of the GLEN ontology are not completely clear (see questions A, B) and I wonder whether this may pose issues in using in practice GLEN (in its current version, as authors mention to evolve it) due to have discarded relevant types of events.", "questions_to_authors": "* A. How does the removal of cognitive Qnode events make the ontology \"more suitable for the event extraction task\" [§2.1]?\n\n* B. What's the motivation for removing general / ambiguous PropBank rolesets like cause.01 and see.01 [§C, §2.1 \"another round of data filtering\"]? Is that there would be too many options for annotators to choose among?\n\n* C. Does the error analysis for type classification [§5.3] refer exclusively to event triggers in ground truth data? If so, it would seem to me that error categories 'candidate set' (e.g., have trigger 'working' for roleset 'work.01', and predict wrong Qnode for 'work.01') and 'extended roleset' (e.g., for same example, predict a Qnode for 'work.XY' with XY != 01) should cover all possible error scenarios, as the trigger should constrain the possible Qnodes that can be picked for event classification (via 'trigger' -> possible rolesets for trigger -> possible Qnodes for such rolesets). Is it that Qnodes unrelated to trigger possible rolesets are considered as candidates following type ranking? Anyhow, it seems to me that error classes 'child', 'parent', 'sibling', and possibly 'unrelated', may form another orthogonal way to classify errors interesting fro practical applications (e.g., a user may still be satisfied if a child type is wrongly chosen since the parent type can be still inferred).", "paper_topic_and_main_contributions": "SUMMARY. The paper focuses on event detection/extraction (EE) - specifically, trigger detection and event type classification - in less investigated scenarios involving large, domain-general event type ontologies. For these scenarios, authors provide a dataset named \"GeneraL-purpose EveNt benchmark (GLEN)\" combining a large-scale Wikidata-based event ontology derived from the target of DWD Overlay mappings, with a large-scale corpus of event annotations derived by mapping PropBank gold standard event annotations to ontology types, obtained leveraging DWD Overlay and manual annotation for dev/test sets. Authors also propose a neural method (based on pre-trained LM - BERT) for trigger detection and type classification tailored to large event ontologies, and evaluate it on GLEN demonstrating superior performance w.r.t. applicable state-of-the-art methods.\n\nCONTRIBUTIONS:\n\n* C1. Domain-general, large-scale Wikidata-based event ontology grounded on PropBank, derived via manual cleaning from DWD Overlay mappings.\n\n* C2. Domain-general, large-scale \"GLEN\" EE dataset derived from ProbBank gold standard annotations involving manual annotation (for dev/test sets).\n\n* C3. EE method tailored to considered EE context.\n\n* C4. Analysis of the GLEN dataset.\n\n* C5. Extensive experiments investigating the performance of proposed method and relevant state-of-the-art systems on the proposed dataset\n\nPOST-REBUTTAL UPDATE. I thank the authors for the clarifications in their rebuttal. After reading it and considering the other reviews as well, I confirm the original scores in my review.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* T1.  [§2.1] \"mainlly\" -> \"mainly\"\n* T2.  [§2.3] \"wider range\" -> \"wider range than ...\" or simply \"wide range\""}, "extra_text": {}}, {"review_id": "W9ondewH1b", "reviewer": "Reviewer_ohVy", "summary": "", "questions": "Q1: it is not clear what happens when multiple triggers are in the same region. Will they all be collected and turned into good predictions?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful comments. \n\nWhile we did start the paper by mentioning that ACE05 is an old dataset, we don’t think being old itself is the biggest fault. The real concern is that due to the very restricted ontology of ACE, models that are trained on it cannot be used for real applications. This hinders the adoption of event extraction models, especially in comparison to other IE models, such as NER and entity linking.\n\n“Given the advances in SoA, one would expect that annotation guidelines have changed a lot in the meantime. One would also expect that old datasets offer no surprises any more” -- interestingly enough, the annotation guidelines have not changed since the majority of papers still use the original annotations in ACE, but newer work has found new settings such as few-shot learning, zero-shot learning or continual learning and adopted the old dataset for such cases. These efforts contribute to the long-reigning popularity of ACE and are orthogonal to our work. \n\n**Regarding model name and the analysis tables**: we think your suggestions are very useful and we will incorporate them in the revised paper. We actually selected examples for every category in the error analysis but had to comment some out to fit the paper limit. We will add them back given the extra page available. \n\n**Title of section 5.3**: we meant to say that these were “remaining errors” after our full model, as opposed to section 5.1 where we show error per stage. We will change this into “error categorization” or “error composition” to be more clear. \n\n\n**Regarding the “unusual paper structure”**: perhaps we were biased by pieces such as “How to Write a Great Research Paper” https://simon.peytonjones.org/great-research-paper/ which advocated for a structure that put related work towards the end of the paper. That being said, we appreciate your feedback on how delaying the Related Work section would cause difficulty to the causal reader. Besides moving the Related Work section after the introduction, we will also add some definitions for the event detection task to make the paper more approachable. \n\nQ1: Yes, our model can detect multiple triggers per sentence. In the trigger detection stage, we perform classification over all candidate spans that are less than 10 token long. In the event type ranking stage, although the ranking is done on the sentence-level, we take top-10 event type candidates, which can be mapped to different triggers in the sentence. Then finally, in the event type classification stage, we perform classification for each of the identified trigger spans in the sentence. \n\nFor example, in the sentence “later she says intimidated into changing her vote to not guilty”, our model can detect 3 events with triggers “intimidated [Q3858303]”, “changing [Q1150070]” and “vote[Q189760]”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- creation of a huge dataset (GLEN) which is almost 2x the size of the previous largest event extraction dataset (MAVEN)\n- development of good trigger identification and event type ranking methods\n- strong analytical section\n- the system will be released as a Docker container, thus enhancing reproducibility", "reasons_to_reject": "- paper is written in an unusual format, leaving related work as one of the later sections of the paper and confusing casual readers\n- section 5.3 about remaining errors is confusing - remaining after what?", "questions_to_authors": "Q1: it is not clear what happens when multiple triggers are in the same region. Will they all be collected and turned into good predictions?", "paper_topic_and_main_contributions": "The paper is well-written and well illustrated. Figure 2 clearly explains the process of turning the collected triggers into predicted results.\n\nThe main topic of the paper is making event extraction pipelines more accesible. A larger dataset with over 200k event mentions is created by using the DWD Overlay which provides mappings between Wikidata and PropBank. The results are generally better than InstructGPT and several older BERT models.\n\nThe paper starts by observing a disturbing fact: namely that the most used event extraction is almost two decades old. This is probably observed based on citations, which makes sense. However, given the advances in SoA, one would expect that annotation guidelines have changed a lot in the mean time. One would also expect that old datasets offer no surprises any more. This turns out to not be the case.\n\nThe paper is written in an unusual format, the Related Work section being a mere afterthought before the Conclusions section. This is extremely difficult to read by the casual reader. As a person familiar with NER/NEL/Slot Filling/Event Extraction I can jump directly into the topic and experimental section, but a casual reader that is not familiar with the topics may need some time to adjust. The classic structure of: Introduction, Related Work, Method, Experiments, Discussion (not only discussion around experimental results, but also around the previous work), Conclusion, Limitation is rather more suited for any paper. In fact I remember a Twitter thread from 2020-2021 that clearly explained this and was heavily retweeted. \n\nQuantitative evaluation results showcase good results - though the evaluated model is simply mentioned as \"Ours\". This is confusing as future work will maybe call the model with a name that's not mentioned anywhere in the paper creating a source of confusion.\n\nTable 3 could have used more rows. For example, it is not clear what happens when multiple triggers are in the same region. Will they all be collected and turned into good predictions?\n\nThe analytical section is rather good, but only provides examples for several error categories.\n\nI have read the rebuttal and think they have addressed my concerns.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I could not detect missing references.", "Typos_Grammar_Style_and_Presentation_Improvements": "Generally good grammar and presentation (save for the related work issue)."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The GeneraL-purpose EveNt benchmark (GLEN) is a notable initiative aimed at advancing event benchmarking. It achieves this by combining a substantial Wikidata-based event ontology, derived from DWD Overlay mappings, with a comprehensive corpus of event annotations. These annotations are created through the mapping of PropBank gold standard event annotations to ontology types, augmented by manual annotation for dev/test sets. As a result, GLEN introduces a novel dataset and presents an innovative event detection method, establishing the first baseline for this dataset. The associated research paper is well-written and effectively illustrated, but it would benefit from reorganizing the related work section to provide better context.

Strengths:

* Innovative Approach and Large-scale Corpus: GLEN introduces a novel dataset and event detection method, contributing to the field of event benchmarking.
* Well-Written Paper: Oveall the paper is well written and easy to follow.

Weaknesses:

* Quality Assessment: GLEN lacks explicit quality assessment metrics for its dataset, such as a histogram showcasing the distribution of data types.
* Limited Scope: The focus of GLEN is primarily on event detection, neglecting event roles, which might limit its applicability in certain contexts.
* Typos and Structure of the Paper: The paper contains a few typographical errors that could affect the overall readability and keeps related work at the end making it hard for casual readers to follow the work.
Decision: Accept-Main