MailEx: Email Event and Argument Extraction
Authors: Saurabh Srivastava
Gaurav Singh
Shou Matsumoto
Ali K Raz
Paulo Costa
Joshua Campbell Poore
Ziyu Yao
Venue: EMNLP 2023 Main (2023)
Keywords: Event Extraction
Email
Information Extraction

Abstract: In this work, we present the first dataset, MailEx, for performing event extraction from conversational email threads. To this end, we first proposed a new taxonomy covering 10 event types and 76 arguments in the email domain. Our final dataset includes 1.5K email threads and ~4K emails, which are annotated with a total of ~8K event instances. To understand the task challenges, we conducted a series of experiments comparing three types of approaches, i.e., fine-tuned sequence labeling, fine-tuned generative extraction, and few-shot in-context learning. Our results showed that the task of email event extraction is far from being addressed, due to challenges lying in, e.g., extracting non-continuous, shared trigger spans, extracting non-named entity arguments, and modeling the email conversational history. Our work thus suggests more future investigations in this domain-specific event extraction task.

MAILEX: Email Event and Argument Extraction
Saurabh Srivastaval, Gaurav Singh', Shou Matsumoto', Ali Razl, Paulo , Joshua Poore", Ziyu Yao George Mason University, *University of Maryland ARLIS {ssrivas6, gsingh33, smatsum2, araz, pcosta, ziyuyao poorejcêumd. edu
Abstract
cannot connect with other relevant tasks. For in- stance, Lampert et al. (2010) even after identify- ing emails containing requests, users will still need to manually search through the lengthy emails to find the actual request and then manage the tasks with separate tools. On the other hand, existing ex- ploration on the email data can cover only subset of potential events in email communications (e.g., requests or to-do tasks), whereas there are many others that also commonly happen and need proper management (e.g., delivery of data or information).
this work, we present the first dataset, МАn.Ee, for performing event extraction from conversational email threads. To this end, first proposed new taxonomy covering 10 types and 76 arguments the email do- main. Our final dataset includes 1.5K email threads and 44K emails, which are annotated with totally ~8K event instances. To under- stand the task challenges, we conducted se- ries of experiments comparing three types of approaches, i.e., fine-tuned sequence labeling, fine-tumed generative extraction, and few-shot in-context learning. Our results showed that the task of email event extraction is far from being addressed, due to challenges lying in, e.g., ex- tracting non-continuous, shared trigger spans, extracting non nomdn entity arguments, and modeling the email conversational history. Our work thus suggests more future investigations in this dopain- event extraction task.!
To facilitate more comprehensive downstream tasks on email data, this paper, we introduce the task event extraction on email threads. Event extraction (Grishman, 1997), or EE, is the task of extracting specific occurrence of an event and its arguments. It is important step for downstream tasks such information retrieval, question swering, knowledge base population, etc. EE has long been studied in the context of news articles (Li et al., 2021; Yu et al., 2022: Du et al., 2022). As far as we know, there has not been such a dataset in the email domain. On the other hand, EE on email threads brings unique challenges such as per- forming information extraction in conversational setting and needing handle much longer and more verbose arguments, which cannot be studied with existing datasets.
1 Introduction
Email has been one of the most widely used com- munication mediums, especially in professional and work environments. As the number of email users continues to rise, service providers are con- stantly looking for ways to improve user experi- ence. With advancements in machine learning and natural language processing, email platforms have introduced range of features aimed at helping users manage their inboxes and automate tasks Freddern-Bekcan, 2008; Kannan al., 2016; Chen al., 2019), 9), concurrent with research on request identification (Lampert et al., 2010), intent identi- fication (Wang et al., 2019), meeting summariza- tion (Deshmukh and Lee, 2022), task management (Zhang et al., 2022; Mukherjee et al., 2020), etc. However, most of the existing work touches only specific aspect of email information and thus
To fill this gap, first developed a taxonomy to describe events and arguments email data. Specifically, we designed a set of event classes and 76 arguments using the speech act theory Cohen et al. (2004). Our event classes cover pro- posals, amendments, and deliveries of actionable events on meetings, actions, and data items (Sec- tion 2). Unlike existing EE datasets, each trigger in the email EE task is described with one Verb and one Noun Act (e.g., Deliververb Datayoun), and the arguments are often long-span, non-named entities (c.g., a description of the meeting agenda), which make the task much more challenging. Based on
'hee source code and dataset can be obtained from https: //gi thub. cwm http//allok//faa /emalaailllell -erEeaallleeeee.. -Eve---eee--- conntExttacte Extraction.

---

this taxonomy, we then proposed a new dataset, MAILEX, which consists of 1.5K email threads and 44K emails annotated with ~8K events. The dataset achieves a substantial agreement between annotators.
acts categorization may lead to skewed data dis- tributions (Table 5), doing so allows us to easily connect the EE task with downstream applications. For example, when an EE model extracts meeting data, a downstream email reminder can be automat- ically set up to provide additional assistance, which will not be feasible if we simply merge all types of data information into one coarse category. Detailed descriptions of all the verb and noun acts be found in Appendix A.1.
Comparing three approaches, i..., fine-tuned se- quence labeling based on BERT (Devlin et al., 2018), fine-tuned generative EE based on BART (Lewis et al., 2019), and in-context learning us- ing GPT-3.5,2 we analyze the challenges trigger and argument extraction. Our results highlight the need for advancements in handling non-continuous, shared triggers and long-span, non-name non-named entity arguments while emphasizing the importance of ef- fectively modeling email history. Moreover, the in- context learning of GPT-3.5 yields much worse per- formance, suggesting the challenge of this domain- specific task in the few-shot setting.
2.2 Event Types and Argument Roles
In total, we defined 10 event types with 76 argu- ment roles, including few "meta semantic roles" which come with pre-defined class spaces. We present three event types as examples below and show the full list in Appendix A.2. In the examples, we also denote the corresponding triggers (under- lined) and [argument roles] (wrapped by "[ I"). Request Data: The event is triggered when the sender seeks data such file fact.
2 Taxonomy for Email Event Extraction
2.1 Verb and Noun Acts
In this work, we focus on extracting commonly seen events (e.g., scheduling meetings) from daily email communications. Our event definition follows the email speech act theory of Cohen al. (2004). An email speech act describes the sender intent using a "verb-noun" pair, such as "Requestyerb Meetingnoun". ". As such, the email speech act carries the "actionable" information by the sender. In Cohen et al. (2004), a set of five verbs and four nouns are proposed (which could form 20 email speech acts). However, our preliminary study email corpora (Minkov et al., 2008; Oard et al., 2015; Ulrich al., 2008) reveals that most of them are not used frequently in daily communications (e.g.. amending an action), or are not relevant to "events" (c.g., exchanging opinions). Therefore, we keep our focus on the most common 10 event types enabled by three verb acts (i.e., Request, Deliver, and Amend) and three noun acts (i.e., Data, Meet- ing, and Action). For the noun act "Data", further consider three sub- cutecatee::::s: (a) Meeting Data, which refers facts related specific meet- ings (e.g., meeting date, location), (b) Action Data which refers to facts related to a specific action or an activity (e.g., a deadline for approving a budget request, the person who approved the request, etc.), and (c) Data which refers to all other information irrelevant to meetings and actions, such as PDF files sent in emails. While this fine-grained noun
Example: Please send me [the sum- mary of our meetingl I Jom Eaftiring (Request Attribute: Data Value)
Here, "Data IdString" refers to the identity or de- scription of the sender-requested data. We also introduce meta semantic role "Request Attribute" to indicate the attribute that the sender queries from the data, which in practice is often the "Data Value", e.g., the specific PDF file of the meeting summary. Deliver Data: The event is triggered when the sender provides or commits to provide certain data.
Example: Attached for your review [the summary of our meeting I Data liString- (Con-
firmation: Positive)
For Deliver events, we introduce "Confirmation" (positive, negative, or tentative") as a meta seman- tic role, affirming if the sender can provide the requested data information (i.e., when the noun act is Data), acknowledge their attendance in meet- ings or participation in action events (i.e., when the noun act Meeting Data or Action Data). Notably, the Confirmation role could be perceived as a form of "data" as well. In a conversational email set- ting, people often reply with brief responses such as "Sure" or "No, it doesn't 'tt work" when someone makes request. By introducing the Confirmation
'Rarely people may give uncertain responses such as "I'm not sure": in that case, we mark as "Unsure".
hhttps::/platforrt openai ptmeeni..cmm/// cow//ocss/odeellllooooooooooo//e

---

role, we can discern the sender's intent even though no concrete event information may be extracted from a short answer.
Data Statisties
Total (train/dev/test)
# of email threads
1,500 (1,200/1 (1,200/150/150 50/150)
# of total emails
3,936 (3, (7/110//)
# of non-event emails
776 (636/70/70)
The event is triggered when the sender requests or indicates changes to a data record. In order to describe the type of change, we introduce fixed set of "Amend Type" verbs including add, delete, and update. Additionally, we have observed that individuals frequently de- scribe changes by providing context followed by the revision, as shown in the example below. Con- sequently, to differentiate between the various roles, we introduce two labels, "Context" and "Revision", and replace all argument roles of the Data act with two sets of copies for each (c.g., "Context: Data Type" and "Revision: Data Type" instead of the original "Data Type"). These modifications allow for more precise differentiation and description of the different aspects of the event and its roles.
Amend Data:
# of annotated events
8,392 (6,571/946/875)
Avg. # of events of the same
3.05
lype uppearing at least twice
Avg. # of words in an email
64,400
Avg. # words in trigger
2.64
Avg. # words in an argument
7.41
Table 1: MAILEX data statistics.
types, and argument roles for each email while con- sidering the context of the email history. Two native English-speaking Computer Science students were recruited for the task and received 12 USD/hr for their work. To ensure accuracy, multiple rounds of training and discussions were conducted. Each email was annotated twice by each annotator, and annotations with agreement on event type, over- lapping trigger words, and argument spans were retained. Specifically, for partially agreed triggers (but with agreement on the event type), we re- tained the overlapped word spans, and for partially agreed arguments (but similarly with agreement on the event type and having overlapped trigger spans), similarly retain the overlapped word span. When two annotators did not agree on the event type or made no overlap in their annotated triggers, we abandoned the annotations completely; for more details and the annotation guideline, see Appendix 1.1. In total, we collected a dataset con- sisting of 1.5K email threads, encompassing ~4K emails and 8K events (Table 1).
Example: Can [youl change [the budget] cersinus ldSiiing from [2K] Value to [3K] REV:Dau (Amend Type: Update)
Note on Non-Continuous, Shared Triggers. Fi- nally, we note that multiple events of the same type could be mentioned in one email. In that case, trig- ger words could be shared partially between events, which makes the dataset more challenging:
Example: Alice will approve the wire request and inform to Susan.
In this example, two Deliver Action Data events share the trigger word "will".
Agreement (IAA), We measure two IAA values, one for triggers and their associ- ated event types (i.e., whether annotators agree on the same trigger words and assign the same event type), and one for the argument roles (i.e., whether annotators agree on the argument role annotations for the same trigger and event type). For both cal- culations, we consider overlapping spans as indi- cating partial agreement and apply Cohen's kappa k (Cohen, 1960) at the word level while comparing the annotations. We obtained a value of 0.791 (i.e., substantial agreement) for the trigger-event type IAA and 0.810 (i.c., almost perfect agreement) for the argument role IAA. For "meta semantic role" annotations, we did observe disagreements be- tween the annotators who had agreed on event trig- gers. We include analyses on the disagreement cases in Appendix B.2.
Inter-Annotan
3 The MAILEX Dataset
3.1 Data Annotation
We utilize the open-source Enron dataset (Minkov et al., 2008) which comprises a collection of email data from 150 users. We considered the top 50 users with the highest inbox counts and randomly selected a range of 20 to 40 email threads for anno- tation. Note that all single-email threads have been removed in the interest of conversational event ex- traction. By focusing on a set of users, MAILEX could support personalization research, which we leave as future work. The annotation process involved annotators marking trigger words, event
http://wwww2.ccccmm http::2ccsccmmcccdceeeeeeeee cet.oo//////no/.....mmmm Some prior work in- stead used Avacado (Oard et al., 2015); we did not choose it hecause it is not completely publicly available.

---

Hisoryl
Labeling
progress
Trigger Extraction
FM. (CONTEXT)
Shared Triggrs
Can acheduue fomenoow,, Alann plevsss aend the presenratins for your disoussion by and of cay today [CONTEXT] youns you for attonding the meeting overyone can boo yone of you please prepare the (DONNEXT)
Dijar "Tpper "Delver Dara (307
The rugt
1eat 223
1 S Do
HDD]
0
1000
HD0
1:00
we clacussed send ma
Ind the yoding pdiat Type
BERT
Currnni Email
Content
[BER
ciocsossl
plosse
frd
the
dult
and the (udiin josks]
diccsadd,
au discussed in
moeting rediine
Aitohted
smmary.pdl
Argunent Extraction
Extracted Aigument for Flole DDea diSting
Tareet Duast Trigee "stuus Ind
Event templotes
[B01] Event Rlecual
trappeed by tnioperi where, lácton
Opan the distt
Desciipdont requested from
of
BERT
[B0T]
[BOT) Rivert Delive Dats tiggened
where, Ibbra bescipptot, aptooi, iDbra vaiuo or aill be de ivered to iDeneer
(CLB) DD
TYPE
Contect [BEP]
Ao discussed
[TRC
plesse Eind Bhe (dul (TRO))
and
thee podieeiineee needs
(SEP
Momborsi at ITima on iDasal (BOT) vaas
- Jin lotal limclaess)
BARI besed Extraction
Invoorlust OPT-3.
Filled Event Templstes
[BOT] Event Deliver Dasa...
Deliver
[B01] Event Deiver Data trapered by iclesse ind the chail
cruf ciecussed bavn, surmaryadll (Datel (EOT) whire, the ridiie nu edite, ietiimantl is will delivered
(BO1I (Delver Data
Enooder
Decoder
Fiw-shhtt Domonntration
Deliver Memeens at mim on (Datel (FDT) shtt sitz
Cunent Enail Templates
Instuct
Contest
in
Gument Eiret
in
Figure I: Illustrations the three approaches we experimented with for email EE,
3.2 Data Statistic and Analysis
budget plan... ... JMtening Agmdda As a result, arguments in MAILEX can be much longer than conventional entity arguments and may even span over few sen- tences. Unlike trigger spans, however, argument spans are always continuous spans.
We present MAILEX statistics in Table 1. By look- ing into the details, MAILEX presents following unique characteristics and challenges:
Imbalanced type and role distribution. As shown in Table 5, the event distributions are imbalanced across different event types (c.g., events related to delivering data more common than amend- ments); similarly for argument roles.
Non-event emails. Some emails contain only non- event information, such as opinions and news in- formation, and an intelligent EE model should not identify any events from them.
Tabular data. MAILEX also includes emails con- taining tabular data, which pose challenges due to their sen-sentence sequential structure (sec Figure 4 for example).
Conversational context. In a conversational set- ting, we observe common patterns between con- secutive emails. For example, request event is typically followed by deliver an amend event. Modeling the email context and capturing this intu- ition can thus be helpful for the task.
Methodology
Multiple events of the same types. Unlike ex- isting datasets, MAILEX often contains multiple instances of the same event type within single email, such multiple deliver data events. When such cases happen, on average the same event type recurs in ~ instances in the same email.
4.1 Task Formulation
Each email thread X = -X1...... Xi...., X7) consists of multiple emails, where T is the total number of emails in thread. Our goal is to extract events from the thread. This involves two sub-taskss (1) Trigger Extraction, where we identify the trig- ger span within each email and determine the as- sociated event type, and (2) Argument Extrac- tion, where we identify spans within each email that serve argument roles for the event instance. During event extraction for a given email X,, only information before the current time step is used context. This emulates practical scenario where an intelligent system incrementally extracts events as new emails arrive. In this work, we explore three approaches to understand the task of email event extraction, as summarized in Figure 1.
Non-continuous, shared triggers. Since MAILEX contains event classes with verb and noun acts, triggers signaling both acts may not necessarily be continuous, especially when they share spans, posing a new challenge for trigger identification. Non-named-entiyy arguments. Argument spans for roles such as "Meeting Agenda" and "Action Description" may not necessarily be named enti- ties; as an example, consider the "Meeting Agenda" argument in "We will discuss the following items to- day (1) Actionable for this month. 2) Next month's 's

---

4.2 BERT-based Sequence Labeling
illustrated the promise of adopting pre-trained au- toregressive language models for EE, where label names are explicitly spelled out during decoding (e.g., "The meeting time 7 AM" for extracting the "Meeting Time" argument) and their semantics can thus be leveraged (Li et al., 2021; Du et al., 2022). Drawing inspiration from there, we design set of event templates (see Figure 1 for example and Appendix C.2 for all templates) and fine-tune BART model perform end-to-end EE. For end-to-end extraction, the model's input comprises the email content and the template. Specifically, we prepare the input sequence as "
 X1 [CONTEXT] X2 
 [BOT] $template: [EOT] 
 [BOT] $templatez ... [EOT] 
 
" where "[BOT]" and "[EOT]" are special tokens indicating the template boundaries. With this setup. BART is trained to generate a sequence templates, extract- ing the events from the email and their respective arguments. Importantly, the model only decodes templates for events present in the email, disregard- ing the ones for absent events. Moreover, in scenar- ios where an email contains multiple instances of the same event, the model produces multiple filled- out templates for each instance, all categorized under the same event type. All the generated tem- plates are delimited via the special tokens "[BOT]" and "[EOT]".
Sequence labeling based on BIO tags is classic approach event extraction (Nguyen al., 2016; Nguyen and Nguyen, 2018). In our work, we fine- tune two BERT models, one for trigger extraction and one for argument extraction, respectively.
For trigger extraction, in order to address the challenge of multiple events within single email (Section 2.2), we additionally introduced shared "S" tag. Each event type assigned BIS tags, such as S/B/I-Request Meeting, while the tag O is used to denote non-trigger words common to all event types. Shared triggers among event instances of the same type are identified using S tags (see Figure 1 for an example). The input to BERT is organized as "[CLS] X1 [CONTEXT] X2. [CONTEXT] [SEP] X₁ [SEP]". Each word in the current email X, is then assigned label from the set of BIOS tags based on its BERT representation.
[CONTEXT] 
 X;
For argument extraction, the BERT model is provided with a target trigger span and its event type. We similarly formulate this task sequence labeling problem. However, unlike trig- ger extraction, arguments of the same event in- stance do not share spans. Therefore, we do not use any "S" tags in argument extraction. We pre- pare the imput as the following to the BERT model: "[CLS] $type [TYPE] X1 [CONTEXT] X2 [CONTEXT] [SEP] Tt,1 [TRG]
BIO
[/TRG] ... Fxpx2 [SEP]". Here, $type is a placeholder for the event type. To encode the trigger span in- formation, introduce a pair of special tokens "[TRG]" and "[/TRG]" to indicate triggers in the current email X,. In the case of non-continuous trigger spans, every segment of the trigger span will be wrapped by this pair of special tokens. The argument role label of each word is then predicted based on its BERT representation.
4.4 In-context Learning with GPT-3.5
We further evaluate the performance of GPT-3.5 to understand if few-shot large language models have been able to perform well in closed-domain EE tasks. Similar to BART-based EE model, we use GPT-3.5 for end-to-end extraction. Our prompt concatenates task instruction, the event tem- plates, few-shot demonstrations, and the context and email body for the test example. We ensure the presence of all possible event types and arguments by carefully selecting K (K=5) shots of demon- strations from the training set. We present our prompt in Figure 8 in Appendix C.3. In experi- ments, we investigated both text-davinci-003 and 5
In addition, the argument extraction model also includes classifier heads for meta semantic roles prediction (Section 2.2), which will be jointly opti- mized in training. We refer readers to Appendix C.I for details about the training and inference of the sequence labeling approach.
5-tubb for a comparison.
Experiments
4.3 BART-based End-to-End Extraction
5.1 Experimental Setup
A drawback of sequence labeling approaches lies in that they cannot leverage the semantic meaning of the label names and thus may need massive annota- tions for effective generalization. Recent work has
Datasets. We split MAILEX by email threads into training, development, and test sets with a ratio of 80, 10, and 10, ensuring that at least once each

---

Trigger
Arguneent
Identiffcation
Classifiatton
Identification
Classfliation
Precision
Recall
F1
Precision
Recall
Precision
Recall
F1
Precision
Recall
BERT-buse bused Sequence Labeling
0.581
0.499
0.537
0.566
0,486
0.523
0.491
0.403
0.454
0.355
0.383
0.368
w ground cruhh triggess
0.653
0.662
0.642
0.651
w/o email thread history
0.577
0.493
0.531
0.532
0.483
0,506
0.188
0.397
0.438
0.335
0.380
0.356
BART-bused Gemerative Extraction
0.701
0.395
0.505
0.701
0.394
0.500
0.592
0.351
0.44]
0.374
0.350
0.363
w/ ground-crud amail thraad history
0.690
0.482
0.568
0.688
0:471
0.560
0.389
0.500
0.679
0.572
0.421
0.370
0.384
In-contest Learning (GPT-3.5)
text-Gavinci-003
0.167
0.171
0.169
0.100
0.101
0.100
0.068
0.069
0.068
0.058
0,060
0.058
groundtruth triggers spt-3. 5-tubo w/ tround-tutth triggers
0.379
0.367
0.349
0:330
0.338
0.183
0.095
0.121
0.098
0.060
0.072
0.058
0.045
0.050
0.056
0.040
0.048
0.256
0.198
0.223
0.242
0.190
0.211
Table 2: Results on MAILEX test set. For both fine-tuned and in-context learning, we additionally report their argument extraction performance when feeding ground-truth triggers ("w ground-truth trigger"). For the former, we also report their overall performance when the email thread history ablated ("w/o email thread history"), " indicates significantly better performance than BART under a Wilcoxon signed-rank test (Wilcoxon, 1992) with significance level = 0.05, whereas no significant difference was observed for Argument Classification F1.
gests that BART occasionally fails to copy from the current email, which leads to low recall. Moreover, Evaluation Metrics We evaluate trigger and ar- gument extraction using Precision, Recall, and F1 scores, following prior work (Du and Cardie, 2020; Sheng et al., 2021). For triggers, consider match when the identified span exactly matches the gold label (Trigger Identification) and is cor- rectly classified into the event type (Trigger Clas- sification). For arguments, we assess Argument Identification (i.e., whether an argument span is conrectly identified) and Argument Classification (i.e., whether the argument span is additionally cor- rectly classified into the true role). Unlike trigger evaluation, partial matching is allowed for argu- ments encourage more fair comparison, espe- cially for non-named entity arguments with long spans. This aligns with similar evaluation strategics used by Li et al. (2021). Finally, we note that the argument evaluation reports an end-to-end extrac- tion performance; for BERT-based sequence label- ing, only the model-extracted triggers are fed for argument extraction during evaluation. More im- plementation details are provided in Appendix D.
event type is present in each of the three sets. The statistics for the sets are shown in Table 1.
for trigger identification and classification, BART achieves close Fls, suggesting that once it identi- fies the span, it accurately classifies the trigger.
Finally, we note much worse overall perfor- mance by both versions of GPT-3.5 in-context learning, which we will carefully discuss in Sec- tion 5.2.5. In light of this underwhelming perfor- mance, our subsequent analyses will mostly focus on the two fine-tuned approaches.
5.2.2 Challenges in Extracting Triggers
Identifying Minimal, Complete Trigger Spans. Our annotation guidelines (Appendix B.1) con- strains a trigger to be a minimal sequence of words or phrases triggering an event. We observed that both models fail to adhere to this constraint and make mistakes by adding additional trivial details, e.g., for email with the ground-truth trigger "Will meet", BERT predicted "Will meet upstairs". Additionally, we noticed few instances where both models fail to identify the complete trigger span, resulting in propagated errors in trigger classi- fication. For example, for an email with the ground- truth trigger "amended the deal", BERT predicted a partial trigger "amended". It is worth emphasiz- ing that in the ground-trui -ruth trigger, the phrase "the deal" informs the recipient about an amendment made on the data "the deal", thereby triggering the Amend Data event. Failure to identify the complete trigger span incorrectly triggered a Deliver Action Data event.
5.2
Experimental Results and Analyses
5.2.1 Main Results
Table 2 shows the model performance. We observe that the BERT-based sequence labeling pipeline and the BART-based approach achieve comparable end-to-end argument classification performance, though the former outperforms the latter in trig- ger extraction. On the other hand, BART exhibits high precision in making decisions, yet struggles in recall. A qualitative examination of the dev set sug-
Classifying the Noun Acts of Triggers. In trigger classification, models struggle to properly classify

---

noun acts associated with triggers. For example. we observed instances where the true event Re- quest Action Data was confused with Request Ac- tion 39% of the time, and Deliver Meeting Data was confused with Deliver Action Data 27% of the time (Figure 6 in Appendix E.1). Such challenges arise from similar words or phrases used by email senders to trigger specific noun acts. For instance, when the trigger is "will attend the seminar" BERT fails to recognize that seminar is type of meet- ing, resulting in incorrect classification as Deliver Action Data instcad of Deliver Meeting Data. This highlights a challenge in MAILEX, where abstract event-type definitions lead to language variations and variations in noun acts of triggers. On the con- trary. previous event extraction datasets (Walker et al., 2006; Wang et al., 2020) have focused mainly on verb-act triggers, overlooking the complexities of noun acts and the resulting language variations.
07
DERT
DERT Co
96
0.2
Langh of Argumantt
Figure 2: Argument classification results MAILEX dev set, categorized by whether the argument is a named entity (left) and by its length (right). For spans of length more than 10 we show macro-average of their Fls. All the models struggle to correctly extract non-named enti- ties, long-span arguments.
9% Argument Classification F1. This implics a stronger argument extraction performance from the former model. In our conjecture, this can be at- tributed to the fact that the pipeline approach has an independently learned argument extraction model, while the BART approach has to learn both extrac- tion tasks within the same model.
Extracting Non-Continuous, Shared Triggers. We observe that BERT was unable to segment spans that trigger multiple events. For example. in emails containing shared triggers with short distances, such as "Attached the report and the redlines.". ". identifies one Deliver Data event with the trigger "Attached the report and the redlines". Meanwhilc, when there is a long distance between the two partially shared triggers, BERT can iden- tify only the first one. We include examples in Appendix E.2. Intriguingly, BART was able to cor- rectly extract shared triggers with shorter distances in the majority of cases, though it still couldn't ' handle the longer distances. These findings are consistent with a similar study conducted by Sheng et al. (2021) where the authors also argue the lim- itations of sequence labeling approaches for such shared triggers.
Extracting Non-Named Entity Arguments. In Figure 2. we break down each model's argument extraction performance by named vs. non-named entity argument as well the argument length. The results indicate that all models struggle extract non-named entity arguments, particularly those with longer spans. This observation thus implies the need for more advanced modeling strategies future research.
5.2.4 Importance of Modeling Email History
In Table 2, we present model performance when ablating the modeling of the email history (i.e., "context" in Figure 1). As expected, we observed performance drops for both BERT and BART in all metrics. This emphasizes the importance of modeling the conversational history in the email thread. To corroborate this, we conducted a study of randomly sampled 50 emails and found that (22%) emails required the previous turn in event decision decision-making. We note that this percentage is much larger than the observed performance drop. We attribute this inconsistency to the ineffective modeling of email history when our approaches simply concatenate all the prior email bodies. This thus calls for future exploration, such as selectively including prior emails only when they are helpful for EE from the current email.
5.2.3 Challenges in Extracting Arguments
Error Propagation from Trigger Extraction. In Table 2, we present each model's performance on argument extraction when ground-truth triggers are fed, so as to understand whether the low end-to- end argument extraction performance of the two fune-tuned models is caused by error propagated by trigger extraction. We note that even with the gold triggers, both models still fall short of achiev- ing perfect argument extraction results, highlight- ing the challenging nature of both extraction tasks. Morcover, with ground-trutt triggers, the sequence labeling pipeline outperforms BART by around

---

prune the email thread input, which could lead a loss of information. Designing prompts that allow large language models to ground long context is thus an important future direction.
RUR
Extra Arguments
Missing Arguments
20%
Bemantically Connect
not exaclly matches spans
Somanticlly
#ggumens
Semantically
6 Related Work
and fabricated arguments
Event Extraction Models. Earlier work on EE tasks has typically followed a pipeline approach to identify triggers before extracting arguments (Ji and Grishman, 2008; Liao and Grishman, 2010; Du
Figure 3: Distribution of erroncous arguments extracted by 3pt-3.5-turbo
5.2.5 Analysis of In-context Learning
and Cardie, 2020). Alternatively, joint sequence labeling approaches (Nguyen et al., 2016; Nguyen and Nguyen, 2018) perform trigger extraction and argument extraction simultaneously, employing a unified decoder that tags the sentence in a single pass. A recent trend formulates EE as an extractive
Both versions of GPT-3,5 in-context learning be- haved substantially worse (e.g., 0.058 and 0.048 Argument Classification FI in end-to-end evalua- tion) compared to the fine-tuned approaches. To understand whether the challenge lies solely in ex- tracting triggers, in Table 2, we similarly present results with ground-truth triggers how ana- lyzed the fine-ture munet models Section 5.2.3. How- ever, the results show that even with ground-truth triggers, the few-shot argument extraction is still very challenging (more than 0.3 Arg. Class. F1 behind the fine-tuned models).
problem (Du and Cardie, 2020; Liu et al., 2020) which induces the language knowl- edge from pre-trained language models by convert- ing EE tasks to reading comprehension tasks via question template. With the help of pre-trained encoder-decoder Transformer architectures such BART and T5 (Raffel et al., 2020), there is also some recent work converting extraction tasks to generation tasks (Li et al., 2021; Lu et al., 2021). Finally, prompt-tuning (Dai et al., 2022: Ma et al., 2022) and few-shot in-context learning (Gao et al., 2023) have emerged as promising solutions to com- bat the "low resources" constraint of EE.
We analyzed 50 randomly sampled erroncous predictions by gpt-3. Surbbo w/ gold triggers, and categorized errors its extracted argument val- ues Figure 3. The most common mistakes made by the models include semantically incorrect argu- ments such as extracting incorrect person as the meeting member (42%). In this case, the incorrect arguments are still valid entities mentioned in the email. However, another common mistake (36%) is generating not only semantically incorrect but also fabricated, non-existing entities in the email as arguments. Approximately 8% of the generated arguments are semantically correct but not exact spans copied from the email, such as a summarized version of the meeting agenda. Other error types in- clude introducing extra arguments (8%) or missing arguments (6%); for the former, the model assigns the sender an extra member in all the failure cases. We include examples in Table 11. In addi- tion, gpt-3. 3.tt-3.5-uue 5utubb also made errors when gen- erating unused argument placeholders the event templates, which we discuss in Appendix E.3. No- tably, text-davinci-003 rarely generates fabri- cated arguments, and it obtains better performance particularly because made much fewer mistakes when generating argument placeholders.
In this work, we experimented with three ap- proaches, i.e., a pipeline of sequence labeling, the BART-based generative extraction, and few-shot in-context learning using GPT-3.5. Particularly for sequence labeling, we introduced a "S" tag to han- dle shared triggers. Our experiments compared these approaches and shed light on future research email EE.
Event Extraction Datasets. The Automatic Con- tent Extraction, ACE05, dataset (Walker al., 2006) has been the standard evaluation benchmark for EE, Similar to ours, there are also datasets fo- cused on specific domains, such as drug safety (Sun et al., 2022), news headlines (Deng et al., 2022), and business and financial domain (Capet et al., 2008). While most existing EE datasets aim to ex- tract information from individual sentences, several attempts have been made extend the extraction task multiple sentences (Ebner et al., 2020) documents (Eirew et al., 2022). As far as we know, MAILEX is the first comprehensive dataset for EE in the email domain. As discussed in Section 3.2,
We also note that due to the word limit imposed by GPT-3.5, for some test examples, we have to

---

it brings multiple unique challenges such as the conversational context and the need model non- named entity arguments, which were not covered by prior datasets.
event constraints (e.g., amendments often follow the proposal of an event), as well as proposing better modeling strategies to handle the long-text, non-named entity arguments in emails. Finally, it could be worth investigating the application of open-source Chat Language Models (c.g., Vicuna (Chiang et al., 2023), FastChat (Zheng et al., 2023), and Koala (Geng et al., 2023)) in this conversa- tional EE task.
Other NLP research on Email Data. Previous research on emails can be categorized into keyword and action extraction (Turney, 2000), request iden- tification (Lampert et al., 2010), modeling action items in emails (Lin et al., 2018), subject line gener- ation (Xue et al., 2020), to-do generation (Mukher- jee al., 2020), and text summarization (Desh- mukh and Lee, 2022). There has also been consid- erable research on identifying speech acts or tasks in emails (Cohen et al., 2004; Carvalho and Cohen, 2005) and how it can be robustly adapted across diverse email corpora (Azarbonyad et al., 2019). Recent work on task management automatically extracts actionable items from emails, generates faithful to-do items, and then aligns them to the correct users (Zhang et al., 2022). MAILEX uni- fies the majority of these tasks (such as handling requests, creating to-dos, ctc) and covers a wide range of events in email communications.
Another limitation of our work lies in the limited contexts of the Enron dataset, which is the source corpus of our annotations. As emails in the Enron dataset are all conversations among Enron employ- or between Enron employees and outsiders, the resulting MAILEX still retains this context foot- print and is not fully open-domain one. However, despite this constraint, our taxonomy of email EE is not limited to only business contexts. As high- lighted Section 2, our taxonomy, inspired by Cohen et al. (2004), is tailored for task-oriented email communications, with the goal of extracting "actionable" items conveyed by the sender. While the majority of the MAILEX focuses business- related dialogues, also touches down the realm of informal and personal communications, Such emails might delve into personal work reflections or family-related job discussions. This diversity is consistent with the findings of Alkhereyf and Rambow (2017), which revealed substantial vol- ume of personal communications in the Enron col- lection. Given that the Enron dataset is, to our knowledge, the only comprehensive and publicly available email corpus, MAILEX offers invaluable potential for subsequent email EE research, despite its specific contextual nature.
7 Conclusion
In this paper, we have proposed a new task of ex- tracting events and their arguments from conver- sational email data. To motivate future research in this direction, we also present a new dataset MXn.Ex, including a new taxonomy to describe common events mentioned in emails. We also con- duct series of evaluations MAILEX, conclud- ing that email EE is far from being addressed and more advanced methodologies are needed.
Limitations
Ethical Statements
While we aim to advocate the new task of EE in the email domain, our approaches can be significantly improved in the future. For example, as pointed out in Section 5.2.4, modeling email history is cru- cial for more accurate EE in a conversational set- ting. While we directly concatenate all the previous emails to extract events from the current turn, fu- ture work can design more specialized architectures for it such as applying an attention mechanism to retrieve only the relevant emails from the history. One could also use the dynamic memory similar to that of Du et al. (2022) and store only the extracted events (as opposed to the raw texts) from the email history. In addition, future work can further ad- vance our approaches by modeling the sequential
Our annotations are based fully open-source dataset (Enron), and our developed models will be open-source as well. We expect that our work can have a strong broader impact. For example, our dataset and the developed models can be used to enable more advanced personal assistants based on daily emails, which can improve workplace pro- ductivity or help people with difficulty in reading and processing large volume of emails. However, given that even the best-performing EE models in our experiments cannot precisely extract the stated information and may even fabricate contents, addi- tional verification tools and proper user guidance will be needed, although we anticipate that the ex-

---

traction performance can be significantly improved in the future.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuma: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Acknowledgements
This work was supported by the United States Govermment under contract FA8702-15-D-0002, via subcontract through the University of Mary- land. The views, opinions, and/or filings con- tained in this material are those of the author(s) and should not be construed as official posi- tion, policy, or decision of the Government of the United States or Carnegie Mellon University or the Software Engineering Institute unless desig- nated by other documentation. This project was also supported by resources provided by the Office of Research Computing at George Mason Univer- sity (https://orf //orc. gmu. gmu and funded in part by grants from the National Science Foundation (Awards Number 1625039 and 2018631). Finally, Saurabh and Ziyu appreciate the funding support from George Mason College of Engineering and Computing.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20( 30.
William Cohen, Vitor Carvalho, and Tom Mitchell. 2004. Learning to classify email into "speech acts". In Proceedings of the 2004 Conference on Empiri- cal Methods in Natural Language Processing, pages 309-33 409-316.
Lu Dai, Bang Wang. Wei Xiang, and Yijun Mo. 2022. Bi-directional iterative prompt-tuning for event argu- ment extraction. arXiv preprint arXiv:22 (tl:2210.11820
Haolin Deng, Yanan Zhang, Yangfan Zhang, Wangyang Ying. Changlong Yu, Jun Gao, Wei Wang. Xiaoling Bai, Nan Yang. Jin Ma, et al. 2022. 2event: Bench- marking open event extraction with a large-scale chi- nese title dataset. arXiv preprint 1.0002221..0086.
Soham Deshmukh and Charles Lee. 2022. Adapting task-oriented dialogue models for email conversa- tions. arXiv preprint arXiv:2208 09433.
References
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and
Sakhar Alkhereyf and Owen Rambow. 2017. Work hard, play hard: Email classification on the avocado and Enron corpora. In Proceedings of TextGraphs- the Workshop on Graph-based Methods for Natu- ral Language Processing, pages 57-65, Vancouver, Canada. Association for Computational Linguistics.
Kristina Toutanova. 2018. Bert: Pre-trrinnng of deep bidirectional transformers for language understand- ing, arXiv preprint arXiv:1810.0480)
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 671-683, Online. Association for Computational Linguistics.
Hosein Azarbonyad, Robert Sim, and Ryen W White. 2019. Domain adaptation for commitment detection in email. In Proceedings of the twelfih ACM inter- national conference web search and data mining, pages 672-680.
Xinya Du, Sha Li. and Heng Ji. 2022. Dynamic global memory for document-level argument extraction. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 5264-5275, Dublin, Ireland. As- sociation for Computational Linguistics.
Philippe Capet, Thomas Delavallade, Takuya Nakamura, Agnes Sandor, Cedric Tarsitano, and Stavroula Voy- atzi. 2008. A risk assessment system with automatic extraction of event types. In Intelligent Information Processing IV: 5 IFIP International Conference on Intelligent Information Processing, October 19-22, 2008, Beijing, China 5, pages 220-229. Springer.
Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence argument linking. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics.
Vitor R Carvalho and William W Cohen. 2005. On the collective classification of email" speech acts". In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 345-352.
Alon Eirew, Avi Caciularu, and Ido Dagan. 2022. Cross- document event coreference search: Task, dataset and modeling. arXiv preprint 2t.i:2220.22260
Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M Dai. Zhifeng Chen, et al. 2019. Gmail smart compose: Real-time assisted writing. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2287-2295.
Tanya Beddem--ekcan 2008. Google calendar. Jour- nal of the Medical Library Association: JMLA. 96(4):394.
Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. 2023. Exploring the feasibility of chatgpt for event extraction. arXiv preprini arXiv:2303.03832

---

Xinyang Geng, Arnv Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic re- search. Blog post.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv: 1.0010..
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2Event: Controllable sequence-to- structure generation for end-to-end event extraction. In P Proceedings of the 59th Annual Meeting of the A Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 2795-2806, Online. Association for Computational Linguistics.
Ralph Grishman. 1997. Information extraction: Tech- niques and challenges. In International summer school on information extraction, pages 10-27. Springer.
Heng Ji and Ralph Grishman. 2008. Refining event extraction through doss-document inference. In Pro- ceedings of ACL-08: HLT, pages 254-262, Colum- bus, Ohio. Association for Computational Linguis- tics.
Yubo Ma, Zehao Wang, Yixin Cao. Mukai Li. Meiqi Chen, Kun Wang, and Jing Shao. 2022. Prompt for extraction? paie: Prompting argument interac- tion for event argument extraction. arXiv preprint arXiv:2202.12100 12109.
Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kauf- mann, Andrew Tomkins, Balint Miklos, Greg Cor- rado, Laszlo Lukacs. Marina Ganea, Peter Young, al. 2016. Smart reply: Automated response sug- gestion for email. In Proceedings of the 22nd ACM SIGKDD international conference knowledge dis- covery and data mining, pages 955-964.
Einat Minkov, Ramnath Balasubramanyan, William W Cohen, and Machine Learning Dep. 2008 Activity- centred search in email. In CEAS.
Andrew Lampert, Robert Dale, and Cecile Paris. 2010. Detecting emails containing requests for action. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the As- sociation for Computational Linguistics, pages 984- 992.
Sudipto Mukherjee, Subhabrata Mukherjee, Marcello Hasegawa, Ahmed Hassan Awadallah, and Ryen White. 2020. Smart to-do: Automatic genera- tion of to-do items from emails. arXiv preprint arXiv:2005.06288
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grish- man, 2016. Joint event extraction via recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 300-309, San Diego, California Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zeutlemoyer. 2019. Bart: De- noising sequence-to- to-sequence pre-training for natural language generation, translation, and comprehension.
Sha Li, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 894-908, Online. Association for Computa- tional Linguistics.
Trung Minh Nguyen and Thien Huu Nguyen. 2018. One for all: Neural joint modeling of entities and events.
Douglas Oard, William Webber, David Kirsch, and Sergey Golitsynskiy. 2015. Avocado research email collection. Philadelphia: Linguistic Data Consor- tium.
Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research. 21(1):5485-555
Shasha Liao and Ralph Grishman. 2010. Using doc- ument level cross-event inference to improve event extraction. In Proceedings of the 48th Annual Meet- ing of the Association for Computational Linguistics, pages 789-797, Uppsala, Sweden. Association for Computational Linguistics.
Jiawei Sheng, Shu Guo, Bowen Yu, Qian Li, Yiming Hei, Lihong Wang, Tingwen Liu, and Hongbo Xu. 2021. CasEB: A joint learning framework with cas- cade decoding for overlapping event extraction. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLLP 2021, pages 164-174, Online. Association for Computational Linguistics.
Chu-Cheng Lin, Dongyeop Kang, Michael Gamon, and Patrick Pantel. 2018. Actionable email intent mod- eling with reparametrized mns. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 32.
Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020. Event extraction as machine reading com- prehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 1641-1651, Online. Association for Computational Linguistics.
Zhaoyue Sun, Jiazheng Li, Gabriele Pergola, Byron C Wallace, Bino John, Nigel Greene, Joseph Kim, and Yulan He. 2022. Phee: A dataset for pharmacovig- ilance event extraction from text. arXiv preprint arXiv:2210. 12560.

---

Peter Turney. 2000.
Learning algorithms for keyphrase extraction. Information retrieval, 2:303- 336.
Can you please send me summary of our meeting yesterday? (2) Deliver: The deliver act provides or commits to provide something, such as file, an answer to query, or information about events (e.g., the location of a meeting). Example: / will send you the summary report of our meeting. (3) Amend: An amend act requests or informs about a change in some earlier proposals, e.g.. change the meeting date or contact information in database. Example: Can you please update the sunmary report of our meeting? ?
Jan Ulrich, Gabriel Murray, and Giuseppe Carenini. 2008. A publicly available annotated corpus for supervised email summarization. In Proc. of aaai email-2008 workshop, chicago, usa.
Christopher Walker, Stephanie Strassel. Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilin- gual training corpus. Linguistic Data Consortium, Philadelphia, 57:45.
Wei Wang, Saghar Hosseini, Ahmed Hassan Awadallah, Paul N Bennett, and Chris Quirk. 2019. Context- aware intent identification email conversations. In Proceedings of the 42nd International ACM SIGIR Conference Research and Development in Infor- mation Retrieval, pages 585-594.
We also define three noun acts, which describe the event entities. (1) Data: Data can be a piece of information, such as a concrete file or an abstract fact. is typically defined with an "IdString" ge. "the summary report of our meeting yesterday") and a "Value" (e.g., an attached PDF file). The fact includes event-relevant information such as the Meeting Data (e.g., the date when meeting will be held) and the Action Data (e.g., the address where a package should be mailed). As we focus actionable events, we do not consider subjective in- formation (e.g., opinions) or objective information that is too complicated to be framed as data (e.g.. news information), but would cover very light "is-a" facts (e.g.. Skilling is the CEO of Enron, where the data IdString is "CEO of Enron" and the Value is "Skilling"). (2) Meeting: We define a meeting gathering of people for a discussion to achieve common goal for entertainment. We also con- sider a phone call or trip as a meeting. (3) Action: An action refers to an activity that has to be done or will be done, such as signing a document or sending a mail package. Note that the "activity" here does not include "meeting", which has been covered by the previous noun act. Similar to verb acts, each noun acts will be signaled by a certain trigger, to be illustrated in the next section.
Xiaozhi Wang. Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. Maven: A massive general domain event detection dataset.
Frank Wilcoxon, 1992. Individual comparisons by rank- ing methods. In Breakthroughs in Statistics: Method- ology and Distribution, pages 196-202. Springer.
Mingfeng Xue, Hang Zhang, and Jiancheng Lv. 2020. Key factors of email subject generation. In Neural Information Processing: 27th International Confer- ence, ICONIP 2020, Bangkok, Thailand, November 18-22, 2020, Proceedings, Part 27, pages 668 675. Springer.
Pengfei Yu, Zixuan Zhang, Clare Voss, Jonathan May, and Heng Ji. 2022. Building an event extractor with only few examples. In Proceedings of the Third Workshop on Deep Learning for Resoure Natu- rul Language Processing, pages 102-109.
Kexun Zhang, Jiaao Chen, and Diyi Yang. 2022. Focus on the action: Learning to highlight and summarize jointly for email to-do items summarization. In Find- ings of the Association for Computational Linguisties: ACL 2022, pages 4106.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, 7i Lin, Zhuohan Li. Dacheng Li, Eric. P Xing, Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. 2023. Judging jludae-a-judge with mt-bench and chatbot arena.
A.2 Complete Event and Argument Role Definitions
We now present all 10 event types and their re- spective argument roles (wrapped within "[- J" in examples). In total, they result in 76 argument roles at the event level by combing the roles from the verb and the noun act for each event type (c.g., for Request Data event, there are argument roles in- cluding Request Members, Request Date, Request Time, Request Attribute, Data Type, Data IdString, Data Value, and Data Owner). We also introduce several "meta semantic roles" with pre-defined
A Taxonomy Details
A.1 Verb and Noun Acts
We consider three verb acts. When introducing these acts, also indicate the corresponding Irig-
gers (underlined) that signal their presence. (1)
Request: The request act is triggered when the sender intends to perform an act of asking or or- dering something formally or informally. Example:

---

Act
Arguments
For Request Data/Meeting Data/Action Data, we introduce meta semantic role "Request Attribute" to indicate the attribute that the sender queries from the data. In practice, we consider four data at- tributes: Type, IdString, Value, and Owner.
Members Date
The recipient of the request event.
Requ
The date when the event will (or has) happen (or bappened).
est
Time
The time when the event will (or has) happen (or happened).
Attribute
The attribute requested for the noun such as data, meeting time, etc.
Deliver Data: The event is triggered when the sender provides or commits to provide certain data.
Members Date
The recipient of the corresponding event. The date when the event will (or has) happen (or happened),
Deliver
Example 1: Attached for your review
Time
The time when the event will (or has) happen (or bappened).
[the summary of our meeting] Dass kistring (Confirmation: Positive);
Confirmation
Acknowledgment the sender (posi-
tive/nggaive/uussue)
Example 2: I don't 't have [that] Dave MiStiing
Type
The amend operation (add/delcte/updatt))
Members
The person to oversee the amend event.
(Confirmation: Negative)
Amend
Time
The time when the amend action happens.
Date
The date when the amend action happens.
For Deliver events, we introduce "Confirmation" (positive, negative, or tentative as a meta seman- tic role, affirming if the sender can provide the requested data information (when the Noun Act is Data), or acknowledge their attendance in meet- ings or participation in action events (when the Noun Act is Meeting Data or Action Data). No- tably, the Confirmation role could be perceived form of "data" as well. In conversational email setting, people often reply with brief responses such as "Sure" or "No, it doesn't It work" when some- one makes a request. By introducing the Confirma- tion role, we can discern the sender's intent even though no concrete event information may be ex- tracted from short answer.
Type
The type of data (e.g.. PDF, is-a facts). A string describing or identifying the data. actual data
IdString
Data
Value
fact.
Owner
The person whom the data belongs.
Members
Attendees of the mecting.
Meel
Agenda
The topic of discussion for the meeting. reference nurne for the meeting.
ing
Name
Location
(physical virtual) place where the meeting will he held.
Date
The date on which the meeting will (or hus)
happen happened).
Time
The time at which the meeting will (or has)
happen (or huppened).
Members
Attendees of the activity. summary
Description
action.
Action
Date
The date on which the activity will (or has)
happen (or happened). at which the activity will (ur has) happen (or happened).
Table 3: Descriptions of argument roles for Verb (upper) and Noun (bottom) acts.
Amend Data: The event is triggered when the sender requests or indicates changes to a data record. In order to describe the type of change, we introduce a fixed set of "Amend Type" verbs including add, delete, or update. Additionally, we have observed that individuals frequently describe changes by providing context followed by the re- vision, as shown in Example 1. Consequently, to differentiate between the various roles, we intro- duce two labels, "Context" and "Revision", and replace all four argument roles (Table 3) for Data act with two sets of copies for each (e.g.. "Context: Data Type" and "Revision: Data Type" instead of the original "Data Type").. These modifications al- low for more precise differentiation and description of the different aspects of the event and its roles.
class spaces for some event types. The complete ar- gument role definitions for each Verb or Noun Act can be found in Table 3, and the list of 76 argument roles can be found in Table 4.
Triggers of each event (for both Verb Act and Noun Act) are underlined. Note that an event trig- ger could span over non-continuous words since people may not necessarily describe verbs and nouns consecutively. This also allows us to keep the trigger words as concise as possible rather than marking a continuous but much longer text span. Request Data: The event is triggered when the sender seeks data such as a file or a fact.
Example 1: Can [you] change [the
Example 1: Please send me [the sum-
budget] CVT.Dass AEString from [2K]exTRaa Vaine [3K] JuVV-Dato Valur (Amend Type: Up- date);
mary of our meeting] hom Wattiin (Request Attribute: Data Value);
Example 2: Who owns [the survey re- port] Dana Mannnng ? (Request Attribute: Data Owner)
Example 2: Can you please update [the summary report] Manigg of our meet- ing? (Amend Type: Update).

---

Request Events
Request Meeting
Request Request Data
Request Action
Marting Mambars
Messing Location
Roqued momber
Action Dati
Maeting Agenda
Meeting Date
Dica laString
Data Ownnrr
Action Members
Meeting
Request
Data Type
Action Description
Mozting Time
Respested Attrhbte
Action Time
Roquest Action Data
Request Meeting Data
Context: Action Time
Context: Roquest Memburs
Conteat: Moeting Date
Context: Moeting Time
Context Action Mambers
Conext: Action Date
Context: Meeting Agenda
Contect: Meeting Members
Description
Consext: Request
Moeting Location
Requesial Auribute
Conical: Modtnng Name
Requesial Atribute
Deliver Events
Deliver Pata
Deliver Action Pata
Deliver Moeting Pata
Members
Data ldString
Action Time
Meeting Members
Meeting Time
Dala Value
Deliver Time
Action Members
Mecting Name
Mecting Date
Deliver Date
Data
Action Deseripion
Menting Agenda
Meeting Location
Deliver Confirmation
Deliver Confimationn
Deliver Confirmation
Amend Events
Amend Data
Amend Meeting Data
Consext: Moeting Members
Moeting
Moeting
Amend
Revisim: Data Type
Contest: Amend Time
Revision: Meeting Monbeers
Conteet: Masting Location
Coniest: Messing Time
Revision: Amand Time Amend Type
Contex:: Data Value
Amend
Consext: Meeting Agenda.
Resision: Meeting Location
Revision: Meeting Time
Revision: Data Value
Revision: Meeting Agenda
Context: Moeting Date
Context: Amond Date
Contes: Amend Membeers
Context: Amera Members
Reviskos Amend Members
Revision: Amend Date
Table 4: All event arguments MAI.Ex. We remove arguments that are trivial (such as Deliver Members for events Deliver Action Data and Deliver Meeting Data) not frequent (such as Data Owner in Amend Data event). In total, we keep 76 argument roles in the final version of MAILEX.
Request Meeting:
The event is triggered when the
|Friday) Meee Date instead? (Amend
sender proposes a meeting.
Type: Update)
Example: [Alice] Members has pro- posed a meeting on (Tuesdaay] Meeting Date-
The event is triggered when the sender proposes an activity or an action (lay- ing sport, signing document, etc.).
Request Action:
stteeng
The sender triggers the event by requesting information about a certain meeting. By using the "Request Attribute" for the event, the sender could request more meet- ing attributes such the Date and Time of the meeting.
Request Meeting Data:
Example: Please (approve Alice's travel
request) Action Deserjtionn
The event is triggered when the sender seeks information about action event.
Request Action Data:
Example: Who lapproved the travel request) Action Descriptionn (Request Attribute:
Where is [the meet- (Tuesday) J Mectiing Date" (Request Attribute: Meeting Location)
Example:
ing] Meeting
Mngg/ngning..o Nome
Action Members)
Deliver Action Data: The event is triggered when the sender provides information about an action event. The "Confirmation" attribute serves the pur- pose of acknowledging the presence of the sender and any other individuals involved in the event.
Deliver Meeting Data: The event is triggered when the sender provides information about cer- tain meeting. The sender acknowledge the presence of both the sender and any other attendees using the "Confirmation" attribute.
Example 1: [John]nasion Menbeer lapproved
the travel request) Action Description (Confirma- tion: Positive);
Example: [Alice] will attend the [Tueedayyoome [Board meeting Jateetng (Confirmation: Positive)
Example 2: [Alice] J
Action Membors
has agreed
/deliver mail) Actis Description- (Confirma-
tion: Positive)
The event is triggered when the sender requests or informs of changes to an already proposed meeting event.
Amend Meeting Data:
B MAILEX Dataset Details
B.1 Annotation Details and Guidelines
Example: Can we reschedule [the meet-
MAILEX annotations were done in multiple rounds due to the challenges discussed in Section 3.1. For
ing I Morring Noms on [Tuesday] ExT Mooting Date

---

Event Type
%
Frequent Argument Roles
event type, overlapping trigger words, and over- lapping argument spans for the same role. Prob- ing into the annotations, we found that the non- overlapping partial text spans are typically trivial words such as an article "the". We use Jaccard sim- ilarity larger than .3 as the "overlapping" criterion. Threads with a total disagreement on event triggers and arguments were discarded. In total, we obtain 1,500 email threads covering 44K emails and ~ events.
Request
7.700
Data ldString (72%)), Request Members (23%), Request Date (2%)
Data
Request Ac- tion
14,985
Action Description (54%), Action Mem- bers (35%), Action Date (6%)
Request Meeting
2.775
Meeting Members (31%), Meeting Agenda (21%), Meeting Date (18%)
Request Ac- tion Data
2,456
Action Description (51%), Action Mem- bers (38%), ), Request Memhers (8%)
Request Meeting
0.541
Meeting Members (31%), Meeting Agenda (21%),, Meeting Date (18%)
Data
Deliver Data
20.452
Data kiString (48%), Data Value (39%),
Deliver Members (10%)
B.2 Examples of Partially Agreed and Disagreed Annotations
Deliver Ac-
34.439
Action Description (46%), Action Mem- bers (41%). Action Date (9%)
tion Data
In practice, most partially agreed annotations hap- pen when annotators inconsistently marked trivial words (c.g., an article "the") or referred to the same entity mentioned with different details (e.g.. "At- tached agreement report" and "Attached report"), while they agree on the actual trigger or argument concepts. This gives us k. value of 0.791 (i.e.. substantial agreement) for the trigger-event type IAA and 0.810 (i.c., almost perfect agreement) for the argument role IAA.
Deliver Meeting
5.176
Meeting Members (34%), Meeting Date (19%), Mocting Time (12%)
Data
Amend
2.054
Amend Members (26%). (Context) Data IdString (25%). (Rovision) Data Value (25%)
Data
Amend
0.569
(Revision) Mecting Time (22%)), (Re- vision) Moeting Date (19%). (Context) Moeting Name (16%)
Meeting
Data
Table 5: Distributions of event types (in percentage) and frequent argument roles in MAILEX. We have not included the rare events Amend Action Data (0.028%) and Non-Event annotations "O" (8.825) in the table.
We sampled few annotations with total or par- tial disagreement and manually analyze them. In most cases, the total disagreement was caused by task complexity and language ambiguity. For exam- ple, in one email, the sender informed the recipient of a "to-do list" to which one annotator marked it as a Deliver Data event since the sender delivered a list of the informative items, while the other an- notator considered it Request Action since the sender had instructed list of actions. Such dis- agreed annotations have been removed from our dataset. For partially disagreed cases, we often ob- served disagreement on trivial words, as discussed in IAA calculation (Section 3.1). We present more examples in Table 6.
consistent annotations, annotators were instructed to annotate one email at time, considering the email history as context (see Figure for the anno- tation interface). Each email could have multiple events, and annotators marked trigger words, event types, and argument roles. For trigger words, an- notators indicated the minimal span of words in the email that triggered an event. Event types were selected from pre-defined labels. Argument roles were annotated using the BIO format, with annota- tors marking the beginning (B) and inside (I) spans the arguments while leaving non-arguments out- side (O). For Amend events, "Context" and "Re- vision" were included in the BI tags (c.g., "B- CNT:Meeting Date" or "B-REV:Meeting Date"). Annotators also assigned pre-defined labels for meta semantic roles from pre-defined labels ac- cordingly.
B.3 Dataset Analysis
Event Types and Argument Roles Distribution. In Table 5, we present the distribution of event types and argument roles in MAILEX. We observe that events related to deliver acts are more frequent than others and argument roles such Members, Descriptions, and IdString are more frequent than Date and Time. Tabular Data in Email Text. As mentioned in Section 3.2, MAILEX could contains emails which have non-sequential sentence struc- ture such as Tables. Figure 4, we present an ex- ample table from MAILEX. For the sake of sim-
Two native English-speaking Computer Science students were recruited for the annotation task and were paid 12 USD per hour. Multiple rounds of training and discussions were conducted ensure an understanding of events and arguments. Each email was annotated twice by each annotator, and we retained event annotations with agreement on

---

(887) yonkhwuks, midrala
At test time, given each email in an email thread, we will first apply the trigger extraction model identify all trigger spans and their corresponding event types from the email. Then each trigger span and its type information will be fed to the argument extraction model for identifying the associated ar- gument roles.
28300" contrncta
hertares
Trease Kurket
2004
redara Hiraect sell
141
38923
Parketorg
2891
Treats So Monkllig lay
orgace Serry kerrcce New 2003
sis,
disigress Usee
echadirong priarity for there Sak contracee ce not resut: in dosect Med prinary
Figure 4: Example table from MAILEX. For the an- notation purpose, asked the annotator to annotate the table header as "Data IdString" for the event type "Deliver Data". The rest of the table rows were asked to be annotated "Data Value".
C.2 Templates for End-to-End Email EE
We present the templates for the task of end-to-end email EE in Tables 8-10. All the templates begin with a sentence concerning event type with a place- holder $trigger for the corresponding trigger span. Following that, the templates include place- holders for the arguments specific to each event type. It is worth noting that the template contents and argument placeholders can vary depending on the meta-semantic roles involved. For instance, dif- ferent templates are used when the sender expresses positive acknowledgment an event compared to when they express negative acknowledgment. This flexibility allows for adaptable and context-aware event extraction from emails.
plicity, asked annotators mark the headers of the table as the description of the table (Data IdString) if no better description has been speci- fied in the email. The rest of the rows were in- structed to be marked as actual data instances (Data Values). In our example, it means to mark the header "Contract
Contract Date" as Data IdString and the remaining rows from "107018. Nov 1 2001 as Data Value. One could also the row and column values to mark more complicated data instances (such as mapping each value in Contract column with each value in Shippers column and then with the values in Contract Date column). Modeling tables such as way presents more informative data to the end user while complicating the task formulation by introducing non-sequential structure. We leave this exploration to the future.
Shippers
C.3 Prompt Example for GPT-3.5
In Figure 8, we present the prompt design for us- ing GPT-3.5 for event extraction. For each evalu- ation instance from the test set, GPT-3.5 is tasked with processing K (K=5) demonstrations, each of which consists of context, current email, and out- put, in addition the task instruction and the event templates. GPT-3.5 is expected to produce a re- sponse by filling in the template for each event the current email with its trigger and corresponding arguments.
Supplementary Modeling Details
C.1 Additional Details about Sequence Labeling
D Implementation Details
Meta Semantic Role Prediction. As introduced in Section 2, some argument roles (e.g., the requested data attributes) have fixed, pre-defined class space. We formulate the identification of each of such argument roles as classification task, where the [CLS] representation will be used as standard BERT-based classification tasks. These additional classification models will be jointly trained with the aforementioned sequence labeling model for argument extraction.
D.1 Reproducibility Details
To train sequence labeling models, we used the BERT-large-
with a batch size of 4 and learning rate of 1e-5, For the generative ap- proach, we used BART-large with a learning rate of 3e-5 and a batch size of 2. All the models were optimized using Adam (Loshchilov and Hutter, 2017) for cross-entropy loss for 100 epochs. We tuned all the hyper-parameteer maintain maximum sequence length of 512 for all our fine-tuned models. When using BART, we trun- cate the input sequence from the left to retain the most recent history or the recent portion of an email. During training, we implement early stopping af-
the dev set. We
Training and Inference. In experiments, the trig- ger extraction and the argument extraction models will be trained independently. During the training time, the ground-truth trigger span and event type will be used for the argument extraction training.

---

Eraan Artolaaion inlerlece
Arrota Coliiir Hatieceerral
Light Thore
00/18:36
Current Turn:
2
6+
Figure 5: Our annotation interface. For each email thread, annotators were shown each email one by one. For each email, they tasked to select event types from drop-down menu and directly select the event triggers by clicking on words. Next, for each word, they annotate the word with the corresponding argument role (with the default being reflecting role).
Example
Annotator
Annotator
ENA won the for Lost Creek Fuel sale with a price CIG Gas Daily plus $.03.
Trigger: won
Trigger: won
Partial
Eveut Class: Deliver Action Data
Event Class: Deliver Action Data
Agree-
ment
Please file the completed hardcopy li- brary/fileroom.
Trigger: Please file
Trigger: Please file the completed hard-
Eveut Class: Request Action
copy
Event Class: Request Action
Team, Here an update on Oakhill: Ricki
Trigger: Here is update
Trigger: said
No
said he is sending us T & D only contract
Event Class: Deliver Data
Event Class: Deliver Action Data
Agree- ment
first thing tomorrow morning.
One of you UBS people with your big bonuses
Trigger: will have to pick Event Class: Deliver Action Data
Trigger: pick it
will have to pick it
Event Class: Roquest Action
Table 6: Examples for agreed and disagreed annotations. For partially agreed triggers, we keep the overlapped triggers ("won" and "Please file") while, for disagreed anmotations, we remove the corresponding event and arguments annotations from the final version of the MAILEX.
ter 5 epochs, monitoring Trigger Classification for trigger extraction and Argument Classification for argument extraction.
rameter to 0.0, which minimizes randomness the output, and the top_p parameter to 1, which restricts the model's choices to only the most prob- able tokens. These settings effectively discourage the GaT-based models from generating content that is not present in the input and encourage them focus on copying and reproducing the input con- tents.
Regarding in-context learning, we set a maxi- mum generation length of 300 tokens with greedy decoding. All experiments were conducted using the default turbo version within the date range of 03/01/2023 to 06/13/2023. In cases where the input demonstrations exceed GPT-3.5's token limitation of 4000, we left-truncate the input sequence en- sure it fits within the specified limit. To enforce content copying and prevent the generation extra- ncous information, we further adjusted the model settings. Specifically, we set the temperature pa-
Finally, for experiments involving "ground-truth triggers" with both the BART- and In-context Learning- based approaches, we feed the templates iteratively one by one.

---

Example Email
Don, Alloched is driniled list proveelures and ideus ov, MHER snul to the foarly crew. <MHEB procedure..eee <MHEE jdeoz.docs-
5.
Ground Truth Triggers: Alloohhd denatled lis (Deliver Duna) 2) Attacied Ideas (Deliver Data)
Cida
832
042
6:2
67
BERT
Trigger 1: Attacted a detuiled list of proceddures and ideas on MHE
Trigger 2: None
cila
30B
044
624
514
BART
Trigger 2:
Trigger 1: Attached detalled
Allached kloas
Please hoid Thursday, December 20th for the Board and the Commultre mectingg from 7:30 ox. 2:00 pmn. C.T.
6:1
Ground Truth Triggers: 1) Please hold Thursday. December 26th for the Board meerings (Roquest Moetigg) 3) Please hold Thurady, December 20th for the Connulttee moetings (Request Meeting)
315
Sereting
RONG
965
Trigzer 1: Please hold
BERT
Trigger 2: None
BART
Please hold Thursby, Daxember 20th for the Bound mectings Pletse hold Thursay, Devember 20th for the Committee maectings
Trigger 2:
Trigger I:
Ataacedd we are forwarding electrontc copics the ANGIS Preposal and fetter
Figure 6: Confusion matrix for event type extraction using BERT- sequence labeling. The majority of the confusion arises in noun acts, e.g., Deliver Meeting Data vs. Deliver Action Data.
Ground Truth Triggers: 1) Attcched we are forwarding electronic copies of
rener fetter (Deliver Datai 2) Arioched we aee forwunding elecimiic vopiees of the ANGIS Proporal (Deliver Data)
BERT
Trigger 1:
Atlached farwarding
Trigger 2:
None
BART
Trigger 1:
Trigger 2:
Attached we are forwarding a copies of the lester None
D.2 Hyperparameter Search
Table 7: Examples of extracting shared triggers by the BERT-based sequence labeling model and the BART model. green and red indicate correct and incorrect ex- tractions, respectively. For all the shared triggers, BERT fails to segment the trigger spans. On the contrary. BART can extract triggers when the trigger segments have shorter distance from the conjunction "and" (sec- ond example). For the third example, both models fail to identify the correct trigger segments. and notably, BART adds contents that are part of the current in- put. We observed that when shared triggers have a long distance in between the conjunctions (such as "and"), even BART struggles to retrieve them correctly.
The BERT- and BART-based models were fine- tuned for 100 epochs using early stopping, whereby training was stopped if the validation results did not improve for epochs. During the experimentation phase, we manually explored different learning rate values within the range [.01, .001, 0001, 100001, .000001] and batch sizes within the range [2, 4, 8, 16, 32]. The best model was selected based on its performance on the validation set.
D.3 Runtime and Devices
The fine-tuned experiments were conducted on NVIDIA A100 80 GB GPU cards. Training each BERT-based model took approximately 4 hours, while the full pipeline. including training and eval- uation, required approximately 8 hours. For the BART-based experiments, the training and eval- uation process took approximately 12 hours. In comparison, the GaT-based experiments were com- pleted within approximately 3-4 hours due to the time constraints imposed by the platforms used,
E.2 Shared Triggers
In Section 5.2.2, we concluded that both fine-tuned models encounter challenges in identifying shared triggers, particularly when the distance between them is significant. We show examples in Table 7. For the first two examples, where the shared trig- gers are relatively close, we observe that BART successfully extracts both triggers, whereas BERT either fails to segment them accurately or fails to detect them altogether. Similarly, in the last exam- ple, both the BERT and BART-based approaches struggle to identify such triggers as the distance between them increases.
Additional Experimental Analyses E
E.1 Classifying Noun Act Triggers
E.3 In-context Learning Analysis
In Section 5.2.2, we discussed the models' inability properly classify the noun acts associated with triggers. In Figure 6, we present the confusion matrix outlining the confusion between the event classes.
Analysis of Actual Argument Values. We con- ducted an analysis of 50 randomly sampled in- stances where GPT-3.5 extracted erroncous argu- ments. We categorized these errors into the fol-

---

where "Deliver Member" is an extra placeholder not provided by the event template. 2) Missing Placeholders: Another common problem while generating the templates was identified to miss out the placeholders for arguments. For example, for the Deliver Data Events, it frequently leaves the "Data Value" placeholders as in "Event Deliver Data is triggered by trigger where Data idString I, (missing Data Value I) of Data Type is or will be delivered Deliver Time
40.0%
Placshokders
Extra Arguments
Missing Arguments P Placeholders
Incorrect Arguments Placehoddess
46.0%
Figure 7: Error distribution while gpt-3. 5..5--uur gen- erates argument placeholders.
to Deliver Members on Deliver Date I" 3) Incorrect Place- holders: For some generated templates, we found that 3pt-3.5-tures incorrectly copies placehold- ers for different events than specified. For exam- ple, for the event Deliver Action Data it generated template "Event Deliver Action Data is triggered by trigger where Action Description is or will be performed by Action Members at | Context: Action Time on | Context: Action Date I" which contains "Context:" label before date and time that are not part of the Deliver Action Data templates.
lowing types: Extra Arguments: These errors oceur when the model includes arguments that are not actual arguments for the events. For exam- ple, email signatures were mistakenly captured as member arguments for certain event types. 2) Miss- ing Arguments: In some cases, when generating filled-in templates with argument values, the model completely misses certain arguments and generates argument placeholders instead. 3) Semantically Correct but not Exact Match Arguments: This type of error arises when the model attempts to sum- marize argument values such as Meeting Agenda or Action Description. Although semantically cor- rect, these arguments are not recognized as exact match arguments by our evaluation script and are therefore considered incorrect. 4) Semantically Incorrect Arguments: These involve arguments which are incorrect. While categorizing such emrors we also include cases where the model adds triv- ial details to the arguments (such as "the" before names). 5) Semantically Incorrect and Fabri- cated Arguments: Arguments that do not appear in the instructions or the current emails fall into this category and are considered both semantically incorrect and fabricated. We provide examples responding to each error class in Table 11.
We randomly sampled 50 generated templates with such errors and plot an error distribution chart showing errors while generating placeholders for arguments. As Figure 7 depicts, most of the rors were made because of missing the correct incorrect placeholders (Category and above).
Analysis on Arguments Placeholders. We also found that gpt-3. 5-turbo struggles in generating consistent argument placeholders (when actual ar- gument values are not expressed in the email). We categorized such errors into 3 categories: 1) Extra Placeholders: While generating the templates, 3. 3pt-3.5-turbo generated more placeholders than expected. For example, for the event Deliver Ac- tion Data, it generated the template Event Deliver Action Data is triggered by trigger where Action
or will performed by Action Members at Time on Date | delivered to Deliver Memberl,

---

Task Instr and Templates
Your task to extract events the emall, along with their corrapooningg triggers and arguments. need to identity the templetes each event and fill the missing intormation with the oxtracted riggers and arguments. The unfllid tomplates are:
Event Request Meating triggered triager where. Mesting Nama is requested among Maating Members at Mesting Time: on
Meeting
Request Mecting Location to discuss Mecting Agenda
triggrrd by triggerl where from Context: Request members to be delivered
idString
Context: Type bay Context:
requested
Context: Request Time on Contoxt: Request Date
(detalls skipped)
Event Amend Data triggered by trigger where
Context: Data idString! Contoxt: Data Value is
Revision:
Event Amond Meeting Data Iriggered by| trigger where, Forl Context: Meeting Name among I Context: Moeting Members Context:
Meeling Time on Context: Meeting Date I all Context: Meeling Location to discuss I Context: Meeling Agenda dale requested to be updated Revision: Meeting Date from Context: Amend Memberss
Context:
Demonstration
Current Email:
group Toney
new
lesting slage will the Build
revieed
accordingly please see the altached emails Would you have sommone in your group work on this and us know
(detatts skipped)
The filed tamplates are:
Event
triggered by Request Time Request Date
Context: Type Context: Data
requested from you
(details skipped)
Context:
Demonstration 2
Pleuse know
any
Emesto,, plsase provide following information: 1 Please contimm
commencad commercial operation 7/8/2001 The oven Commercial Operations start date wes July 09 2001 . 2) (details skipped)
The filed templates are:
Event
triggered by calculation
calculation,, reviewed signed
Context: Type by
requested from Ernesto delivered Context: Request
Request Date
Event Request Data triggored by ... (dotals skipped)
(demonstrations K; details skpped)
Test Case
Context:
Please come by 502 al your converience komorrow as UDS require you complete missing information on your application form Kind
regards, Karen Buckley Human Resources Tel: 713 345 4667
The filled templates
Figure 8: Prompt for event extraction using GPT-3.5. In experiments, K=5, and ensure that the selected 5 demonstrations cover all event types and arguments.

---

Event Type
Template
Example
Example: (Alice (Alice howw entm has proposed
a.meerting (Tuerday Iosoe fow
Request
Event Request Meeting is triggered by trigger where, MMeeting| is requested among Meeting Members at Time on Date at Location to discuss Agenda
Template: Event Request Meeting triggered by proposed a meeting
Meeting
where, Meeting is requested among Alice at Time on Tuesday
Location to discuss| agenda
Example: Please send me [the suomorry
o/rrr meeting (Request Attribate: Data Value);
When Reghttr is Value: Event Request Data triggered by |tigger where. Data| | of Type
Request
by Owner is requested from Request Members be delivered Time Date
Template: Request Data is triggered by Please send me the summaryy where, the summory of our meeting of TType]
Data
When Reqaar Data Owner: Event Request Data triggered by trigger | where, Owner of
Data of Type |is requested from Request Members to be delivered at Time on Date
requested from Request
Members to be delivered Time Date
Example:Pleose Tapproue Alice's 's travel
Request
Event Request Action is triggered by trigger| where, Action is requested from | Action Members at TTmee| on Date
Template:Event Request Action triggered by Please approve where, approve Alice's travel reguest
Action
requested from Action Members al
Time on Date
When RegAttris Members: Request Action triggered by trigger where.
Example:
Who Japproved travel (Roquest Attribute:
Action Members requested for Action at Time on Date from Requesi Members |:
reque maceen?
When ReqAttr is Action Date: Event Request Action Data triggered by trigger where, Date
Action Members)
Roquest
is requested Action by Action Members at Time from Request Members
Action
When RegAttr is Action Time: Event Request Action Datu triggered by trigger where. Time
Template:
Event Request Action
Data
is requested for Action by Action Memberr on| Date | from Request Membeer |:
Data is urigered by | Who approved I where, Action Members is requested for approved the request Time on Date from Request Members
When RegAttr is Action Deseription: Event Request Action Data is triggered by Irigger| where.
Action Description requested for Action by Action Members Date from Request Memhers
When Reghttr is Meeting Members: Event Request Meeting Data is triggered by trigger where.
Meeting Members is requested for Mesting at | |'Tmee | on Date at Location to discuss
Agenda from Request Members
Example: Where is (the meetingluuwee tow [Thvesdyy] Tameene (Request Attribute: Meeting Location)
When Regdltt Date: Event Request Meeting Data is triggered by trigger where, Date is
requested for |Meeting among Meeting Members at Tame at Location to discuss Agenda]
from Requesi Members
Request Meeting
When RegAmt Time: Event Request Meeting Data is triggered hy trigger | where, Time
Example: Event Request Meeting Data is riggered by: Where is the meet- ing where, Location is requested for the meeting among Meeting Members Time Tuesday In discuss Agenda from Request Members
is requested for Meeting among Mecting Members on Date at Location to discuss
Agenda from Request Members |:
When RegAnr Location; Event Request Meeting Data is triggered by trigger where, Location
is requested for Mecting among Meeting Members at Time on Tuesday to discuss
Agenda from Request Members |:
When RegAtrr Agenda: Event Request Moeting Data is triggered by trigger| where, Agenda is requested for Meeting | ammng (Meeting) Members at Time on Date at Location from
Request Members
Table 8: Generation templates for end-to-end Request event extraction.

---

Event Type
Template
Example
Example: Attached your review [the
summary of our meeting stwoe
When Confirmation Positive: Deliver Data triggered trigger
Value of Type| is will be delivered to Deliver Members at Time Date |:
Template:
Deliver
Deliver
When Confirmation is Negative: Event Deliver Data is triggered by trigger where, Data
uiggered Attached the summury where, the summury of our meeting Value Type is will delivered
Value of Type is will delivered Deliver Members
Date
When Confirmation Unsur:: Event Deliver Data triggered trigeer where. Data
Value of Type| is will probably be delivered to Deliver Members at Time on Date
Deliver Members,
Time
Date
Example: [Allce has agreed to [deliver mail) postooos (Confirnation: Positive)
triggered by trigger where, Action will be performed Action Members [at Time on Date
Confermation Poxitive:
Deliver
When Confirmation Negative: Event Deliver Action Data is triggered by urigeer where.
Action
Action not or will not be performed by Action Members at TTine on Date |:
Template:
Event Deliver Action Data triggered agreed where, deliver moid is or will he performed by Alice I
Data
When Confrmmtion Unsure: Event Deliver Action Dara triggered hy trigger where,
Action| is probably or will probably he performed by Action Members aa Time on Date
at Time on Date
When Confirmation Posirive: Deliver Meeting Data triggered by trigger where, Meeting is will be attended Meeting Members at Time Date at Location to
Example: [Alice]usc will attend [Tuesday [Board meeting tow-
discuss Agenda |;
(Conlimmation: Positive)
Deliver Meeting
When Confirmation Negutive: Event Deliver Meeting Data is triggered by: trigger| where. be allended by| Meeting Members al Time on Dute Location
Meeting not
Deliver Moeting Data is triggered by will aztend meeting |
Template:
Data
to discuss Agenda
When Confirmation Unzu:: Event Deliver Meeting Data triggered by trigger| where. Meeting prohably probably attended by Meeting Members Time on Date at Location discuss Agenda
where, Board Meering is or will be attended by Alice day at Location discuss Agenda
Table 9: Generation templates for end-to-end Deliver event extraction.

---

Event Type
Template
Example
Example:
Iyou momn change
When Amend Action is Update: Event Amend Duta is triggered by trigger where, Cut:
[budge]loens crone from [2K]errssss
Data Cut: Value is or requested to be updateed to Rev: Value from Amtnnd Mombers at
[3K] [3K]avmans (Amend Type: Updale))
Time on Datelp I:
Amend
When Amoud Action is Add: Event Amend Data triggered by rrggeer where, for Cnt: Data
Template: Event Amend Data
Data
Rev: Value is or requested to he added from Amend Memhers| at Time on Date |;
triggered by chonge the budget where, for buulget |. |K is or requested be
When Amend Action is Defeie: Event Amecd Data is triggered by uigger where, for Cut:
Data Value or requested be removed from| Amend Members at T'ime on Dute
updated from you at ['Tme on
Date
To update meeting members: Event Amend Meeting Data is triggered by trigger where, for Moeting among Cut: Meeting Members| at Cat: Time on Car: Date at Cat: Location
discuss Cat: Agenda meeting members requested he updated to Rev: Masting
Memhers from Amend Members
To updote meeting date: Event Amend Meeting Data is triggered hy trigger | where, for meeting Moeting among Cat: Meeting Members at Time on| Cat: Dute
Location to discuss| Cut: Agenda date requested to updated to Rev: Date [rom Amend
Members :;
update meeting time: Event
Meeting Data triggened by trigger where, meeting
Meeting among Cut: Meeting Members at Cat: Time on Cat: Date at Cut: Location to discuss Cat: Agenda I, time is or requested to be updated to Rev: Time from Amend
Members
To updlare meeting location: Event Amend Meeting Data is triggered by trigger | where, for meeting Meeting among Cnt: Meeting Memhers at Cnt: Time on Cnt: Date at Cnt: Location to discuss| Cul: Agenda i, location is or requested to be updated to Rev: Location
Ameud Menubers
Example: Can we reschedule Itee meet-
To apulate meeting agenda: Event Amend Meeting Data triggered by trigger | where, for meeting Meeting among Cnt: Meeting Members att Cnt: Time on Cnt: Date at Cnt: Location discuss Agenda agenda is requested updated Rev: Agenda from
ing ongloorannng new [Tuesdayl comen
Fridayylaowwe instend? (Amend Type: Uplate)
Amend Members
Amend Meeting
Event Amend Meeting Data is triggered by | rescheduue the meeting where, for moeting the meet- ing among Cut: Meeting Members CCt: Time| on Cnt: Theeslay at Cot: Location to discuss Cut: Agenda date requested to updated Friday from Amend Members!
Template:
Event Amend Mecting Data triggered trigger where, for Meeting among Cat: Meeting Memhers ar CCt: Time on Oht: Date it | Cnt: Location to discuss Cai: Agenda (. meeting members Rev: Meeting Members is requested to be added from Amend Memberrs
To add meeting members:
Dala
To add meeting dute: Event Amend Meeting Data triggered by Irigger where. for Meeting among | Cnt: Mceting Memhers at Cnt: Time at Cnt: Location to discuss Cne: Agenda
date
or requested to added from Amend Momhers
add moeting time: Event Amend Meeting Dara triggered by trigger where. for Meeting
among | Cut: Moeting Members| on| Cal: Date at Cut: Location to discuss| Cul: Agenda |-
time Rev: Time is or requested to be added from Amend Members |:
To add meeting location: Event Amend Meeting Data triggered by trigger where, for
Meeting among Cot: Meeting Members at Cot: Time| on Cot: Date to discuss Cut:
Agenda location Rev: Location is or requested to be added from Amend Members |:
To add meeting agenda: Event Amend Meeting Data triggered by trigger where, Meeting among Cni: Meeting Members at Cut: Time on Cut: Date at Cut: Location
agenda Res: Agenda| is or requested to be added from Amend Memhers |;
To remove meeting members: Event Amend Meeting Data is triggerd by trigger wweree, for Meeting umong Cat: Meeting Members at Cat: Time on Cat: Date at Cut: Location to Rev: Moeting Members or requested be removed
discuss Cul: Agenda meeting members
from Amend Members
Table 10: Generation templates for end-to-end Amend event extraction.

---

Error Category
Example: Laura Giambrune discovered a PNR balance of 10,000 Dih on the above noted contract
Gold Template: Event Deliver Action Data is triggered by discovered I where. I discovered a. PNR balance of 10,000 Dth on the above noted contract is or be performed by I Laura Giambrone Action Time Action Date
G.T....-turbo: Event Deliver Action Data is triggered discovered where, , Laura Gianbrone di disveeeded the above noted contract is will be performed by Laurr Giambrune a Action Time oo Action Date
PNR balance of 10,000
Semantically Incorrect Arguments
GPT-3.5 davinci-003: Event Deliver Action Data is triggered by I discovered I where. II discovered a PNR balance of 10,000 Dth is or will be performed by Laura Giambrone at Action Time on Action Date
Example: How about we all go dinner Mark on Sunday night How does that sound ? -Eric
Gold Template: Event Request Meeting is triggered by How about we all go to dinner where. Meating Name is requested among we at night on Sunday at Mark 's to discuss dinner! I
GPT-3.5-trbo: Event Request Meeting is triggered by How about we all go to dinner I where, , there is no trigger or context in the enail to fill in any of the event templates.
Semantically Incorrect and Fabricuted Arguments
GPT-3.5 davinei-003: Event Request Meeting triggered How about we all go dinner where. Mecting
Name is requested among I we all a Meeting Time I oa Sunday night at Moeting Location to discuss How does that sound
Example: Hi Shelley I have meeting at 2:00-3:30 30 and meeting Kevin at 4:00. Can I come your office at 3:30
Gold Template: Event Amend Meeting Data is triggered by I Can come to your office 3:30 ? where For
Contex: Meeting Name among 11 lllat Context: Meeting Time on I Context: Meeting Date at your office to discuss I Context: Mecting Agenda time is or requested to be updated to I 3:30 I from I Context: Amend
Members
GPT.3.5-trro: Event Amend Meeting Duta is triggered Can I come your office 3:30 where For |Metingg Name I among Meeting Members I at Context: Meeting Time on Context: Moeting Date I at Shelley's office
Semantically Correct but not Exact Matched Arguments
or requested to updated to 3:30 from Context: Amend Members
GPT-3.5 davinci-003: Event Amend Mecting Data is triggered by I Can come to your office 3:30 where.
For the meeting among II and Kevin at I Context: Meeting Time on I Context: Meeting Date at Context: Meeting Location agenda discuss or requested to added Shelley
Example: Where the website for ERCOT 2JD
Gold Template: Event Request Data triggered by Where is the website I where. / the website for ERCOT of 1Context: Data Type by Context: Data Owner is requested from Context: Request membees to he delivered
Context: Request Time om Context: Request Date I
GPT-335-turb:: Event Request Data triggered by Where is the website I where. / the website for ERCOT of |Context Data Type by I JD is requested from Rika to be delivered at Context: Roquest Time on Context:
Extra Arguments
Request Date
GPT-3.5 davinci-0033 Event Request Data is triggered by Where the website where, , website for ERCOT of Context: Data Type by Context: Data Owner is requested from ID to be delivered at Context: Request Time Context: Request Date
Example: Attuched 2 nd draft of the gtrly letter reflecting more of my changes, repts- ( rev2-rj ). ).doc
AAP memo for girly
Gold Template: Event Deliver Data triggered by Attached is 2 nd draft where. . nd draft the qtrly letter reflecting more of my changes I. AAP memo for qurly repts- ( (rev2-rj ) ddoe of Data Type is or will be delivered to Deliver Members at Deliver Time on Deliver Date!
GPT-3.5-turbo: Evvent Deliver Data is triggered by Attached is 2 nd draft I where. I 2nd draft I, Data Value lo Data Type is will be delivered to Deliver Members at Deliver Time Deliver Date!
Missing Arguments
GPT:3.5 davinci-0033: Event Deliver Data is triggered by Attached is nd draft where. 1 nd draft of the qtdy letter I. Data Value of Data Type is or will be delivered to Deliver Members at Deliver Time on Deliver Date
Table II: Analysis of in-context learning-bass approaches when ground truth triggers are fed to variants of GPT-3.5. The errors made by models are highlighted in red and the ground-trui templates are highlighted in gold.


Reviews:
[{"review_id": "rL74iKrFVH", "reviewer": "Reviewer_uHNK", "summary": "", "questions": "1. How do you deal with the partially agreed annotations? What's the criteria for a better annotation?\n2. For the case in line 425, have you tried to limit the vocabulary space to the words in the email context?\n3. In line 404-408, do you consider two spans are partial matching if they overlap with each other?\n4. In the BART model, does the input context contain the concatenation of templates for all events? In Li et al., 2021, they firstly identify the event triggers and only include the template of a specific event in the input context, extracting arguments for one event at a time.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank you for taking the time to review our paper and giving thoughtful comments! \n\n**1. Clarity regarding the criteria for designing the schema.**\n\nOur criteria for designing the event schema are inspired by Cohen et al. (2004), who defined task-oriented speech acts for email communication. We extended the definition of these speech acts to perform the task of event extraction on emails. Please refer to Section 2.1 for clarification and Appendix A.1 for definitions of the verb and noun acts used in our dataset. \n\nWe differentiate the event type “Request Meeting Data” from “Request Data” to accommodate for the intrinsically different types of “Data” in email events. Specifically, our schema considers three types of data: (a) “Meeting Data”, which refers to facts related to specific meetings (e.g., meeting date, location), (b) “Action Data”, which refers to facts related to a specific action or an activity (e.g., a deadline for approving a budget request, the person who approved the request, etc.), and (c) “Data” refers to all other information irrelevant to meetings and actions, such as PDF files sent in emails.  \n\nNotably, doing this allows us to easily connect the event extraction task with downstream applications; for example, when an EE model extracts meeting information, a downstream email reminder can be automatically set up to provide additional assistance. However, this will not be feasible if we simply merge all types of data information into one coarse category. While in our current dataset, only a small portion of “Request Meeting Data” events were annotated, we included this fine-grained event type to facilitate follow-up research and broader downstream applications in the future.  \n\nIn our revised draft, we will clarify the definition of different data types and emphasize this motivation. \n\n**2. Concern about long and non-continuous arguments.** \n\nWe first clarify that, unlike triggers, arguments are always continuous spans in our dataset. As mentioned in lines 261-270, arguments can span across multiple words or sentences due to the presence of non-named entities such as “Meeting Agenda” or “Meeting Location”, but they are always continuous. \n\nAnnotating long-span arguments, as the reviewer pointed out, is indeed challenging. Therefore, we provided very careful instructions to our annotators, including instructing them to annotate only a minimal span for each argument. In addition, when two annotators partially agreed on an annotated argument span, we retained their overlap, which further reduced any redundant words in each argument. Due to the space limit, we have included most of our annotation details in Appendix B, including examples of partially agreed annotations in B.2, from which we can confirm no ambiguity in the actual argument annotation, despite its long length. \n\n**3. Details on partially agreed annotations and criteria for better annotation.**\n\nFor partially agreed triggers (but with agreement on the event type), we retained the overlapped word spans. For partially agreed arguments (but similarly with agreement on the event type and having overlapped trigger spans), we similarly retained the overlapped word span. When two annotators did not agree on the event type or made no overlap in their annotated triggers, we abandoned the annotations completely. We will clarify it in our revised draft.  \n\nIn terms of the criteria for better annotations, we included the annotation guidelines in Appendix B.1 due to space limit. In summary, to facilitate the annotation, we provided detailed definitions and examples of each event type, an intuitive interface for annotation (Figure 4), and instructions to reduce confusion. Annotators also received training before starting the formal annotation. \n\n**4. Have you tried to limit the vocabulary space to the words in the email context?**\n\nNo, we have not tried the suggested solution. First, we’d like to note that limiting the decoder vocabulary to only words from the current email is not a feasible solution because the decoder has to copy words from the template as well. Second, even if we constrain the decoder to select words from both the current email and the templates, it won’t solve the low recall issue. In fact, even without any constraint (as in our current experiments), BART has learned to decode words only from these two sources (i.e., the current email and the templates), so adding the suggested constraint will not really change the model performance. In our observation, the high precision and low recall of BART in argument extraction typically happened when BART copied correct but incomplete argument spans, and those spans are typically long-length, non-named entity spans. To solve the issue, future work should look into how to improve BART in understanding non-named entity arguments.  \n\n**5. In L404-408, do you consider two spans a partial match if they overlap with each other?**\n\nYes, we consider overlapped spans as partially matched. As mentioned in lines 404-408, we follow this strategy to encourage a more fair comparison (such as both the spans “The president” and “president” will be acceptable). When there is only a partial match (as opposed to a complete, exact match), the model will incur a lower argument identification/classification performance. Similar strategies have been adopted in previous studies such as Li et al. (2021). \n\n**6. For BART, do you input all the event templates?**\n\nYes, for BART, the input context contains the concatenation of templates for all events. We designed it for the interest of end-to-end EE modeling. We hypothesized that triggers and arguments should correlate with each other. Extracting them in an end-to-end fashion may improve the overall performance as has been demonstrated in previous works (Hsu et al., 2022, Lu et al., 2021, and Du et al., 2022., inter alia). \n\nIn contrast, Li et al. (2021) follow a pipeline approach similar to our BERT-based method, except that for argument extraction they adopted BART rather than BERT. While this is also feasible, it could lead to cascading of errors (Du et. al, 2021) as errors from the triggers extraction stage could not be used to reinforce the argument learning. \n\nNotably in Table 2, we did provide results comparing BERT and BART for argument extraction, when both were fed with ground-truth triggers. In this experiment, we only input the targeted event template and gold trigger to BART and expect it to extract arguments for one event at a time. It is shown that BART obtained a worse argument classification F1 than BERT. \n\n\n**References:** \n\nHsu*, I.H., Huang*, K.H., Boschee, E., Miller, S., Natarajan, P., Chang, K.W., & Peng, N. (2022). DEGREE: A Data-Efficient Generative Event Extraction Model. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL). \n\nLu, Y., Lin, H., Xu, J., Han, X., Tang, J., Li, A., Sun, L., Liao, M., & Chen, S. (2021). Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 2795–2806). Association for Computational Linguistics. \n\nDu, X., Li, S., & Ji, H. (2022). Dynamic Global Memory for Document-level Argument Extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5264–5275). Association for Computational Linguistics \n\nDu, X., Rush, A., & Cardie, C. (2021). Template Filling with Generative Transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 909–914). Association for Computational Linguistics. \n\nLi, Sha, Heng Ji, and Jiawei Han. \"Document-Level Event Argument Extraction by Conditional Generation.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021. \n\nCohen, William, Vitor Carvalho, and Tom Mitchell. \"Learning to classify email into “speech acts”.\" Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. 2004.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper has several strengths:\n- Overall, it is well-written with a clear and well-motivated introduction.\n- The high-quality annotated dataset will facilitate event extraction research in the email domain and prompt the study of email assistance.", "reasons_to_reject": "1. There is a lack of clarity regarding the criteria of designing the schema. For example, why differentiate the event type \"Request Meeting Data\" from \"Request Data\" as the previous event only cover 0.71% of all events?\n2. The average number of words of arguments is 7.41 and some of them are non-continuous spans. This will cause difficulties and ambiguities in identifying the arguments.", "questions_to_authors": "1. How do you deal with the partially agreed annotations? What's the criteria for a better annotation?\n2. For the case in line 425, have you tried to limit the vocabulary space to the words in the email context?\n3. In line 404-408, do you consider two spans are partial matching if they overlap with each other?\n4. In the BART model, does the input context contain the concatenation of templates for all events? In Li et al., 2021, they firstly identify the event triggers and only include the template of a specific event in the input context, extracting arguments for one event at a time.", "paper_topic_and_main_contributions": "The paper presents the first dataset for event extraction in the email domain, with a new taxonomy covering 10 event types and 76 arguments. The authors implements three models, including sequence labeling, template-based generative extraction, and few-shot in-context learning, on the dataset and make comprehensive analysis on the challenges of the task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UugkVM48jZ", "reviewer": "Reviewer_FtbS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We deeply appreciate your recognition of the usability of the MailEx dataset and our analyses of unique traits and challenges in performing email event extraction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The dataset is novel and has addressed some of the critical characteristics of email data. The proposed dataset can be a valuable study resource in this unexplored research area.\n2. The authors have adequately described all the sections of the manuscript, including the taxonomy of the dataset, annotation details, and a detailed analysis of the dataset statistics and unique characteristics. \n3. The authors have described three types of baselines for the proposed dataset and compared the results. The authors have provided the experimental setting of the models that improve the reproducibility of the baseline models.\n4. The authors also performed the qualitative analysis of the results based on the unique challenges of event and argument extraction of the proposed dataset.\n", "reasons_to_reject": "1. No reason to reject the work. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents an email event extraction dataset that facilitates comprehensive downstream tasks on email data. The task is novel and has not been addressed previously in the literature. The authors first propose a taxonomy of the event extraction dataset before annotating the dataset. To understand the challenges of the dataset, the authors have proposed three types of experiments and reported the results. The authors also discuss some of the unique challenges of email event extraction.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xoOXULs9wI", "reviewer": "Reviewer_LSSd", "summary": "", "questions": "How many domains do these emails come from? All business emails from Enron?\n\nHave you tried to annotate more fine-grained event types? If yes, what makes you decide not to include them?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments on our work!  \n\n\n**1.  Comment on the proposed event type being not fine enough and based on business context** \n\nOur taxonomy is inspired by Cohen et. al. (2004), which is fine-grained for email communication in a task-oriented setting and is not limited to only business. We emphasize in lines 100-115 that the focus of this work is to extract commonly seen events from daily email communications. Although business communication is intuitively a major use case of emails, as elaborated in our response to Q3, Enron itself contains a significant portion of emails sent for personal purposes. Based on our preliminary study on 3 email corpora, we keep our focus on the 10 most common event types enabled by three verb and three noun acts. Some of our event types are very broadly defined; for example, a “Meeting” in our definition can be both a formal meeting (business or not, in-person or via phone calls) and an informal social gathering (e.g., birthday party, a camping trip, impromptu outing, etc.).  \n\nWhile most context of MailEx is about business usage, we highlight that (1) as stated above, our event schema itself is not limited to the business context, and that (2) to the best of our knowledge, Enron is the only open-source, publicly available, and large-scale email corpus. Despite the context limitation (which we also discussed in Limitations), we believe MailEx is still a valuable resource for future research in conversational, email EE.  \n\nFinally, while creating a holistic event ontology by “includ(ing) all the event types that other datasets use” could be an interesting idea, this is not trivial as it requires schema-matching across different datasets and their contexts. Training EE models to understand such a comprehensive event ontology will also require a sufficiently complex dataset, which as far as we know is not available. Instead, our focus in this work is to propose a new task of “conversational EE” in the email context, providing baseline models and inspiring future research to look into this rarely explored field. \n\n**2. Concern about event templates being not easy to understand and their impact on GPT-3.5’s result.** \n\nThe templates designed for performing EE on emails have been carefully designed, inspired by previous research such as Du et. al (2022). These templates are easy to understand and would not be the reason for GPT-3.5’s low performance. As evidenced by Table 2, the performance of BART-based approaches using the same templates is on par with the sequence labeling-based approaches. \n\nInstead, we hypothesize the low performance of the GPT-based model could have occurred due to the intrinsic inability of LLMs to understand the nuanced languages and their implied event information. In Section 5.2.5, we provided more careful categorization and analysis of the errors made by GPT-3.5.  \n\n\n\n**3. How many domains do these emails come from? All business emails from Enron?**\n\nMost of the emails in MailEx are in a business context, although sometimes the emails could be personal as well (e.g., exchanging thoughts about their job or work with their relatives). In fact, prior work (Alkhereyf and Rambow, 2017) found that a significant portion of Enron emails were written for personal purposes.  \n\n \n**4. Annotation and inclusion of more fine-grained event types.** \n\nWe first clarify that our current event schema definition is already fine-grained. For example, we distinguish among Meeting Data, Action Data, and other general types of Data, as Reviewer uHNK also noticed. Our schema follows the email speech acts of Cohen et al. (2004) and retains only the most frequent event types in the Enron dataset. There are also noun acts such as “Opinion” and “Thanking” which we drop from our schema because they do not directly connect to “events”.  \n\nIncluding additional event types that are rare in Enron is not very meaningful due to the lack of data context. While this is a limitation caused by the choice of Enron, we note again the limited email data resources available for NLP research, and that MailEx is the first dataset utilizing available resources toward filling the gap of conversational, email event extraction research.  \n\n**References:**\n\nCohen, William, Vitor Carvalho, and Tom Mitchell. \"Learning to classify email into “speech acts”.\" Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. 2004. \n\nDu, X., Li, S., & Ji, H. (2022). Dynamic Global Memory for Document-level Argument Extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5264–5275). Association for Computational Linguistics \n\nAlkhereyf, Sakhar, and Owen Rambow. \"Work hard, play hard: Email classification on the Avocado and Enron corpora.\" Proceedings of TextGraphs-11: the Workshop on Graph-based Methods for Natural Language Processing. 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new dataset for Email domain itself is interesting and could be important for real applications. \nThe proposed baseline methods do not perform very well on this dataset, which leaves a large space for future works to further improve.", "reasons_to_reject": "Though the dataset is very interesting, the granularity of proposed event type is not fine enough. All the event type is about a specific task for business, like \"schedule a meeting\" and \"request data\". This is more like emails in a company. However, if we are talking about event in emails, emails should also cover all the event types of other texts. People can talk about any events in email, but the key difference is: the context of email is more daily, which may make the task harder. In my opinion, the authors should consider extend the type set to include all the event types that other datasets use. \n\nThe result of GPT-3.5 could be problematic, since the event template provided in the prompt is not easy to understand.", "questions_to_authors": "How many domains do these emails come from? All business emails from Enron?\n\nHave you tried to annotate more fine-grained event types? If yes, what makes you decide not to include them?", "paper_topic_and_main_contributions": "This paper focuses on the task Event and Argument Extraction in Email domain. This paper designs a new dataset that annotates event types and arguments on emails, and they evaluate three different baselines on the dataset: BERT-based sequence labeling, BART-based e2e extraction, and GPT3.5, and they show that these methods' performance is low.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZF7OeHgpOK", "reviewer": "Reviewer_UBTU", "summary": "", "questions": "Q1. How did you decide upon the three verb acts: request, deliver, and amend? Were their other email actions which were ignored?\nQ2. Do you think pretraining the language modeling objective of the models on conversation data would help your cause? If so, why did you not use it? I am assuming such specialized pretrained models are readily available.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude for your valuable time and thoughtful assessment. Your recognition of the utility of MailEx, along with its distinctive complexities in executing event extraction from emails, is deeply acknowledged. Furthermore, we express our gratitude for your engagement with our comprehensive study using novel baselines and highlighting the shortcomings of LLMs in a few-shot setting. \n\n**1. Suggestions on improving the writing of sections and statistical tests to further support model comparison.** \n\nThank you for the suggestion. We will refine the writing of section 4.3 to improve comprehension. \n\nFor statistical tests, we have performed a Wilcoxon signed-rank test (Wilcoxon, 1945), which is used when comparing two matched samples. We compared BERT’s and BART’s trigger classification and argument classification F1 performance on the dev set, setting a significance level $\\alpha$ = 0.05. \n\nWe observed for triggers, with a p-value of 0.0329, that BERT is significantly better than BART. However, for arguments, with a p-value of 0.287, we conclude that there is no statistically significant difference between BART and BERT. These results support our model comparison claims made in lines 417-422. We will add the results to our revised draft. \n\n**2. How did you identify the three verb acts: request, deliver, and amend? Did you ignore other email actions?** \n\nThe verb acts, i.e., request, deliver, and amend, are inspired by the previous work on email speech act (Cohen et al., 2004). In the original paper, Cohen et al. proposed a total of 9 verb acts but then reduced them to a subset of 5 verbs based on their frequency. We further trim them off by (1) merging “propose” with “request” into “request” due to their shared purpose of encouraging recipients to undertake actions, whether through suggested courses or direct appeals for assistance, and (2) merging “commit” and “deliver” into “deliver” but introducing a “confirmation” slot (lines 147-155) to indicate the intent of commitment.  \n \n\n**3.  Pretraining the language modeling objective of the models on conversation data** \n\n \n\nThis is a good suggestion, but it seems to be a very open-ended question.  \n\nFirst of all, we identify a domain gap between the conversation data that existing chat LMs such as Vicuna (Chiang, Wei-Lin, et al.), FastChat (Zheng et al.), and Koala (Geng et al.) have been pre-trained on, and the target email conversation. That is, the business-dominant context and the formal language usage in EmailEx could be very different from the conversation data used to pre-train these chat LMs.   \n\nAlternatively, instead of using these pre-trained chat LMs, one could also pre-train the base LM (i.e., BERT or BART in our work) on conversational email data, before fine-tuning them in the annotated EE task. We think that this could help improve the model's performance and would be excited to try it in our future work! \n\nFinally, we note that in our in-context learning experiment, gpt-3.5-turbo is better optimized for chats compared with text-davinci-003, but the former performs worse than the latter. This shows the challenge of Email EE beyond simply conversational language understanding, e.g., copying exact long-text argument spans without hallucination. \n\nIn this work, we focused on introducing the dataset and analyzing the performance of classic EE models on the new task and hence were not able to experiment with the suggested idea thoroughly, but we are excited to investigate this research question carefully in our future work. \n\n**References:** \n\nCohen, William, Vitor Carvalho, and Tom Mitchell. \"Learning to classify email into “speech acts”.\" Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. 2004. \n\nChiang, Wei-Lin, et al. \"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.\" See https://vicuna. lmsys. org (accessed 14 April 2023) (2023). \n\nLianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, & Ion Stoica. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. \n\n \n\nXinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, & Dawn Song. (2023). Koala: A Dialogue Model for Academic Research. \n\n \n\nFrank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin 1(6):80–83.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper makes several contributions towards the information extraction field. First, they define a taxonomy for event extraction in corporate email threads. Second, they release the MailEx dataset which would be a valuable resource for evaluating IE models. Third, they contribute novel event extraction models and highlight the deficiencies of few-shot LLMs.", "reasons_to_reject": "I did not find any major technical weaknesses in the paper. However, the authors could improve the language of certain sections such as the end-to-end extraction model discussion to improve comprehension. I would also recommend that the authors show statistical test results to support their model comparison claims.", "questions_to_authors": "Q1. How did you decide upon the three verb acts: request, deliver, and amend? Were their other email actions which were ignored?\nQ2. Do you think pretraining the language modeling objective of the models on conversation data would help your cause? If so, why did you not use it? I am assuming such specialized pretrained models are readily available.", "paper_topic_and_main_contributions": "This paper studies the event extraction task in email threads. Event extraction comprises of event trigger identification, event trigger classification, argument identification, and argument classification. The authors define a taxonomy of event types (verb + noun acts) and argument types. They apply this taxonomy to annotate 1.5K email threads between Enron employees from the Enron email thread dataset. The experiment sequence labeling, sequence to sequence, and few-shot generative approaches on the event extraction tasks. The authors observe that sequence labeling models perform the best on this closed domain task. Finding non-contiguous triggers and non-named entity arguments are challenging.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a dataset for event extraction from a conversational email threads. The dataset could be a valuable contribution to the research community. The choice of event types is very narrow and in line with a very old literature. The intuition or criterion of the schema used is not well motivated and prompting should be described in a more comprehensible manner.
Decision: Accept-Main