Venue: NAACL 2024

# Event Detection from Social Media for Epidemic Prediction

Tanmay Parekh\({}^{\dagger}\) Anh Mac\({}^{\dagger}\) Jiarui Yu\({}^{\dagger}\) Yuxuan Dong\({}^{\dagger}\)

**Syed Shahriar\({}^{\dagger}\) Bonnie Liu\({}^{\dagger}\) Eric Yang\({}^{\dagger}\) Kuan-Hao Huang\({}^{\lx@sectionsign}\) Wei Wang\({}^{\dagger}\) Nanyun Peng\({}^{\dagger}\) Kai-Wei Chang\({}^{\dagger}\) +\({}^{\dagger}\)Computer Science Department, University of California, Los Angeles ++\({}^{\lx@sectionsign}\)Department of Computer Science, University of Illinois Urbana-Champaign {tbarekh, weiwang, violetpeng, kwchang}@cs.ucla.edu**

###### Abstract

Social media is an easy-to-access platform providing timely updates about societal trends and events. Discussions regarding epidemic-related events such as infections, symptoms, and social interactions can be crucial for informing policymaking during epidemic outbreaks. In our work, we pioneer exploiting Event Detection (ED) for better preparedness and early warnings of any upcoming epidemic by developing a framework to extract and analyze epidemiological events from social media posts. To this end, we curate an epidemic event ontology comprising seven disease-agnostic event types and construct a Twitter dataset SPEED with human-annotated events focused on the COVID-19 pandemic. Experimentation reveals how ED models trained on COVID-based SPEED can effectively detect epidemic events for three unseen epidemics of Monkeypox, Zika, and Dengue; while models trained on existing ED datasets fail miserably. Furthermore, we show that reporting sharp increases in the extracted events by our framework can provide warnings 4-9 weeks earlier than the WHO epidemic declaration for Monkeypox. This utility of our framework lays the foundations for better preparedness against emerging epidemics.

## 1 Introduction

Early warnings and effective control measures are among the most important tools for policymakers to be prepared against the threat of any epidemic [8]. World Health Organization (WHO) reports suggest that \(65\%\) of the first reports about infectious diseases and outbreaks originate from informal sources and the internet [23]. Social media is an important information source here, as it is more timely than other alternatives like news and public health [7], more publicly accessible than clinical notes [10], and possesses a huge volume of content.1 This underscores the need for an automated system monitoring social media to provide early and effective epidemic prediction.

Footnote 1: A daily average of 20 million tweets were posted about COVID-19 from May 15 – May 31, 2020.

To this end, we pioneer to leverage the task of Event Detection (ED) for epidemic prediction. ED involves identifying and categorizing significant events based on a pre-defined ontology [24, 1]. Compared to existing epidemiological keyword and sentence-classification approaches [1, 19], ED requires a deeper semantic understanding. This enhanced understanding aids in more effective disease-agnostic extraction of epidemic events from social media. By reporting sharp increases in extracted epidemic-related events, we can provide early epidemic warnings, as shown for Monkeypox in Figure 1 - highlighting the applicability of ED for epidemic prediction.

Existing ED datasets are unsuitable for establishing a framework to extract epidemic-related events from social media, as they focus on general-purpose events in news and wikipedia domains,

Figure 1: Number of reported Monkeypox cases and extracted events by our trained ED model from May 11 to Nov 11, 2022. Arrows indicate how our system could provide early epidemic warnings about 4-9 weeks before the WHO declared Monkeypox as a concern. We also show other baseline models for comparison.



while other epidemiological works are disease-specific and too fine-grained (SS 6). Thus, we construct our own epidemic ED ontology and dataset for social media. Our created ontology comprises seven event types - _infect_, _spread_, _symptom_, _prevent_, _cure_, _control_, _death_ - chosen based on their relevance for epidemics, frequency in social media, and their applicability to various diseases. We further validate our ontology through clinical sources and public health experts. For the dataset, we choose Twitter as the social media platform and focus on the COVID-19 pandemic. Using our curated ontology and expert annotation, we create our dataset **SPEED** (**S**ocial **P**latform based **E**pidemic **E**vent **D**etection) comprising \(1{,}975\) tweets and \(2{,}217\) event mentions. We complete our ED framework by training ED models Du and Cardie (2020); Hsu et al. (2022) on SPEED. Overall, SPEED provides disease-agnostic coverage of epidemic events for social media; thus, serving as a valuable dataset for epidemic prediction.

To validate the utility of our ED framework for disease-agnostic epidemic prediction, we perform two evaluations for three unseen diseases Monkepox, Zika, and Dengue. First, we evaluate if our framework trained on our COVID-only SPEED dataset can detect epidemic events for the unseen diseases. Experiments reveal that our framework can successfully extract epidemic events, providing gains up to \(29\%\) F1 over the best few-shot model and \(10\%\) F1 gain over supervised models trained on limited target disease data.

Our second evaluation validates if aggregation of our extracted events can provide early epidemic warnings. Comparing our extracted events with the actual reported cases, we show that our framework can provide warnings up to 4-9 weeks earlier than the WHO declaration for the Monkeypox epidemic (Figure 1). Such early warnings aided with timely action can potentially lead to 2-4x reduction in the number of infections and deaths Kamalrathne et al. (2023). These results underscore the strong utility of our dataset and framework for upcoming epidemic prediction and preparedness.

The contribution of this work is threefold, first, we pioneer to utilize Event Detection to develop an effective framework capable of extracting events from social media and providing early warnings for any unforeseeable epidemic. To support the proposed framework, our second contribution is the design of a disease-agnostic social-media tailored ontology and dataset SPEED. Our final contribution is extensive experiments to demonstrate the inadequacy of existing methods and the substantial improvements achieved by models trained on SPEED. This signifies the pivotal role of our dataset and framework in enhancing the efficacy of epidemic prediction. We release the data, code, and trained models at [https://github.com/PlusLabNLP/SPEED](https://github.com/PlusLabNLP/SPEED).

## 2 From Event Detection to Epidemic Prediction

Given a social media post, Event Detection (ED) Sundheim (1992); Doddington et al. (2004) extracts and classifies significant events of interest. By designing disease-agnostic epidemic-based events, we aim to train ED models to extract epidemic events from social media posts for any possible disease. By detecting abnormal influx in the trends of extracted epidemic events from social media, we can thus provide early epidemic warnings for any possible disease, as we show for Monkeypox in Figure 1. Existing epidemiological approaches Lejeune et al. (2015); Lybarger et al. (2021) are simple keyword or sentence classification-based and less accurate. Other works like COVIDKB Zong et al. (2022) and ExcavatorCovid Min et al. (2021) are disease-specific and utilize events for building knowledge bases. To the best of our knowledge, we are the first ones to leverage event detection to extract epidemic events from social media and provide early warnings for any possible disease.

Formal Task DefinitionFollowing ACE 2005 guidelines Doddington et al. (2004), we define an **event** to be something that happens or describes a change of state and is labeled by a specific **event type**. An **event mention** is the sentence wherein the event is described. Each event mention comprises an **event trigger**, which is the word/phrase that most distinctly highlights the occurrence of the event. **Event Detection** is technically defined

Figure 2: Illustration for the task of Event Detection. Event mentions: Event _symptom_ and trigger _sneezed_ (1st sentence), Event _infect_ and trigger _positive_ (2nd sentence), Event _death_ and trigger _died_ (2nd sentence).

 

as the task of identifying event triggers from sentences and classifying them into one of the predefined event types (defined by an **event ontology**). The subtask of identifying event triggers is called **Trigger Identification** and classification into event types is **Triger Classification**[1]. Figure 2 shows examples for three event mentions for the events _symptom_, _infect_, and _death_.

## 3 Ontology Creation and Data Collection

We choose social media as our document source as it provides faster and more timely worldly information than news and public health [9] and is more publicly accessible than clinical notes [50]. Owing to its public access and huge content volume, we consider **Twitter2** as the social media platform and consider the recent **COVID-19 pandemic** as the primary disease.

Footnote 2: [https://www.twitter.com/](https://www.twitter.com/)

Existing epidemiological ontologies are typically disease-specific, too fine-grained, or limited in coverage (SS 6 and Table 8). Similarly, standard ED datasets don't comprise epidemiological events and mostly focus on news or Wikipedia domains (SS 6). Due to these limitations, we create our own event ontology and dataset SPEED for detecting disease-agnostic epidemics from social media. Figure 3 provides a brief overview of our data creation process, with further details discussed below.

### Ontology Creation

Taking inspiration from medical sources like BCEO [47], IDO [1], and the ExcavatorCovid [13], we curate a wide range of epidemic-related event types. Next, we merge similar event types across these different ontologies (e.g. _Outbreak_ event type). To create a disease-agnostic ontology, we filter out event types biased for specific diseases (e.g. _Mask Wearing_ for COVID-19) and create disease-agnostic definitions using aid from public-health experts. Finally, we categorize these events into three abstractions: personal (individual-oriented events), social (large population events), and medical (medically focused events) types. We report our initial ontology comprising \(18\) event types in Table 9 and share additional specifications in SS A.1.

**Social Media Relevance** To tailor our curated ontology for social media, we conduct a deeper analysis of the event types based on their frequency and specificity. Our goal is to filter and merge event types that occur less frequently and less distinctively in social media. To this end, using human expertise and external tools like Thesaurus,3 we first associate each event type with specific keywords. Then we rank the event types based on the specificity and frequency of their keywords in social media posts. Based on this ranking, we merge and discard the lower ranked event types (e.g. _Respond_ and _Prefique_). Furthermore, we conduct human studies and merge event types to ensure better pairwise distinction (e.g. _Treatment_ is merged with _Cure_). We provide the keywords associated with each event type and the example social media posts in Table 10 along with additional details in SS A.2.

Footnote 3: [https://www.thesaurus.com/](https://www.thesaurus.com/)

#### Ontology Validation and Coverage

Elemental soundness is ensured for our ontology since it is derived from established epidemiological ontologies. To further certify this soundness, two public health experts (epidemiologists working in the Department of Public Health) validate the sufficiency and comprehensiveness of our ontology and event definitions. To verify if our ontology is characteristic of any disease, we assess our ontology coverage for four diverse diseases by estimating the percentage of event occurrence in disease-related tweets. Notably, we observe a coverage of \(50\%\) for COVID-19, \(44\%\) for Monkeypox, \(70\%\) for Dengue, and \(73\%\) for Zika (details in SSA A.4). Our coverage is better than the coverages of standard ED datasets ACE [1] (27%) and ERE [21] (43%), in turn, confirming the

Figure 3: Overview of our dataset creation process with three major steps: Ontology Creation, Data Processing, and Data Annotation.

 

robust disease coverage of our ontology.

Our final ontology comprises seven primary event types tailored for social media, disease-agnostic, and encompassing crucial aspects of an epidemic. We present our ontology in Table 1 along with event definitions and example event mentions.

### Data Processing

To access a wide range of tweets related to COVID-19, we utilized the Twitter COVID-19 Endpoint released in April 2020. We used a randomized selection of **331 million tweets** between May 15 - May 31 2020, as our base dataset. For preprocessing tweets, we follow Pota et al. (2021): (1) we anonymize personal information like phone numbers, emails, and handles, (2) we normalize any retweets and URLs, (3) we remove emojis and split hashtags, (4) we filter out tweets only in English.

**Event-based Filtering** Most tweets in our base dataset expressed subjective sentiments, while only 3% comprised mentions aligned with our event ontology.4 To reduce annotation costs, we further filter these tweets using a simple _sentence embedding_ similarity technique. Specifically, each event type is linked to a seed repository of \(5\)-\(10\) diverse tweets. Example seed tweets per event are shown in Table 11 in SS A.3. Query tweets are filtered based on their sentence-level similarity Reimers and Gurevych (2019) with this event-based seed repository.5 This step filters about \(95\%\) tweets from our base dataset, leading to 20x reduction in the annotation cost.

Footnote 4: Based on keyword-based study conducted on 1,000 tweets

Footnote 5: We use a filtering threshold of \(0.9\).

* wherein we over-sample tweets linked to less frequent types (e.g. _prevent_) and under-sample the more frequent ones (e.g. _death_). Such a uniform sampling has proven to ensure model robustness Parekh et al. (2023)
- as also validated by our experiments (SS B)
- and in turn, would make SPEED generalizable to a wider range of diseases. In total, we sample \(1{,}975\) tweets which are utilized for ED annotation.

### Data Annotation

For ED annotation, annotators are tasked with identifying whether a given tweet mentions any event outlined in our ontology. If an event is present, annotators are required to identify the specific event trigger. We design our annotation guidelines following the standard ACE dataset Doddington et al. (2004) and amend them through several rounds of preliminary annotations to ensure annotator consistency. Additional details are provided in SS C.

**Annotator Details** To ensure high annotation quality and consistency, we chose six experts instead of crowdsourced workers. These experts are computer science students studying NLP and well-versed for ED. They were further trained through multiple rounds of annotations and feedback.

**Inter-annotator agreement (IAA)** We used Fleiss' Kappa Fleiss (1971) for measuring IAA. We conduct two phases of IAA studies: (1) _Guideline

\begin{table}
\begin{tabular}{l|p{86.7pt}|p{113.8pt}} \hline \hline
**Event Type** & **Event Definition** & **Example Event Mentions** \\ \hline \multirow{2}{*}{Infect} & The process of a disease/pathogen invading host(s) & 1. Children can also **catch** COVID-19 ... \\  & The prosces of a disease spreading or prevailing massively at a large scale & 2. If you have antibodies, you **had** the virus. Period. \\ \hline \multirow{2}{*}{Sprdom} & Individuals displaying physiological features indicating the abnormality of organisms & 1. (user) (user) Still **coughing** two months after being infected by this stupid virus ... \\  & Individuals trying to prevent infection of a disease & 1. – wearing mask is the way to **brevent** COVID-19 \\  & A disease & 2. – an **d**antibody that has been successful at **blocking** the virus \\ \hline \multirow{2}{*}{Control} & Collective efforts trying to impede the spread of an epidemic & 1. Social Distancing **reduces** the spread of covid ... \\  & & 2. (user) COVID is still among us! Wearing masks **saves** lives! \\ \hline \multirow{2}{*}{Cure} & Stopping infection and relieving individuals from infections/symptoms & 1. ... **recovered** corona virus patients cant get it again \\ \cline{2-2}  & End of life of individuals due to an infectious disease & 2. – patients are **treated** separately at most places \\ \hline \hline \end{tabular}
\end{table}
Table 1: Event ontology comprising seven event types promoting epidemic preparedness along with their definitions and two example event mentions. The trigger words are marked in **bold**.

 

_Improvement:_ Three annotators participated in three annotation rounds to improve the guidelines through collaborative discussion of disagreements. IAA score rose from \(0.44\) in the first round to \(0.59\) (\(70\) samples) in the final round. (2) _Agreement Improvement:_ All annotators participated in three rounds of annotations to boost consistency. IAA score improved from \(0.56\) in the first round to a strong \(0.65\) (\(50\) samples) in the final round.

**Quality Control** We further ensure high annotation quality through: (1) _Multi-Annotation:_ each tweet is annotated by two annotators, disagreements resolved by a third, and (2) _Flagging:_ annotators flag ambiguous annotations, resolved by a third annotator via discussion. These, coupled with good IAA scores, ensure the high quality of our annotations.

### Data Analysis

Our dataset SPEED comprises seven event types with \(2{,}217\) event mentions annotated over \(1{,}975\) tweets. We compare SPEED with other ED datasets like ACE Doddington et al. (2004), ERE Song et al. (2015), M\({}^{2}\)E\({}^{2}\)Li et al. (2020), MLEE Pyysalo et al. (2012), FewEvent Deng et al. (2020), and MAVEN Wang et al. (2020) in Table 2. We show how other datasets focus on the news, biomedical, general, and Wikipedia domains, while SPEED is the first-ever ED dataset for social media, specifically Twitter. Furthermore, none of the previous datasets comprise any of the epidemiological event types present in SPEED (SS D.1).

Comparable DotasizeSince we only focus on \(7\) event types, SPEED has relatively lesser number of sentences and event mentions. However, SPEED has a high \(316\) average mentions per event type (column 5 in Table 2), more than most other standard datasets. We compare the distribution of event mentions per sentence with other ED datasets like ACE and MAVEN in Figure 4. We observe that the event density of our dataset is less than MAVEN but better than ACE. This shows that SPEED is fairly dense and reasonably sized ED dataset.

Diverse and ChallengingWe show the diversity of trigger words in SPEED and compare it with other datasets in Table 3. We note that SPEED has a strong average number of triggers per event mention. This demonstrates how SPEED is a diverse and challenging ED dataset.

## 4 Epidemic Prediction

For our ED framework, we utilize our curated dataset SPEED to train various ED models (SS 4.1). To validate the utility of models for the application of epidemic prediction, we perform evaluations using two tasks: (1) Epidemic event detection and (2) Early warning prediction. Epidemic event detection performs a formal ED evaluation of the models for detecting epidemic-based events. On the other hand, early warning prediction practically evaluates if the extracted events by the model can be aggregated to provide any early epidemic warnings.

Since SPEED focuses solely on COVID-19, we conduct these epidemic prediction evaluations for three unseen epidemics of _Monkeypox_ (2022), _Zika_ (2017), and _Dengue_ (2018). These diseases are fairly distinct too, as Monkeypox causes rashes and rarely fatal, Zika causes birth defects, and Dengue causes high fever and can b

Figure 4: Distribution of number event mentions per sentence. Here % indicates percentage.

\begin{table}
\begin{tabular}{l c c c c l} \hline \hline \multirow{2}{*}{**Dataset**} & \#Event Types & \# Sent & \#EM & \multicolumn{1}{c}{**Avg. EM**} \\  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & **per Event** & **Domain** \\ \hline
**ACE** & \(33\) & \(18,927\) & \(5,055\) & \(153\) & News \\
**ERE** & \(38\) & \(17,108\) & \(7,284\) & \(192\) & News \\
**M\({}^{2}\)E\({}^{2}\)** & \(8\) & \(6,013\) & \(1,105\) & \(138\) & News \\
**MILE** & \(29\) & \(286\) & \(6,575\) & \(227\) & Biomedical \\
**FewEvent** & \(100\) & \(12,573\) & \(12,573\) & \(126\) & General \\
**MAVEN** & \(168\) & \(49,873118,732\) & **707** & Wikipedia \\
**SPEED** & \(7\) & \(1,975\) & \(2,217\) & **317** & Social Media \\ \hline \hline \end{tabular}
\end{table}
Table 2: Data Statistics for SPEED dataset and comparison with other standard ED datasets. # = “number of”, Avg. = average, Sent = sentences, EM = event mentions.

 

tions, we utilize and modify the raw Twitter dumps provided by Thakur (2022) for Monkeypox and Dias (2020) for Zika and Dengue.

### Epidemic Event Detection

To validate if our SPEED-trained models can extract events for any epidemic, we perform traditional ED evaluation of these models for unseen diseases of Monkeypox, Zika, and Dengue. Following Ahn (2006), we report the F1-score for trigger identification (**Tri-I**) and classification (**Tri-C**).

Data SetupTo train our ED models, we split the SPEED into 80-20 split for training and development sets. For testing, we sample tweets from the Twitter dumps of Monkeypox, Zika, and Dengue. Since the original data doesn't comprise event-based annotations, we utilize the same human experts who annotated SPEED to annotate the raw tweets for ED and create the evaluation dataset. We provide statistics for our data setup in Table 4. We release the training COVID data along with this evaluation data for Zika, Monkeypox, and Dengue for future benchmarking.

ED ModelsFor training models using SPEED for our ED framework, we consider the following supervised models: (1) DyGIE++ (Wadden et al., 2019), (2) BERT-QA Du and Cardie (2020), (3) DEGREE Hsu et al. (2022), (4) TagPrime Hsu et al. (2023). We utilized the TextEE framework Huang et al. (2023) to implement these models and provide more details in SS E.

Baseline ModelsAs baselines, we consider zero-shot ED models (**Zero-shot**) that do not train on any supervised data and solely utilize the event definitions. We consider the following zero-shot models: (1) TE Lyu et al. (2021), (2) WSD Yao et al. (2021), (3) ETypeClus Shen et al. (2021). Additional model implementation details is provided in SS E. We also consider transferring from existing datasets (**Transfer from Existing Datasets**) by training models on standard ED datasets like ACE Doddington et al. (2004) and MAVEN Wang et al. (2020) without fine-tuning on epidemic ED data.

As stronger baselines, we also consider models utilizing epidemic ED data. Here, we consider models using few-shot target disease data without any model training (**No Training**) like: (1) Keyword Lejeune et al. (2015), an epidemiological model utilizing curated event-specific keywords to detect events, and (2) GPT-3.5 Brown et al. (2020), a large-language model (LLM) using GPT-3.5-turbo with seven target disease in-context ED examples. Finally, we consider super-strong baselines training ED models on limited target epidemic data (**Trained on Target Epidemic**). Specifically, we sample 300 event mentions for the target disease (exclusive from the test data) and use human experts to annotate these tweets (as done for COVID annotations) to create the target epidemic data. Note that these models are added for comparison, but they are practically infeasible for epidemic prediction, as it takes 3-6 weeks after the first infection to collect such target disease data.

\begin{table}
\begin{tabular}{l|r r r} \hline \hline  & \multicolumn{1}{c}{**Disease**} & \multicolumn{1}{c}{**\# Sent**} & \multicolumn{1}{c}{**\# EM**} \\ \hline
**Train** & COVID & \(1,601\) & \(1,746\) \\ \hline
**Dev** & COVID & \(374\) & \(471\) \\ \hline
**Test** & Monkeypox & \(286\) & \(398\) \\  & Zika + Dengue & \(300\) & \(274\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Statistics for data splits for epidemic event detection evaluation. # = “number of”, Sent = sentences, EM = event mentions.

\begin{table}
\begin{tabular}{l c c|c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c|}{**Monkeypox**} & \multicolumn{1}{c}{**Zika + Dengue**} \\  & **Tri-I** & **Tri-C** & **Tri-I** & **Tri-C** \\ \hline \multicolumn{5}{c}{Zero-shot} \\ \hline TE & 16.70 & 12.11 & 12.69 & 9.06 \\ WSD & 22.04 & 4.35 & 27.93 & 5.85 \\ ETypeClus & 18.31 & 6.78 & 13.99 & 5.33 \\ \hline \multicolumn{5}{c}{Transfer from Existing Datasets} \\ \hline ACE - TagPrime & 4.80 & 0 & 23.64 & 0 \\ ACE - DEGREE & 12.15 & 5.14 & 14.47 & 0 \\ MAVEN - TagPrime & 29.16 & 0 & 33.97 & 0 \\ MAVEN - DEGREE & 27.94 & 0 & 32.04 & 0 \\ \hline \multicolumn{5}{c}{No Training} \\ \hline Keyword & 36.40 & 25.09 & 25.93 & 21.69 \\ GPT-3.5 & 42.23 & 35.33 & 53.22 & 14.27 \\ \hline \multicolumn{5}{c}{Trained on Target Epidemic} \\ \hline BERT-QA & 59.8 & 54.08 & 64.99 & 57.02 \\ DEGREE & 59.58 & 54.12 & 67.34 & 62.9 \\ TagPrime & 55.57 & 49.65 & 70.92 & 63.93 \\ DyGLE++ & 55.83 & 50.31 & **73.24** & **65.65** \\ \hline \multicolumn{5}{c}{Trained on SPEED (Our Framework)} \\ \hline BERT-QA & **67.38** & **64.17** & 67.95 & 56.81 \\ DEGREE & 62.95 & 61.45 & 64.14 & 56.63 \\ TagPrime & 64.71 & 61.92 & 62.08 & 54.47 \\ DyGLE++ & 62.76 & 59.82 & 66.4 & 56.4 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluating ED models trained on SPEED for detecting events for new epidemics of Monkeypox, Zika, and Dengue in terms of F1 scores.

 

ResultsWe present our results in Table 5. Firstly, none of the existing data transfer, zero-shot, or no training-based models perform well for our task, mainly owing to the domain shift of social media and unseen epidemic events. Overall, ED models trained on SPEED perform the best, thus **demonstrating the capability of our ED framework to detect epidemic events for new diseases**. Compared with models trained on the target epidemic, SPEED-trained models provide a gain of \(10\) F1 points for Monkeypox and a reasonable performance for Zika and Dengue. This outcome is particularly encouraging, as it **demonstrates the resilience of our framework, making it highly applicable during the early stages of an epidemic, when minimal to no epidemic-specific data is accessible**.

### Early Warning Prediction

As a practical validation of the utility of our framework, we evaluate if SPEED-trained ED models are capable of providing early warnings for an unknown epidemic. More specifically, we aggregate the extracted event mentions by our framework over a time period and report any sharp increase in the rolling average of detected events as an epidemic warning. For evaluation, we compare it with the actual number of disease infections reported in the same period. Naturally, the earlier we provide an epidemic warning, the better the framework is deemed. For this evaluation, we choose Monkeypox as the unseen disease and its outbreak from May 11 to Nov 11, 2022, as the unknown period. We utilize the Twitter dump by Thakur (2022) as the datasource. We also include four other baselines for comparison: (1) MAVEN, an ED model trained on existing ED dataset MAVEN Wang et al. (2020), (2) Clustering, a zero-shot classification baseline based on ETypeClus Shen et al. (2021), (3) Keyword, a no-training baseline based on a previous epidemiological model Lejeune et al. (2015), (4) COVIDKB, a BERT-based classification model trained on a previous COVIDKB Zong et al. (2022) dataset.

ResultsWe report the number of epidemic events extracted by the BERT-QA trained on SPEED along with the actual number of Monkeypox cases reported in the US6 from May 11 to Nov 11, 2022, in Figure 1. As indicated by the arrows, our model could potentially provide two sets of early warnings around May 23 (9 weeks earlier, when first cases were detected) and June 29 (4 weeks earlier, when cases started rising) before the outbreak reached its peak around July 30. Comparatively, MAVENTrained model fails completely, while the clustering, keyword, and COVIDKB models have super weak trends to provide any kind of warnings. In fact, all ED models trained on our SPEED data are capable of providing these early signals as shown in Figure 5 (further event-wise analysis in Appendix F). To further validate if the early warnings are raised owing to some actual Monkeypox epidemic discussion, we show some tweets classified as epidemic events by our system in Table 6. These tweets demonstrate that our system is picking the right signal to provide early epidemic warnings. Overall, this robust outcome underscores the **ractical utility of our framework to provide early epidemic warnings and ensure better preparedness for any potential epidemic.**

Footnote 6: As reported by CDC at [https://www.cdc.gov/poravirus/mpox/response/2022/mpx-trends.html](https://www.cdc.gov/poravirus/mpox/response/2022/mpx-trends.html)

## 5 Analysis and Discussion

In this section, we provide additional analyses to support the utility of our ED framework and also provide intuitions for why our framework works better than previous works.

### Event-based Disease Profiling

Our ED framework offers the additional utility of generating event-based disease profiles using public sentiments. These disease profiles can be generated by plotting the percentage of mentions per event type extracted by our framework. Using 500k tweets, we depict the profiles for COVID, Monkeypox, and Zika+Dengue in Figure 6.

Distinctive profiles emerge for each disease; COVID majorly comprises _control_, Monkeypox exhibits a bias toward _infect_ and _spread_, while

Figure 5: Number of reported Monkeypox cases and the number of extracted events from our four trained models from May 11 to Nov 11, 2022.

 

Zika+Dengue emphasizes _control_ and _death_. These trends align with the higher fatality rate of Zika and Dengue Paixao et al. (2022), recent discoveries of transmission routes of Monkeypox Kozlov et al. (2022), and the need for mass public control measures for the COVID pandemic Guner et al. (2020). Relatively, Monkeypox also shows low mentions for _death_, _cure_ - which aligns with low fatality and no available cure for Monkeypox Kmiec and Kirchhoff (2022). Overall, these profiles can provide policymakers with valuable insights about new unknown outbreaks to implement more informed and effective interventions.

### Why does SPEED generalize?

We provide a qualitative analysis of why COVID-based SPEED helps detect epidemic events for other unseen diseases compared to previous epidemiological works Collier et al. (2008); Lejeune et al. (2015) and attribute it to the difference in the task formulation and annotation schema. We demonstrate this difference (highlighted in **bold**) through illustrative examples for _Infect_ and _Symptom_ events in Table 7. As evident, keyword-based modeling requires annotating highly precise but disease-specific keywords like _COVID-19_, _fewer_, etc. On the other hand, our ED annotation formulation emphasizes the annotation of disease-agnostic triggers like _infected_, _symptoms_, etc. This provides SPEED and our framework superior generalizability without new annotation to unseen diseases.

## 6 Related Work

Event Extraction DatasetsEvent Extraction (EE) is the task of detecting events (Event Detection) and extracting structured information about specific roles linked to the event (Event Argument Extraction) from natural text. Earliest works for this task can be dated back to MUC Sundheim (1992); Grishman and Sundheim (1996) and the more standard ACE Doddington et al. (2004). Over the years, ACE was extended to various datasets like ERE Song et al. (2015) and TAC KBP Ellis et al. (2015). Recent progress has been the creation of massive datasets and huge event ontologies with datasets like MAVEN Wang et al. (2020), RAMS Ebner et al. (2020), WikiEvents Li et al. (2021), DocEE Tong et al. (2022), GENEVA Parekh et al. (2023) and GLEN Zhan et al. (2023). These ontologies and datasets cater to general-purpose events and do not comprise epidemiological event types.

Epidemiological OntologiesEarliest works Lindberg et al. (1993); Rector et al. (1996) defined highly rich taxonomies for describing technical concepts used by biomedical experts. Further developments led to the creation of SNOMED CT Stearns et al. (2001) and PHSkb Doyle et al. (2005) that define a list of reportable events used for communication between public health experts. BioCaster Collier et al. (2008) and PULS Du et al. (2011) extended ontologies for the news domain. Recent works of NCBI Dogan et al. (2014), IDO Babcock et al. (2021) and DO Schriml et al. (2022) focus on comprehensively organizing human diseases. In light of the recent COVID-19 pandemic, CIDO He et al. (2020) define a technical taxonomy for coronavirus, while ExcavatorCovid

\begin{table}
\begin{tabular}{p{227.6pt}} \hline The first known **case**[infect] of human to dog \#(monkey pox) **transmission**[spread] has been documented in medical journal, the Lancet. The dog **capabit**[infect] it from its owners, a non-monogamous gay couple in Paris. The greyround **developed**[symptom] an anal ulceration & mucoctaneous lesions. (url) \\ \hline (user) Release Topxx for **treatment**[\(curr\)] of monkey pox people are suffering \\ \hline (user) Wait what? I thought we were going to get Monkey Pox **vaccine**[prevent] in October. \\ \hline Monkey Pox is not an STI. It is not an STI. It is not an STI. **H**(monkey box is not an sti) It can be **transferred**[spread] via sex the same way Covid can. \\ \hline (user) Most of those people **have**[infect] Monkey Pox now. \\ \hline (user) More **dead**[death] in Chicago over a weekend than the world wide health emergency monkey pox \\ \hline \end{tabular}
\end{table}
Table 6: Sample illustrations of various tweets (anonymized) predicted as Monkeypox epidemic-related by our ED framework. The triggers are highlighted in **bold** with the respective event names in [].

Figure 6: Disease profiles of public opinions generated by plotting the percentage of extracted event mentions for COVID-19, Monkeypox and Zika.

 

(Min et al., 2021) automatically extract COVID-19 events and relations between them. Most of these ontologies are too fine-grained or limited to specific events, and can't be directly used for ED from social media, as also shown in Table 8.

Epidemiological Information ExtractionEarly works utilized search-engine queries and clickthrough rates for predicting influenza trends Eysenbach (2006); Ginsberg et al. (2009). Information extraction from Twitter has also been quite successful for predicting influenza trends Signorini et al. (2011); Lamb et al. (2013); Paul et al. (2014). Over the years, various biomedical monitoring systems have been developed like BioCaster Collier et al. (2008); Meng et al. (2022), HeathMap Freifeld et al. (2008), DANIEL Lejeune et al. (2015), EpiCore Olsen (2017). Extensions to support multilingual systems has also been explored Lejeune et al. (2015); Mutuvi et al. (2020); Sahnoun and Lejeune (2021). For the COVID-19 pandemic, several frameworks like CACT Lybarger et al. (2021) and COVIDKB Zong et al. (2022); Mendes et al. (2023) were developed for extracting symptoms, infection and treatment diagnosis. Most of these systems are disease-specific, focus on news and clinical domains, and use keyword/rule-based or simple BERT-based models, as shown in Table 8. In our work, we explore exploiting ED while focusing specifically on the social media domain.

## 7 Conclusion and Future Work

In this work, we develop an Event Detection (ED) framework to extract events from social media to provide early epidemic warnings. To facilitate this, we create our Twitter-based dataset SPEED comprising seven event types. Through experimentation, we show how existing models fail; while models trained on SPEED can effectively extract events and provide early warnings for unseen emerging epidemics. More broadly, our work demonstrates how event extraction can exploit social media to aid policy-making for better epidemic preparedness.

## Acknowledgements

We thank I-Hung Hsu, Jiao Sun, Alex Spangher, Yu Zhou, Hritik Bansal, Yufei Tian, Chujie Zheng, and Rohan Wadhawan for their valuable insights, paper reviews, and constructive comments. We thank the anonymous reviewers and the area editors across multiple rounds of reviews for their feedback. This work was partially supported by NSF 2106859, 2200274, AFOSR MURI via Grant FA9550- 22-1-0380, Defense Advanced Research Project Agency (DARPA) grant #HR00112290103/HR0011260656, and a Cisco Sponsored Research Award.

## Limitations

Our work focuses majorly on a single source of social media - Twitter. We haven't explored other social media platforms and how ED would work on those platforms in our work. We leave that for future work, but are optimistic that our models should be able to generalize across platforms. Secondly,

\begin{table}
\begin{tabular}{l c l l} \hline \hline  & **Disease** & **Infect Event Example** & **Symptom Event Example** \\ \hline Keyword-based & COVID-19 & Three students infected with **COVID-19** & COVID-19 symptoms include **fever**, **cough**, ... \\ Keyword-based & Monkeypox & How do you catch **Monkeybox?** & Monkeypox may cause **rashes** and **itching** ... \\ SPEED (Ours) & COVID-19 & Three students **infected** with COVID-19 & COVID-19 **symptoms** include fever, cough, ... \\ SPEED (Ours) & Monkeypox & How do you **catch** Monkexpox? & Monkeypox may **cause** rashes and itching ... \\ \hline \hline \end{tabular}
\end{table}
Table 7: Qualitative analysis for annotation difference between previous keyword-based epidemiological datasets Collier et al. (2008); Lejeune et al. (2015) and SPEED’s Event Detection based annotation schema. Our annotation schema is less disease-specific and thus, better generalizable to a wide range of diseases.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{3}{c}{**Sent-**} & \multicolumn{3}{c}{**Social Per.**} \\  & **Source Level** & **Trig.** & **Eve. Eye.** & **SWG** \\ \hline SPEED (Ours) & Twitter & ✓ & ✓ & ✓ & ✓ & ✓ \\ COVIDKB & Twitter & ✓ & ✗ & ✗ & ✓ & ✓ \\ CACT & Clinical & ✗ & ✓ & ✗ & \(\sim\) & ✓ \\ ExcavatorCovid News & ✗ & ✓ & ✓ & ✓ & ✗ \\ BioCaster & News & ✗ & ✗ & ✓ & ✓ & ✗ \\ DANIEL & News & ✗ & \(\sim\) & ✗ & \(\sim\) & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 8: Objective comparison of various epidemiological datasets COVIDRB Zong et al. (2022), CACT Lybarger et al. (2021), ExcavatorCovid Min et al. (2021), BioCaster Collier et al. (2008), and DANIEL Lejeune et al. (2015) with our dataset SPEED. We objectify the source of data (Data Source), the level of annotation granularity (Sentence Level), the presence of trigger information (Trigger Present), the presence of social and personal events (Social Events and Personal Events), and the suitability of ontology for social media (SMG – Social Media Granular). \(\sim\) indicates partial presence.

 

our work mainly only focuses on ED as the primary task, while its sister task Event Argument Extraction (EAE) is not explored. We hope to extend our work for EAE as part of our future work. Finally, we would like to show the generalization of our models on a vast range of diseases. However owing to budget constraints and the lack of publically available Twitter data for other diseases, we couldn't perform such a study. However, we believe showing results on three diseases lays the foundation for generalizability of our model.

## Ethical Considerations

One strong assumption in our work is the availability of internet and social media for discussions about epidemics. Since not everyone has equal access to these platforms, our dataset, models, and results do not represent the whole world uniformly. Thus, our work can be biased and should be considered with other sources for better representation.

Our dataset SPEED is based on actual tweets posted by people all over the world. We attempted our best to anonymize any kind of private information in the tweets, but we can never be completely thorough, and there might be some private information embedded still in our dataset. Furthermore, these tweets were sentimental and may possess stark emotional, racial, and political viewpoints and biases. We do not attempt to clean any of such extreme data in our work (as our focus was on ED only) and these biases should be considered if being used for other applications.

Since our ED models are trained on SPEED, they may possess some of the dataset-based social biases. Since we don't focus on bias mitigation, these models should be used with due consideration.

Lastly, we do not claim that our models can be used off-the-shelf for epidemic prediction as it hasn't been thoroughly tested and can have false positives and negatives too. Furthermore, our results are shown on academic datasets and do not utilize all possible Twitter data. We majorly throw light to show these model capabilities and motivate future work in this direction. The usage of these systems for practical purposes should be appropriately considered.

## References

* Ahn (2006) David Ahn. 2006. The stages of event extraction. In _Proceedings of the Workshop on Annotating and Reasoning about Time and Events_, pages 1-8, Sydney, Australia. Association for Computational Linguistics.
* Babcock et al. (2021) Shane Babcock, John Beverley, Lindsay G. Cowell, and Barry Smith. 2021. The infectious disease ontology in the age of COVID-19. _J. Biomed. Semant._, 12(1):13.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. _CoRR_, abs/2005.14165.
* Choi et al. (2022) YunSeok Choi, Hyojun Kim, and Jee-Hyong Lee. 2022. TABS: Efficient textual adversarial attack for pre-trained NL code model using semantic beam search. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5490-5498, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Collier et al. (2008) Nigel Collier, Son Doan, Ai Kawazoe, Reiko Matsuda Goodwin, Mike Conway, Yoshio Mateno, Hung Quoc Ngo, Dinh Dien, Asance Kawrtrakul, Koichi Takeuchi, Mika Shigematsu, and Kiyosu Taniguchi. 2008. Biocenter: detecting public health rumors with a web-based text mining system. _Bioinform._, 24(24):2940-2941.
* Deng et al. (2020)Shumin Deng, Ningyu Zhang, Jiaojian Kang, Yichi Zhang, Wei Zhang, and Huajun Chen. 2020. Meta-learning with dynamic-memory-based prototypical network for few-shot event detection. In _WSDM '20: The Thirteenth ACM International Conference on Web Search and Data Mining, Houston, TX, USA. February 3-7, 2020_, pages 151-159. ACM.
* Dias (2020) Guilherme Dias. 2020. Tweets dataset on Zika virus.
* tasks, data, and evaluation. In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)_, Lisbon, Portugal. European Language Resources Association (ELRA).
* Dogan et al. (2014) Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. NCBI disease corpus: A resource for disease name recognition and concept normalization. _J. Biomed. Informatics_, 47:1-10.
* Doyle et al. (2005) Timothy J. Doyle, Haobo Ma, Samuel L. Groseclose, and Richard S. Hopkins. 2005. Phskb: A knowledge-base to support notifiable disease surveillance. _BMC Medical Informatics__Dechs. Mak._, 5:27.

 

Mian Du, Peter von Etter, Mikhail Kopotev, Mikhail Novikov, Natalia Tarbeya, and Roman Yangarber. 2011. Building support tools for russian-language information extraction. In _Text, Speech and Dialogue - 14th International Conference, TSD 2011, Pilsen, Czech Republic, September 1-5, 2011. Proceedings_, volume 6836 of _Lecture Notes in Computer Science_, pages 380-387. Springer.
* Du and Cardie (2020) Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 671-683, Online. Association for Computational Linguistics.
* Ebner et al. (2020) Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence argument linking. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8057-8077, Online. Association for Computational Linguistics.
* Ellis et al. (2015) Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie M. Strassel. 2015. Overview of linguistic resources for the TAC KBP 2015 evaluations: Methodologies and results. In _Proceedings of the 2015 Text Analysis Conference, TAC 2015, Gaithersburg, Maryland, USA, November 16-17, 2015, 2015_. NIST.
* Eysenbach (2006) Gunther Eysenbach. 2006. Infodemiology: Tracking flu-related searches on the web for syndromic surveillance. In _AMIA 2006, American Medical Informatics Association Annual Symposium, Washington, DC, USA, November 11-15, 2006_. AMIA.
* Fleiss (1971) Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. _Psychological bulletin_, 76(5):378.
* Freifeld et al. (2008) Clark C. Freifeld, Kenneth D. Mandli, Ben Y. Reis, and John S. Brownstein. 2008. Model formulation: Healthmap: Global infectious disease monitoring through automated classification and visualization of internet media reports. _J. Am. Medical Informatics Assoc._, 15(2):150-157.
* Ginsberg et al. (2009) Jeremy Ginsberg, Matthew H Mohebbi, Rajan S Patel, Lynnette Brammer, Mark S Smolinski, and Larry Brilliant. 2009. Detecting influenza epidemics using search engine query data. _Nature_, 457(7232):1012-1014.
* Grishman and Sundheim (1996) Ralph Grishman and Beth Sundheim. 1996. Message Understanding Conference- 6: A brief history. In _COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics_.
* Guner et al. (2020) Hatice Rahmet Guner, Imran Hasanoglu, and Firdevs Aktas. 2020. Covid-19: Prevention and control measures in community. _Turkish Journal of medical sciences_, 50(9):571-577.
* He et al. (2015) Yongqun He, Hong Yu, Edison Ong, Yang Wang, Yingtong Liu, Anthony Huffman, Hsin-Hui Huang, John Beverley, Asiyah Yu Lin, William D. Duncan, Sivaram Arabani, Jiangan Xie, Junguk Hur, Xiaolin Yang, Luonan Chen, Gilbert S. Omen, Brian D. Athey, and Barry Smith. 2020. CIDO: the community-based coronavirus infections disease ontology. In _Proceedings of the 11th International Conference on Biomedical Ontologics (ICBO) joint with the 10th Workshop on Ontologies and Data in Life Sciences (ODLS) and part of the Bolzano Summer of Knowledge (BoSK 2020), Virtual conference hosted in Bolzano, Italy, September 17, 2020_, volume 2807 of _CEUR Workshop Proceedings_, pages 1-10. CEUR-WS.org.
* Heymann et al. (2001) David L Heymann, Guenael R Rodier, et al. 2001. Hot spots in a wired world: Who surveillance of emerging and re-emerging infectious diseases. _The Lancet infectious diseases_, 1(5):345-353.
* Hsu et al. (2022) I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. 2022. DEGREE: A data-efficient generation-based event extraction model. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1890-1908, Seattle, United States. Association for Computational Linguistics.
* Hsu et al. (2023) I-Hung Hsu, Kuan-Hao Huang, Shuning Zhang, Wenxin Cheng, Premkumar Natarajan, Kai-Wei Chang, and Nanyun Peng. 2023. A simple and unified tagging model with priming for relational structure predictions. In _Proceedings of the 61st Conference of the Association for Computational Linguistics_, Toronto, Canada. Association for Computational Linguistics.
* Huang et al. (2023) Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Premkumar Natarajan, Kai-Wei Chang, Nanyun Peng, and Heng Ji. 2023. A revealuation of event extraction: Past, present, and future challenges. _CoRR_, abs/2311.09562.
* Kamalrathne et al. (2023) Thushara Kamalrathne, Dilanth Amaratunga, Richard Haigh, and Lahiru Kodituwakku. 2023. Need for effective detection and early warnings for epidemic and pandemic preparedness planning in the context of multi-hazards: Lessons from the covid-19 pandemic. _International Journal of Disaster Risk Reduction_, 92:103724.
* Kmiec and Kirchhoff (2022) Dorota Kmiec and Frank Kirchhoff. 2022. Monkeypox: a new threat? _International journal of molecular sciences_, 23(14):7866.
* Kozlov et al. (2022) Max Kozlov et al. 2022. How deadly is monkeypox? what scientists know. _Nature_, 609(7928):663.
* Lamb et al. (2013) Alex Lamb, Michael J. Paul, and Mark Dredze. 2013. Separating fact from fear: Tracking flu infections on Twitter. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 789-795, Atlanta, Georgia. Association for Computational Linguistics 

Gael Lejeune, Romain Brixtel, Antoine Doucet, and Nadine Lucas. 2015. Multilingual event extraction for epidemic detection. _Artif. Intell. Medicine_, 65(2):131-143.
* Li et al. [2020] Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, and Shih-Fu Chang. 2020. Cross-media structured common space for multimedia event extraction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2557-2568, Online. Association for Computational Linguistics.
* Li et al. [2021] Sha Li, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 894-908, Online. Association for Computational Linguistics.
* Li et al. [2022] Sha Li, Heng Ji, and Jiawei Han. 2022. Open relation and event type discovery with type abstraction. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 6864-6877, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Lindberg et al. [1993] D. A. Lindberg, B. L. Humphreys, and A. T. McCray. 1993. The unified medical language system. _Methods of information in medicine_, 32(4):281---291.
* Lybarger et al. [2021] Kevin Lybarger, Mari Ostendorf, Matthew Thompson, and Melia Yetisgen. 2021. Extracting COVID-19 diagnoses and symptoms from clinical text: A new annotated corpus and neural event extraction framework. _J. Biomed. Informatics_, 117:103761.
* Lyu et al. [2021] Qing Lyu, Hongming Zhang, Elior Sulem, and Dan Roth. 2021. Zero-shot event extraction via transfer learning: Challenges and insights. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 322-332, Online. Association for Computational Linguistics.
* Mendes et al. [2023] Ethan Mendes, Yang Chen, Wei Xu, and Alan Ritter. 2023. Human-in-the-loop evaluation for early misinformation detection: A case study of COVID-19 treatments. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15817-15835, Toronto, Canada. Association for Computational Linguistics.
* Meng et al. [2022] Zaiqiao Meng, Anya Okhmatovskaia, Maxime Polleri, Yannan Shen, Guido Powell, Zihao Fu, Iris Ganser, Meiru Zhang, Nicholas B. King, David L. Buckeridge, and Nigel Collier. 2022. Biocaster in 2021: automatic disease outbreaks detection from global news media. _Bioinform._, 38(18):4446-4448.
* Min et al. [2021a] Bonan Min, Benjamin Rozonoyer, Haoling Qiu, Alexander Zamanian, Nianwen Xue, and Jessica MacBride. 2021a. ExcavatorCovid: Extracting events and relations from text corpora for temporal and causal analysis for COVID-19. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 63-71, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Min et al. [2021b] Bonn Min, Benjamin Rozonoyer, Haoling Qiu, Alexander Zamanian, Nianwen Xue, and Jessica MacBride. 2021b. Excavatorcovid: Extracting events and relations from text corpora for temporal and causal analysis for COVID-19. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 63-71, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Mutavi et al. [2020] Stephen Mutavi, Antoine Doucet, Gael Lejeune, and Moses Odeo. 2020. A dataset for multi-lingual epidemiological event extraction. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 4139-4144, Marseille, France. European Language Resources Association.
* Olsen [2017] Jennifer M Olsen. 2017. Epicore: crowdsourcing health professionals to verify disease outbreaks. _Online Journal of Public Health Informatics_, 9(1).
* OpenAI [2021] OpenAI. 2021. ChatGPT: Large-scale language model. Accessed: June 17, 2023.
* Veyseh et al. [2022] Amir P B Veyseh, Javid Ebrahim, Franck Demoncourt, and Thien Nguyen. 2022. MEE: A novel multilingual event extraction dataset. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9603-9613, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Paixao et al. [2022] Enny S Paixao, Luciana L Cardim, Maria CN Costa, Elizabeth B Brickley, Rita CO de Carvalho-Sauer, Eduardo H Carmo, Roberto FS Andrade, Moreno S Rodrigues, Rafael V veiga, Larissa C Costa, et al. 2022. Mortality from congenital zika syndrome--nationwide cohort study in brazil. _New England Journal of Medicine_, 386(8):757-767.
* Parekh et al. [2023] Tanmay Parekh, I-Hung Hsu, Kuan-Hao Huang, Kai-Wei Chang, and Nanyun Peng. 2023. Geneva: Benchmarking generalizability for event argument extraction with hundreds of event types and argument roles. In _Proceedings of the 61st Conference of the Association for Computational Linguistics_, Toronto, Canada. Association for Computational Linguistics.
* Paul et al. [2014] Michael J Paul, Mark Dredze, and David Broniatowski. 2014. Twitter improves influenza forecasting. _PLoS currents_, 6.
* Pota et al. [2021] Marco Pota, Mirko Ventura, Hamido Fujita, and Massimo Esposito. 2021. Multilingual evaluation of pre-processing for bert-based sentiment analysis of tweets. _Expert Syst. Appl._, 181:115119.
* Pyyssalo et al. [2021] Sampo Pyyssalo, Tomoko Ohta, Makoto Miwa, Hanscheol Cho, Junichi Tsujii, and Sophia Ananiadou.

 

2012. Event extraction across multiple levels of biological organization. _Bioinform._, 28(18):575-581.
* Rector et al. [1996] Alan L Rector, Jeremy E Rogers, and Pam Pole. 1996. The galen high level ontology. In _Medical Informatics Europe'96_, pages 174-178. IOS Press.
* Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Sahnoun and Lejeune [2021] Sihem Sahnoun and Gael Lejeune. 2021. Multilingual epidemic event extraction : From simple classification methods to open information extraction (OIE) and ontology. In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_, pages 1227-1233, Held Online. INCOMA Ltd.
* Schriml et al. [2022] Lynn M. Schriml, James B. Munro, Mike Schro, Dustin Olley, Carrie McCracken, Victor Felix, J Allen Baron, Rebecca C. Jackson, Susan M. Bello, Cynthia Bearer, Richard Lichenstein, Katharine Bisordi, Nicole Campion, Michelle G. Giglo, and Carol Greene. 2022. The human disease ontology 2022 update. _Nucleic Acids Res._, 50(D1):1255-1261.
* Shen et al. [2021] Jiaming Shen, Yunyi Zhang, Heng Ji, and Jiawei Han. 2021. Corpus-based open-domain event type induction. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5427-5440, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Signorini et al. [2011] Alessio Signorini, Alberto Maria Segre, and Philip M Polgreen. 2011. The use of twitter to track levels of disease activity and public concern in the us during the influenza a h1n1 pandemic. _PloS one_, 6(5):e19467.
* Song et al. [2015] Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. 2015. From light to rich ERE: Annotation of entities, relations, and events. In _Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation_, pages 89-98, Denver, Colorado. Association for Computational Linguistics.
* Stearns et al. [2001] Michael Q. Stearns, Colin Price, Kent A. Spackman, and Amy Y. Wang. 2001. SNOMED clinical terms: overview of the development process and project status. In _AMIA 2001, American Medical Informatics Association Annual Symposium, Washington, DC, USA, November 3-7, 2001_. AMIA.
* Sundheim [1992] Beth M. Sundheim. 1992. Overview of the fourth Message Understanding Evaluation and Conference. In _Fourth Message Understanding Conference (MUC-4): Proceedings of a Conference held in McLean, Virginia, June 16-18, 1992_.
* Thakur [2022] Nirmalya Thakur. 2022. Monkeypox2022tweets: A large-scale twitter dataset on the 2022 monkeyopuck task, findings from analysis of tweets, and open research questions. _Infectious Disease Reports_, 14(6):855-883.
* Tong et al. [2022] MeiHan Tong, Bin Xu, Shuai Wang, Meihuan Han, Yixin Cao, Jiangqi Zhu, Siyu Chen, Lei Hou, and Juanzi Li. 2022. DocEE: A large-scale and fine-grained benchmark for document-level event extraction. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3970-3982, Seattle, United States. Association for Computational Linguistics.
* Wadden et al. [2019] David Wadden, Ulme Winnberg, Yi Luan, and Hannaneh Hajishrizi. 2019. Entity, relation, and event extraction with contextualized span representations. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5784-5789, Hong Kong, China. Association for Computational Linguistics.
* Wang et al. [2020] Xiaozhi Wang, Ziqj Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzli Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A Massive General Domain Event Detection Dataset. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1652-1671, Online. Association for Computational Linguistics.
* Yao et al. [2021] Wenlin Yao, Xiaoman Pan, Lifeng Jin, Jianshu Chen, Dian Yu, and Dong Yu. 2021. Connect-the-dots: Bridging semantics between words and definitions via aligning word sense inventories. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7741-7751, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Zhan et al. [2023] Qiusi Zhan, Sha Li, Kathryn Conger, Martha Palmer, Heng Ji, and Jiawei Han. 2023. GLEN: general-purpose event detection for thousands of types. _CoRR_, abs/2303.09093.
* Zong et al. [2022] Shi Zong, Ashutosh Baheti, Wei Xu, and Alan Ritter. 2022. Extracting a knowledge base of COVID-19 events from social media. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 3810-3823, Gycongju, Republic of Korea. International Committee on Computational Linguistics 



## Appendix A Ontology Creation - Additional Details

### Complete ontology

Here, we first describe the selection steps for event types for our ontology as follows:

1. _Curation of event types_: We scan through existing medical ontologies like BCEO Collier et al. (2008), IDO Babcock et al. (2021), and the ExcavatorCovid Min et al. (2021) and curate a large list of event types for infectious and epidemic-related diseases.
2. _Merge event types across ontologies_: Since these existing ontologies may have repetitive event types, we perform a merging step. Specifically, two human experts manually examine and merge event types that are exactly similar in our curated list of event types.
3. _Filter out disease-specific event types_: Some event types in our curated list are specific to certain diseases. We identify and filter out such event types (e.g. Mask Wearing for COVID-19 which may not be observed for other diseases). We utilize opinions from public health experts to aid this step ensuring our event types are disease-agnostic.
4. _Definition Correction_: Utilizing aid from public health experts, we add and refine definitions for the curated set of event types and ensure they are disease-agnostic.
5. Organization - Following ExcavatorCovid Min et al. (2021), we organize our curated list of event types into three larger categories: social (events involving larger populations), personal (individual-oriented events), and medical (medically focused events) types.

Our complete initial event ontology comprises \(18\) event types along with their event definitions organized into three abstract categories as shown in Table 9.

### Initial analysis of events

Our initial ontology (SS A.1) was constructed using previous ontologies and human knowledge. But the relevance of each event type for social media (specifically Twitter) remains unknown. To evaluate this relevance, we first associate each event type with event-specific keywords. Then we utilize frequency and specificity as two guiding heuristics for further filtering/merging of event types in our curated ontology. We utilize the base Twitter dataset for SPEED for conducting this analysis. We describe each of these steps in more detail here:

Keyword AssociationIn order to objectively conduct this analysis, we associate each event type with a set of keywords.7 This association involves two simple steps:

Footnote 7: We release these keywords as part of our final code.

1. _Human expert curation_: For each event type, a human expert curates 2-3 simple yet specific keywords for each event based on common-sense knowledge. For example, for the _Curc_ event, the set of curated keywords were [curc, recovery].
2. _Thesaurus-based expansion_: For each human-expert curated list, we utilize an external resource - Thesaurus8 to further find event-relevant keywords. Human experts manually curate keywords from this thesaurus list such that the curated keyword is not generic (e.g. _display_ is filtered out for event _Symptom_ since it has other meanings as well). Footnote 8: [https://www.thesaurus.com/](https://www.thesaurus.com/)

We provide some example keywords for each event type along with some social media posts in which they appear in Table 10. We can also see how some example social media posts with the keywords can have different meanings as well. But we ignore them as this is a preliminary analysis.

Frequency-based filteringUsing frequency, we aim to filter out event types that are less mentioned in social media. To approximately estimate the frequency of each event type in social media, we count the number of social media posts containing any of the curated keywords for each event type. We show the keyword-count based frequency for each event type in Figure 7. We observe that most events under the medical abstraction occur much lesser than others. Furthermore, the variance in frequency is large as the most frequent event type _control_ is 180 times more likely to occur than the least frequent event type _variant_. Since such low-frequency events (e.g. _Variant_, _Cause_, _Prefigure_, etc.) are less likely to be mentioned in a smaller sample of data, we discard or merge such events for our final ontology.

Specificity-based filteringSpecificity ensures that each event type is uniquely identifiable with a

good confidence and mainly aims to reduce ambiguity and make the event types more distinct. To estimate specificity, for each curated keyword of an event type, we randomly sample a small number of non-duplicate social media posts. Human experts then manually evaluate the keyword specificity based on the percentage of posts wherein the semantic meaning of the keyword matches the definition of its event and is specific only to this event type. This specificity and distinctivity classifies keywords as high, medium, or low.

For example, the _Control_ event comprises high specificity keywords such as _quarantine_, _protocol_, _guidelines_; medium specificity keywords such as _restrict_, _postpone_, _investigate_; and low specificity keywords such as _battle_, _separation_, _limitation_. On the other hand, the event _Prefigure_ doesn't have any high specificity keywords, but only medium specificity keywords such as _foreshadow_ and low specificity keywords such as _forerell_.

Our analysis suggests that medium and low specificity keywords are more likely to give false positives relative to high specificity ones. Thus, we filter/merge event types that have a high number of low-confidence keywords (e.g. _Intrude_, _Promote_).

Final OntologyThus, with the above filtering and merging, we shrink our ontology from \(18\) event types to seven event types that are distinguishable, frequent, and have a low false-positive rate. We provide details about the action taken for each event type type with respect to the final ontology in Table 9.

### Association of Seed Tweets with Events

For event-based filtering, we associate each event type with a set of 5-10 seed tweets. In Table 11, we provide a couple of the seed tweets for each event type for reference. We also release these seed tweets as part of the main code.

\begin{table}
\begin{tabular}{l|l|c} \hline \hline
**Event name** & **Event Definition** & **Action for Final Ontology** \\ \hline \multicolumn{4}{c}{Social Scale Events} \\ \hline Prefigure Outbreak & The signal that precedes the occurrence of a potential epidemic. & Discarded \\  & The process of disease spreading among a certain amount of the population at a massive scale. & Merged into _Spread_ \\ Spread & The process of disease spreading among a certain amount of the population but at a local scale. & Final Event \\ Control & Collective efforts trying to impede the spread of a epidemic. & Final Event \\ Promote & The relationship of a disease driver leading to the breakout of a disease. & Discarded \\ \hline \multicolumn{4}{c}{Personal Scale Events} \\ \hline Prevent Infection Symptom & Individuals trying to prevent the infection of disease. & Final Event \\  & The process of a disease/pathogen invading host(s). & Final Event \\  & Individuals displaying physiological features indicating the abnormality of organisms. & Final Event \\ Treatment & The process that a patient is going through with the aim of recovering from symptoms. & Merged into _Cure_ \\ Cure & Stopping infection and relieving individuals from infections/symptoms. & Final Event \\ Immunize & The process by which an organism gains immunization against an infectious agent. & Merged into _Prevent_ \\ Death & End of life of individuals due to infectious disease. & Final Event \\ \hline \multicolumn{4}{c}{Medical Scale Events} \\ \hline Cause Variant Intrued Respond & The causal relationship of a pathogen and a disease. & Discarded \\  & An alternation of a disease with genetic code-carrying mutations. & Discarded \\  & The process of an infectious agent intruding on its host. & Merged into _Infetect_ \\  & The process of a host responding to an infection. & Discarded \\  & The process of suppressing and slowing down the infection of a virus. & Merged into _Cure_ \\ Transmission route & The process of a pathogen entering another host from a source. & Discarded \\ \hline \hline \end{tabular}
\end{table}
Table 9: Complete initial epidemic event ontology comprising \(18\) event types along with their event definitions organized into \(3\) higher-level abstract categories. We also present details about the action taken for each event type in the final ontology.

 



### Coverage analysis of ontology

To quantitatively verify the coverage of our ontology, we conduct an analysis on four diseases with very different characteristics - COVID-19, Monkeypox, Dengue, and Zika. For each disease, we randomly sample \(300\) tweets and then filter them if they are related to the disease or not. Next, we annotate the filtered disease-related tweets based on our ontology and evaluate the proportion of event occurrences relative to the number of disease-related tweets. We find that our ontology has high coverage of \(50\%\) for COVID-19, \(44\%\) for Monkeypox, \(70\%\) for Dengue, and \(73\%\) for Zika. This in turn assures that our ontology can be used to detect epidemic events for various different kinds of diseases.

Event Type DistributionAs part of our analysis, we also study our ontology's event type distribution for each disease and its correlation with the disease properties and outbreak stage. We show this event distribution in Figure 8 for each of the diseases. We note that distributions for Dengue and Monkeypox exhibit a strong focus on _spread_ and _infect_ events. This makes sense as the data for these diseases was collected at earlier stages of the outbreak when mitigation measures were not being discussed yet. On the other hand, for COVID-19, the distribution is vastly dominated by _control_ and _death_ events. Our

\begin{table}
\begin{tabular}{p{62.6pt}|p{113.8pt}|p{113.8pt}} \hline \hline
* Business Insider (url) \\ \hline Outbreak & epidemic, pandemic, outbreak, ... & (user) If Boris goes, who is Nicola going to follow in the coronavirus **epidemic?** \\ \hline Spread & outbreak, spread, contagious, ... & COVID-19 **outbreak** detected in immigration detention centres ... \\ \hline Control & quarantine, isolate, restriction, protocol, ... & The doctors in my family are losing their minds seeing these videos of people flagrantly violating social distancing **protocols**. \\ \hline Promote & boost, lead, accelerate & (user) Covid-19 Will **Accelerate** the AI Health Care Revolution (url) \\ \hline  & & Personal Scale Events \\ \hline Prevent & prevent, protect, avoid, vaccine, ... & (user) why can’t they wear masks and god will cover them anyways then you have double the **protection** \\ \hline Infect & infect, spread, outbreak, infection, ... & (user) wars of COVID-19 **spread** in group gatherings after 2 people infect dozens at church. (url) \\ \hline Symptom & symptom, fever, ill, sick, cache, ... & ... average person dying from coronavirus has serious **illness** and is about to die anyway. \\ \hline Treatment & doctor, drug, hospitalize, nurse, ... & Hyperbaric oxygen **therapy** can help fight one of the major issues(lack of oxygen) caused by covid/corona virus? \\ \hline Cure & recover, cure, heal, ... & Patients who’ve **recovered** from Covid-19 and test positive again aren’t contagious, a study says (url) \\ \hline Immunize & immunize, antibody, vaccine, ... & We need to hope the number of cases go up in order to Herd **immunize**, and continually watch the # of hospitalizations. \\ \hline Death & death, decease, kill, ... & Highest **death** toll in Europe. Economy on the floor. Quarantine for international travellers months after the horse bolted. \\ \hline  & & Medical Scale Events \\ \hline Cause & cause, lead & Once again, 1-2 infections **lead** to a large number of new infections. \\ \hline Variant & lineage, variant, mutate & \#DeltaVariant surging in U.S. New data show Delta much more contagious than previous **versions** of \#COVID19. \\ \hline Intrude & attack, intrude, harm & (user) Peru found 100\% cure zero **harm** ivermectin. Pinned tweet medical paper. Plagues over. Retweet. \\ \hline Respond & respond, react & Aid agencies face huge challenges **responding** to \#COVID19 in countries affected by conflict ... \\ \hline Regulate & regulate, limit, decelerate, slow, ... & How can you **regulate** a child in school after lockdown? \\ \hline Transmission route & airborne, contact, air, water, ... & (user) Very unfortunate. You can’t workout with a mask on, is Corona **airborne**? Just asking.. \\ \hline \hline \end{tabular}
\end{table}
Table 10: All the events in our ontology along with their associated keywords and example social media posts.


COVID-19 data was collected in May 2020 when the outbreak had vastly spread in America. Thus our distribution reflects more notions of lockdowns and control measures as well reflects the deadly nature of the disease.

## Appendix B Uniform Sampling v/s Random Sampling for Data Selection

Previously Parekh et al. (2023) had shown how uniform sampling of data for events can yield more robust model performance. To validate the same for our ontology and data, we conduct additional experiments comparing uniform sampling with random sampling. More specifically, we annotate \(200\) tweets that conform to a 'real distribution'9 based on random sampling and compare the trained models on this data with models trained on \(200\) tweets of uniform-sampling data. We further annotated \(300\) tweets based on the 'real-distribution' which was used for the evaluation of these two sampling techniques.

Footnote 9: Event-based filtering was still applied before sampling.

We present our results in Table 12 averaged over three model runs. We show that in terms of best model performance, uniform sampling is better by \(5.5\) F1 points compared to random sampling. On average, uniform-sampling trained models outperform the random-sampling trained models by up to \(11\) points. Both these results prove how despite train-test distribution differences, uniform sampling leads to better training of downstream models.

Generalizability to Other DiseasesWe also evaluate the models trained on the uniform and random-sampled data for generalizability to other diseases of Monkeypox, Zika, and Dengue. We show the results in Table 13. Clearly, we can see superior generalizability of uniform-sampling trained models as they outperform random-sampling trained models by \(37\) F1 points for Monkeypox and \(28\) F1 points for Zika + Dengue. Overall, this result strongly highlights the impact of uniform sampling

Figure 8: Event type distribution of the disease-related tweets for each disease. Numbers on the axis represent count of mentions for a given event type.

Figure 7: Frequency of occurrence based on keyword search for all event types in the initial complete ontology.

 

for robust and generalizable model training.

## Appendix C Annotation Guidelines and Interface

### Annotation Guidelines

Inspired by Doddington et al. (2004), we develop an extensive set of instructions with tricky cases and examples that have been developed through multiple rounds of expert annotation studies. We present the task summary with the major instructions in Figure 14. To reduce ambiguity in trigger selection, we present extensive examples and tricky cases with priority orders as shown in Figure 15. Finally, we also provide a wide range of annotated positive and negative examples as part of the guidelines and show those in Figure 16.

### Annotation Interface

We utilize Amazon Mechanical Turk10 as the interface for quick annotation. To annotate, annotators can select any word and label it into one of the seven pre-defined event types. Event definitions and examples are provided alongside for reference. Each batch (also known as HIT) comprises five tweets for flexibility in annotations. We show the interface and various utilities in Figure 17, 18, and 19 respectively.

Footnote 10: [https://www.mturk.com/](https://www.mturk.com/) 



### Event Distribution Analysis

As part of data processing, we attempt to sample tweets in a more uniform distribution between the event types (SS 3.2). In Figure 9, we show the distribution of our dataset in terms of event types. In contrast to tail-ending distributions of other standard datasets like ACE Doddington et al. (2004) and MAVEN Wang et al. (2020) as shown in Figures 10 and 11 respectively, our distribution of event mentions is more uniform.

### Monkeypox Test Data Statistics

We share the data statistics of the evaluation dataset used for Monkeypox in Table 15 split according to each event type. We observe that there is a disparity in distribution across different event types, with _spread_ mostly discussed and _cure_ and _death_ are least discussed.

### Zika + Dengue Test Data Statistics

We share the data statistics of the evaluation dataset used for Zika + Dengue in Table 16 split according to each event type. We observe a more even distribution of event types with more focus on _infect_, _spread_, and _death_ well-discussed.

## Appendix E ED models and Implementation Details

We present details about each ED model that we benchmark along with the extensive set of hyperparameters and oth

\begin{table}
\begin{tabular}{l r} \hline \hline
**Event Type** & **\# Event Mentions** \\ \hline infect & \(57\) \\ spread & \(53\) \\ symptom & \(34\) \\ prevent & \(22\) \\ control & \(28\) \\ cure & \(20\) \\ death & \(60\) \\ \hline
**Total** & **274** \\ \hline \hline \end{tabular}
\end{table}
Table 16: Data Statistics for the evaluation dataset for Zika+Dengue Event Detection categorized by event types.

Figure 11: Distribution of event mentions for the event types in the MAVEN dataset.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Dataset** & **Partial** & **Exact** \\  & **Match** & **Match** \\ \hline ACE Doddington et al. (2004) & \(14\%\) & \(0\%\) \\ ERE Song et al. (2015) & \(14\%\) & \(0\%\) \\ MAVEN Wang et al. (2020) & \(42\%\) & \(0\%\) \\ MEE P Byvesh et al. (2022) & \(14\%\) & \(0\%\) \\ M\({}^{2}\)E Li et al. (2020) & \(14\%\) & \(0\%\) \\ MLEE Pyysalo et al. (2012) & \(0\%\) & \(0\%\) \\ FewEvent Deng et al. (2020) & \(28\%\) & \(0\%\) \\ \hline \hline \end{tabular}
\end{table}
Table 14: Comparison of SPEED with ACE and MAVEN in terms of unique trigger words and average number of triggers per event mention. Avg = Average.

Figure 10: Distribution of event mentions for the event types in the ACE dataset.

Figure 9: Distribution of event mentions per event type for our dataset SPEED.





### Te

TE (Lyu et al., 2021) is a pre-trained model that formulates ED as a textual entailment and question-answering task. We run our experiments for TE on an NVIDIA 1080Ti machine with support for 8 GPUs. Our hyperparameters are as listed in the original paper.

### Wsd

WSD (Yao et al., 2021) is a classification model using on the joint encoding of the contextualized trigger and event definitions. We run our experiments for WSD on an NVIDIA A100 machine with support for 8 GPUs. The major hyperparameters for this model are listed in Table 17.

### Tabs

Tabs(Li et al., 2022) is an event type induction model, wherein the goal is to discover new event types without a pre-defined event ontology. It utilizes two complementary trigger embedding spaces (mask view and token view) for classification. To adapt this for ED, we follow the end-to-end event discovery setting in (Choi et al., 2022) while making the following modifications: (1) **Dataset Composition**: We utilize ACE (Doddington et al., 2004) dataset for training and development and our SPEED dataset for testing. Our training data comprises 26 known event types from ACE, the validation set comprises 7 ACE event types, while our test set comprises 7 event types from SPEED. (2) **Candidate Trigger Extraction**: To improve trigger coverage, we extract all nouns and non-auxiliary verbs as candidate trigger mentions. (3) **Evaluation Setup**: Trigger identification (**Tri-I**) F1 score is evaluated using the extracted candidate triggers. For trigger classification (**Tri-C**), we first find the best cluster assignment of the predicted event clusters to the gold event types and then evaluate the F1 score.

We run our experiments for Tabs on an NVIDIA RTX 2080 Ti machine with support for 8 GPUs. The major hyperparameters for this model are listed in Table 18.

### ETypeClus

ETypeClus (Shen et al., 2021) extracts salient predicate-object pairs and clusters their embeddings in a spherical latent space. For consistency across our evaluations, we follow the re-implementation of the ETypeClus model in (Choi et al., 2022), which consists of the latent space clustering stage of the ETypeClus pipeline and uses the embeddings of trigger mentions to be the input features. We utilize the contextualized embeddings of the candidate triggers extracted from SPEED for unsupervised training. The candidate trigger extraction process and the evaluation setup are the same as described in SS E.3.

We run our experiments for ETypeClus on an NVIDIA RTX 2080 Ti machine with support for 8 GPUs. The major hyperparameters for this model are listed in Table 19.

### Bert-Qa

BERT-QA (Du and Cardie, 2020) is a classification model utilizing label semantics by formulating event detection as a question-answering task. We run our experiments for BERT-QA on an NVIDIA RTX A6000 machine with support for 8 GPUs. The major hyperparameters for this model are listed in Table 20.

\begin{table}
\begin{tabular}{l r} \hline \hline Pre-trained LM & \multicolumn{1}{c}{RoBERTa-Large} \\ Training Batch Size & \(64\) \\ Eval Batch Size & \(8\) \\ Learning Rate & \(0.00001\) \\ Weight Decay & \(0.01\) \\ \# Training Epochs & \(7\) \\ Max Sentence Length & \(128\) \\ Max gradient norm & \(1\) \\ \hline \hline \end{tabular}
\end{table}
Table 17: Hyperparameter details for WSD model.

\begin{table}
\begin{tabular}{l r} \hline \hline Pre-trained LM & \multicolumn{1}{c}{BERT-Base} \\ Training Batch Size & \(64\) \\ Eval Batch Size & \(8\) \\ Gradient Accumulation Steps & \(2\) \\ Learning Rate & \(0.00005\) \\ Gradient Clipping & \(1\) \\ \# Pretrain Epochs & \(10\) \\ \# Training Epochs & \(30\) \\ Consistency Loss Weight & \(0.2\) \\ \# Target Unknown Event Types & \(30\) \\ \hline \hline \end{tabular}
\end{table}
Table 18: Hyperparameter details for TABS model.





### Degree

DEGREE Hsu et al. (2022) is a generation-based model prompting using natural language templates. We run our experiments for DEGREE on an NVIDIA RTX A6000 machine with support for 8 GPUs. The major hyperparameters for this model are listed in Table 21.

### TagPrime

TagPrime Hsu et al. (2023) is a sequence tagger priming words to input text to convey more task-specific information. We run our experiments for TagPrime on an NVIDIA RTX A6000 machine with support for 8 GPUs. The major hyperparameters for this model are listed in Table 22.

### DyGIE++

DyGIE++ Wadden et al. (2019) is a multi-task classification-based model utilizing local and global context via span graph propagation. We run our experiments for DyGIE++ on an NVIDIA RTX A6000 machine with support for 8 GPUs. The major hyperparameters for this model are listed in Table 23.

### Keyword

This baseline model basically curates a list of keywords specific to each event and predicts a trigger for a particular event if it matches one of the curated event keywords. Event keywords are curated by expert annotators based on the gold triggers appearing in the SPEED dataset and classified as high confidence, medium confidence, and low confidence based on their occurrence counts and false positive rates (as described in SS A.2.11 Although this baseline accesses gold test data, it is meant to be a baseline to provide the upper cap for models of this family.

Footnote 11: We will release the set of keywords with our final code.

### Gpt-3

We use the GPT-3.5 turbo model as the base GPT model. We experiment with ChatGPT OpenAI (OpenAI, 2021) for tuning our prompts that ensure output consistency. Our final prompt (as shown in Figure 12) comprises a task definition, ontology details, 1 example for each event type, and the final test query. We conducted a looser evaluation for GPT and only match if the predicted trigger text matches the gold trigger text (we didn't check the exact span match basically).

\begin{table}
\begin{tabular}{l r} \hline Pre-trained LM & RoBERTa-Large \\ Training Batch Size & \(64\) \\ Learning Rate & \(0.001\) \\ Weight Decay & \(0.001\) \\ Gradient Clipping & \(5\) \\ Training Epochs & \(5\) \\ Max Sequence Length & \(175\) \\ Linear Layer Dropout & \(0.2\) \\ \hline \end{tabular}
\end{table}
Table 22: Hyperparameter details for TagPrime model.

\begin{table}
\begin{tabular}{l r} \hline Pre-trained LM & BoBERTa-Large \\ Training Batch Size & \(64\) \\ Eval Batch Size & \(8\) \\ Learning Rate & \(0.001\) \\ Weight Decay & \(0.001\) \\ Gradient Clipping & \(5\) \\ Training Epochs & \(100\) \\ Warmup Epochs & \(5\) \\ Maximum Sequence Length & \(175\) \\ Linear Layer Dropout & \(0.2\) \\ \hline \end{tabular}
\end{table}
Table 20: Hyperparameter details for BERT_QA model.

\begin{table}
\begin{tabular}{l r} \hline Pre-trained LM & RoBERTa-Large \\ Training Batch Size & \(32\) \\ Learning Rate & \(0.0001\) \\ Weight Decay & \(0.00001\) \\ Gradient Clipping & \(5\) \\ Warmup Epochs & \(5\) \\ Max Sequence Length & \(175\) \\ Linear Layer Dropout & \(0.2\) \\ \hline \end{tabular}
\end{table}
Table 21: Hyperparameter details for DEGREE model.

\begin{table}
\begin{tabular}{l r} \hline Pre-trained LM & RoBERTa-Large \\ Training Batch Size & \(64\) \\ Eval Batch Size & \(8\) \\ Learning Rate & \(0.001\) \\ Weight Decay & \(0.001\) \\ Gradient Clipping & \(5\) \\ Training Epochs & \(100\) \\ Warmup Epochs & \(5\) \\ Maz Sequence Length & \(15\) \\ Maz Sequence Length & \(0.2\) \\ \hline \end{tabular}
\end{table}
Table 22: Hyperparameter details for TagPrime model.





## Appendix F Predicting Early Warnings for Monkeypox

### Event-wise Analysis

As BERT-QA yields the strongest early warning signal (shown in Figure 5), we conduct an analysis at a more granular level on the contribution of each event type to the early warning signal based on the trained BERT-QA output. We present the results in Figure 13, which leads to the following observations: (1) **Strength of indication varies among event types**: As indicated in Figure 13, event type _infect_ and _spread_ are strong indicators of the incoming surge in reported cases, while event type _prevent_ and _control_ can serve as indicators of medium strength. Event type _symptom_, _cure_, and _death_ are weak indicators that barely contribute to the early warning signal. (2) **Distribution across event types can potentially reveal high-level disease characteristics**: We can infer some properties of diseases based on the frequency of mentions about particular events. For example, _death_ is less mentioned, which can indicate that _Monkeypox_ is less fatal compared to other epidemics like COVID. We would like to mention that these are hypothetical properties based on predictions of our best model (which can be imperfect) and should be taken with a pinch of salt.

Figure 12: Illustration of the prompt used for GPT-3 model. It includes a task description, followed by ontology details of event types and their definitions. Next, we show some in-context examples for each event type and finally, provide the test sentence.

Figure 13: Number of reported Monkeypox cases and the number of extracted events for each SPEED event type from our trained BERT-QA model from \(XX\) to \(XX\)

An Event is defined as something happens in a sentence. In this task, we are trying to identity whether one or more of the following events exist in a given string: _infect, spread, symptom,prevent,control, cure, and death_. And if an event exist, what is the major triggering word that mostly manifest its occurrence.

Figure 14: Task summary and the major annotation guidelines.

 

Here are more detailed instructions for how to choose the most appropriate triggering word.

Goat: Look for the one word that MOST LIKELY manifests the event's occurrence. You can use the following priority order for annotation:

1. Most of the times, the trigger of the event will be the main verb in the sentence.

2. If the verb is ambiguous/vague, the trigger would be a noun semantically related to the event.

3. (Rare case) If no such noun exist, the trigger would be any adjective/adverb that is realated to the event.

4. If still confused, use your best judgement to select the trigger.

In the following illustrations, correct trigger words are marked blue.

Figure 15: Guidelines to choose the proper triggering word.

 

* [16] M. C. Bader, A. K. D'Auria, and R. J. S. Lees, "Theoretical performance of the wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based wild-based-wild-based wild-based wild-based-wild-based wild-based-wild-based wild-based-wild-based wild-based-wild-based wild-based-wild-based wild-based-wild-based wild-based-wild-based wild-based-wild-based wild-based-wild-based wild-based-wild-based-wild-based wild 

Figure 19: Illustration of the format and options available for a completed annotation in the interface.

Figure 18: Illustration of selection of a word within a tweet for annotation in the interface.