Venue: NAACL 2022

# DEGREE: A Data-Efficient Generation-Based Event Extraction Model

I-Hung Hsu\({}^{*\dagger}\)  Kuan-Hao Huang1 Elizabeth Boschee2 Scott Miller3 Premkumar Natarajan4 Kai-Wei Chang5 Nanyun Peng67 Information Science Institute, University of Southern California

Footnote 1: The authors contribute equally.

Footnote 2: footnotemark:

###### Abstract

Event extraction requires high-quality expert human annotations, which are usually expensive. Therefore, learning a _data-efficient_ event extraction model that can be trained with _only a few_ labeled examples has become a crucial challenge. In this paper, we focus on _low-resource end-to-end_ event extraction and propose Degree, a data-efficient model that formulates event extraction as a conditional generation problem. Given a passage and a manually designed prompt, Degree learns to summarize the events mentioned in the passage into a natural sentence that follows a predefined pattern. The final event predictions are then extracted from the generated sentence with a deterministic algorithm. Degree has three advantages to learn well with less training data. First, our designed prompts provide semantic guidance for Degree to leverage _label semantics_ and thus better capture the event arguments. Moreover, Degree is capable of using additional _weakly-supervised_ information, such as the description of events encoded in the prompts. Finally, Degree learns triggers and arguments jointly in an _end-to-end manner_, which encourages the model to better utilize the shared knowledge and dependencies among them. Our experimental results demonstrate the strong performance of Degree for low-resource event extraction.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Event extraction (EE) aims to extract events, each of which consists of a trigger and several participants (arguments) with their specific roles, from a given passage. For example, in Figure 1, a _Justice:Execute_ event is triggered by the word "_execution__"_ and this event contains three argument roles, including an _Agent_ (_Indonesia_) who carries out the execution, a _Person_ who is executed (_convicts_), and a _Place_ where the event occurs (not mentioned in the passage). Previous work Yang et al. (2019a); Fincke et al. (2021) usually divides EE into two subtasks: (1) **event detection**, which identifies event triggers and their types, and (2) **event argument extraction**, which extracts the arguments and their roles for given event triggers. EE has been shown to benefit a wide range of applications, e.g., building knowledge graphs Zhang et al. (2020), question answering Berant et al. (2014); Han et al. (2021), and other downstream studies Han et al. (2019a); Hogenboom et al. (2016); Sun and Peng (2021).

Most prior works on EE rely on a large amount of annotated data for training Nguyen and Grishman (2015); Nguyen et al. (2016); Han et al. (2019b); Du and Cardie (2020); Huang et al. (2020); Huang and Peng (2021); Paolini et al. (2021). However, high-quality event annotations are expensive to obtain. For example, the ACE 2005 corpus Doddington et al. (2004), one of the most widely used EE datasets, requires two rounds of annotations by linguistics experts. The high annotation costs make these models hard to be extended to new domains and new event types. Therefore, how to learn a _data-efficient_ EE model trained with _only a few_ annotated examples is a crucial challenge.

In this paper, we focus on _low-resource_ event extraction, where only a small amount of training examples are available for training. We propose Degree (**D**ata-**E**fficient **G**ene**Ration-Based **E**vent **E**xtraction), a generation-based model that takes a passage and a manually designed prompt as the input, and learns to summarize the passage into a

Figure 1: Two examples of events (_Justice:Execute_ and _Justice:Appeal_) extracted from the given passage.



natural sentence following a predefined template, as illustrated in Figure 2. The event triggers and arguments can then be extracted from the generated sentence by using a deterministic algorithm.

Degree enjoys the following advantages to learn well with less training data. First, the framework provides _label semantics_ via the designed template in the prompts. As the example in Figure 2 shows, the word _"somewhere"_ in the prompt guides the model to predict words being similar to _location_ for the role _Place_. In addition, the sentence structure of the template and the word _"attacked"_ depict the semantic relation between the role _Attacker_ and the role _Target_. With these kinds of guidance, Degree can make more accurate predictions with less training examples. Second, the prompts can incorporate additional weak-supervision signal about the task, such as the description of the event and similar keywords. These resources are usually readily available. For example, in our experiments, we take the information from the annotation guideline, which is provided along with the dataset. This information facilitates Degree to learn under a low-resource situation. Finally, Degree is designed for end-to-end event extraction and can solve event detection and event argument extraction at the same time. Leveraging the shared knowledge and dependencies between the two tasks makes our model more data-efficient.

Existing works on EE usually have only one or two of above-mentioned advantages. For example, previous classification-based models Nguyen et al. (2016); Wang et al. (2019); Yang et al. (2019); Wadden et al. (2019); Lin et al. (2020) can hardly encode label semantics and other weak supervision signals. Recently proposed generation-based models for event extraction solved the problem in a pipeline fashion; therefore, they cannot leverage shared knowledge between subtasks Paolini et al. (2021); Li et al. (2021). Furthermore, their generated outputs are not natural sentences, which hinders the utilization of label semantics Paolini et al. (2021); Lu et al. (2021). As a result, our model Degree can achieve significantly better performance than prior approaches on low-resource event extraction, as we will demonstrate in Section 3.

Our contributions can be summarized as follows:

* We propose Degree, a generation-based event extraction model that learns well with less data by better incorporating label semantics and shared knowledge between subtasks (Section 2).
* Experiments on ACE 2005 Doddington et al. (2004) and ERE-EN Song et al. (2015) demonstrate the strong performance of Degree in the low-resource setting (Section 3).
* We present comprehensive ablation studies in both the low-resource and the high-resource setting to better understand the strengths and weaknesses of our model (Section 4).

Our code and models can be found at [https://github.com/PlusLabNLP/DEGREE](https://github.com/PlusLabNLP/DEGREE).

Figure 2: An illustration of Degree for predicting a _Contact:Attack_ event. The input of Degree consists of the given passage and our design prompt that contains an event type description, event keywords, and a E2E template. Degree is trained to generate an output to fill in the placeholders (underlined words) in the E2E template with triggers and arguments. The final event prediction is then decoded from the generated output.

 

Data-Efficient Event Extraction

We introduce Degree, a generation-based model for low-resource event extraction. Unlike previous works (Yang et al., 2019; Li et al., 2021), which separate event extraction into two pipelined tasks (event detection and event argument extraction), Degree is designed for the end-to-end event extraction and predict event triggers and arguments at the same time.

### The Degree Model

We formulate event extraction as a conditional generation problem. As illustrated in Figure 2, given a passage and our designed prompt, Degree generates an output following a particular format. The final predictions of event triggers and argument roles can be then parsed from the generated output with a deterministic algorithm. Compared to previous classification-based models (Wang et al., 2019; Yang et al., 2019; Wadden et al., 2019; Lin et al., 2020), the generation framework provides a flexible way to include additional information and guidance. By designing appropriate prompts, we encourage Degree to better capture the dependencies between entities and, therefore, to reduce the number of training examples needed.

The desired prompt not only provides information but also defines the output format. As shown in Figure 2, it contains the following components:

* **Event type definition** describes the definition for the given event type.1 For example, _"The event is related to conflict and some violent physical act."_ describes a _Conflict:Attack_ event. Footnote 1: The definition can be derived from the annotation guidelines, which are provided along with the datasets.
* **Event keywords** presents some words that are semantically related to the given event type. For example, _war_, _attack_, and _terrorism_ are three event keywords for the _Conflict:Attack_ event. In practice, we collect three words that appear as the triggers in the example sentences from the annotation guidelines.
* **E2E template** defines the expected output format and can be separated into two parts. The first part is called ED template, which is designed as _"Event trigger is \(<\)Trigger\(>\)"_, where \(``{<}\)Trigger\(>\)"_ is a special token serving as a placeholder. The second part is the EAE template, which differs based on the given event type. For example, in Figure 2, the EAE template for a _Conflict:Attack_ event is _"some attacker attacked some facility someone or some organization by some way in somewhere"_. Each underlined string starting with _"some-"_ serves as a placeholder corresponding to an argument role for a _Conflict:Attack_ event. For instance, _"some way"_ corresponds to the role _Instrument_ and _"somewhere"_ corresponds to the role _Place_. Notice that every event type has its own EAE template. We list three EAE templates in Table 1. The full list of EAE templates and the construction details can be found in Appendix A.

### Training

The training objective of Degree is to generate an output that replaces the placeholders in E2E template with the gold labels. Take Figure 2 as an example, Degree is expected to replace \(``{<}\)Trigger\(>\)" with the gold trigger (_detonated_), replace _"some attacker"_ with the gold argument for role _Attacker_ (_Palestinian_), and replace _"some way"_ with the gold argument for role _Instrument_ (_bomb_). If there are multiple arguments for the same role, they are concatenated with _"and"_; if there is no predicted argument for one role, the model should keep the corresponding placeholder (i.e, _"some-"_ in the E2E template). For the case that there are multiple triggers for the given event type in the input passage, Degree is trained to generate the output text that contains multiple E2E template such that each E2E template corresponds to one trigger and its argument roles. The hyperparameter settings are detailed in Appendix B.

### Inference

We enumerate all event types and generate an output for each event type. After we obtain the generated sentences, we compare the outputs with E2E template to determine the predicted triggers and arguments in string format. Finally, we apply string matching to convert the predicted string to span offsets in the passage. If the predicted string appears in the passage multiple times, we choose all span offsets that match for trigger predictions and choose the one closest to the given trigger span for argument predictions.

### Discussion

Notice that the E2E template plays an important role for Degree. First, it serves as the control signal and defines the expected output format. Second, 

it provides label semantics to help Degree make accurate predictions. Those placeholders (words starting with _"some-"_) in the E2E template give Degree some hints about the entity types of arguments. For instance, when seeing _" somewhere"_, Degree tends to generate a location rather than a person. In addition, the words other than _"some-"_ describe the relationships between roles. For example, Degree knows the relationship between the role _Attacker_ and the role _Target_ (who is attacking and who is attacked) due to E2E template. This guidance helps Degree learn the dependencies between entities.

Unlike previous generation-based approaches Paolini et al. (2021); Li et al. (2020); Huang et al. (2021), we intentionally write E2E templates in natural sentences. This not only uses label semantics better but also makes the model easier to leverage the knowledge from the pre-trained decoder. In Section 4, we will provide experiments to demonstrate the advantage of using natural sentences.

Cost of template constructingDegree does require human effort to design the templates; however, writing those templates is much easier and more effortless than collecting complicated event annotations. As shown in Table 1, we keep the EAE templates as simple and short as possible. Therefore, it takes only about one minute for people who are not linguistic experts to compose a template. In fact, several prior works Liu et al. (2020); Du and Cardie (2020); Li et al. (2020) also use constructed templates as weakly-supervised signals to improve models. In Section 4, we will study how different templates affect the performance.

Efficiency Considerations.Degree requires to enumerate all event types during inference, which could cause efficiency considerations when extending to applications that contain many event types. This issue is minor for our experiments on the two datasets (ACE 2005 and ERE-EN), which are relatively small scales in terms of the number of event types. Due to the high cost of annotations, there is hardly any public datasets for end-to-end event extraction on a large scale,2 and we cannot provide a more thorough studies when the experiments scale up. We leave the work on benchmarking and improving the efficiency of Degree in the scenario considering more diverse and comprehensive types of events as future work.

Footnote 2: To the best of our knowledge, MAVEN Wang et al. (2020) is the only publicly available large-scale event dataset. However, the dataset only focuses on event detection without considering event arguments.

### Degree in Pipeline Framework

Degree is flexible and can be easily modified to Degree(Pipe), which first focuses event detection (ED) and then solves event argument extraction (EAE). Degree(Pipe) consists of two models: (1) Degree(ED), which aims to exact event triggers for the given event type, and (2) Degree(EAE), which identifies argument roles for the given event type and the corresponding trigger. Degree(ED) and Degree(EAE) are similar to Degree but with different prompts and output formats. We describe the difference as follows.

Degree(Ed).The prompt of Degree(ED) contains the following components:

* **Event type definition** is the same as the ones for Degree.
* **Event keywords** is the same as the one for Degree.
* **ED template** is designed as _"Event trigger is <Trigger>"_, which is actually the first part of the E2E template. Similar to Degree, the objective of Degree(ED) is to generate an output that replaces _"<Trigger>"_ in the ED template with event triggers.

Degree(EAE).The prompt of Degree(EAE) contains the following components:

* **Event type definition** is the same as the one for Degree.
* **Query trigger** is a string that indicates the trigger word for the given event type. For example, _"The event trigger word is detonated"_ points out that _"detonated"_ is the given trigger.

\begin{table}
\begin{tabular}{|l|p{398.7pt}|} \hline
**Event Type** & **EAE Template** \\ \hline Life:Divorce & somebody divorced in somewhere. \\ \hline Transaction:Transfer-Ownership & someone got something from some seller in somewhere. \\ Justice:Sue & somebody was used by some other in somewhere. The adjudication was judged by some adjudicator. \\ \hline \end{tabular}
\end{table}
Table 1: Three examples of EAE templates for the ACE 2005 corpus.

 

* **EAE template** is an event-type-specific template mentioned previously. It is actually the second part of E2E template.

Similar to Degree, the goal for Degree(EAE) is to generate an outputs that replace the placeholders in EAE template with event arguments.

In Section 3, we will compare Degree with Degree(Pipe) to study the benefit of dealing with event extraction in an end-to-end manner under the low-resource setting.

## 3 Experiments

We conduct experiments for _low-resource_ event extraction to study how Degree performs.

### Experimental Settings

Datasets.We consider ACE 2005 Doddington et al. (2004) and follow the pre-processing in Wadden et al. (2019) and Lin et al. (2020), resulting in two variants: **ACE05-E** and **ACE05-E\({}^{+}\)**. Both contain 33 event types and 22 argument roles. In addition, we consider **EEE-EN**Song et al. (2015) and adopt the pre-processing in Lin et al. (2020), which keeps 38 event types and 21 argument roles.

Data split for low-resource setting.We generate different proportions (1%, 2%, 3%, 5%, 10%, 20%, 30%, and 50%) of training data to study the influence of the size of the training set and use the original development set and test set for evaluation. Appendix C lists more details about the split generation process and the data statistics.

Evaluation metrics.We consider the same criteria in prior works Wadden et al. (2019); Lin et al. (2020). (1) **Trigger F1-score**: an trigger is correctly identified (Tri-I) if its offset matches the gold one; it is correctly classified (Tri-C) if its event type also matches the gold one. (2) **Argument F1-score**: an argument is correctly identified (Arg-I) if its offset and event type match the gold ones; it is correctly classified (Arg-C) if its role matches as well.

Compared baselines.We consider the following classification-based models: (1) **OneIE**Lin et al. (2020), the current state-of-the-art (SOTA) EE model trained with designed global features. (2) **BERT_QA**Du and Cardie (2020), which views EE tasks as a sequence of extractive question answering problems. Since it learns a classifier to indicate the position of the predicted span, we view it as a classification model. We also consider the following generation-based models: (3) **TANL**Paolini et al. (2021), which treats EE tasks as translation tasks between augmented natural languages. (4) **Text2Event**Lu et al. (2021), a sequence-to-structure model that converts the input passage to a tree-like event structure. Note that the outputs of both generation-based baselines are not natural sentences. Therefore, it is more difficult for them to utilize the label semantics. All the implementation details can be found in Appendix D. It is worth noting that we train OneIE with named entity annotations, as the original papers suggest, while the other models are trained without entity annotations.

### Main Results

Table 2 shows the trigger classification F1-scores and the argument classification F1-scores in three data sets with different proportions of training data. The results are visualized in Figure 3. Since our task is _end-to-end_ event extraction, the argument classification F1-score is the more important metric that we considered when comparing models.

From the figure and the table, we can observe that both Degree and Degree(Pipe) outperform all other baselines when using less than 10% of the training data. The performance gap becomes much more significant under the extremely low data situation. For example, when only 1% of the training data is available, Degree and Degree(Pipe) achieve more than 15 points of improvement in trigger classification F1 scores and more than 5 points in argument classification F1 scores. This demonstrates the effectiveness of our design. The generation-based model with carefully designed prompts is able to utilize the label semantics and the additional weakly supervised signals, thus helping learning under the low-resource regime.

Another interesting finding is that Degree and Degree(Pipe) seem to be more beneficial for predicting arguments than for predicting triggers. For example, OneIE, the strongest baseline, requires 20% of training data to achieve competitive performance on trigger prediction to Degree and Degree(Pipe); however, it requires about 50% of training data to achieve competitive performance in predicting arguments. The reason is that the ability to capture dependencies becomes more important for argument prediction than trigger prediction since arguments are usually strongly dependent on each other compared to triggers. Therefore, the improvements of our models for argument prediction 

are more significant.

Furthermore, we observe that Degree is slightly better than Degree(Pipe) under the low-resource setting. This provides empirical evidence on the benefit of jointly predicting triggers and arguments in a low-resource setting.

Finally, we perform additional experiments on few-shot and zero-shot experiments. The results can be found in Appendix E.

### High-Resource Event Extraction

Although we focus on data-efficient learning for low-resource event extraction, to better understand the advantages and disadvantages of our model, we additionally study Degree in the high-resource setting for controlled comparisons.

Compared baselines.In addition to the EE models mentioned above: **OneIE**Lin et al. (2020), **BERT_QA**Du and Cardie (2020), **TANL**Paolini et al. (2021), and **Text2Event**Lu et al. (2021), we



also consider the following baselines focusing on the high-resource setting. **dbRNN**[21] is classification-based model that adds dependency bridges for event extraction. **DyGIE++**[30] is a classification-based model with span graph propagation technique. **Joint3EE**[47] is a classification-based model jointly trained with annotations of entity, trigger, and argument. **MQAEE**[9] converts EE to a series of question answering problems for argument extraction . **BART-Gen**[9] is a generation-based model focusing on _only_ event argument extraction.3 Appendix D shows the implementation details for the baselines.

Footnote 3: We follow the original paper and use TapKey as their event detection model.

Results for event extraction.Table 3 shows the results of high-resource event extraction. In terms of trigger predictions (Tri-C), Degree and Degree(Pipe) outperforms all the baselines except for OneIE, the current state-of-the-art model. For argument predictions (Arg-C), our models have slightly better performance than OneIE in two out of the three datasets. When enough training examples are available, models can learn more sophisticated features from data, which do not necessarily follow the learned dependencies. Therefore, the advantage of Degree over Degree(Pipe) becomes less obvious. This result justifies our hypothesis that Degree has better performance for the _low-resource setting_ because of its ability to better capture dependencies.

Results for event argument extraction.In Table 4, we additionally study the performance for event argument extraction task, where the model makes argument predictions _with the gold trigger provided_. Interestingly, Degree(EAE) achieves pretty strong performance and outperforms other baselines with a large margin. Combining the results in Table 3, we hypothesize that event argument extraction is a more challenging task than event trigger detection and it requires more training examples to learn well. Hence, our proposed model, which takes the advantage of using label semantics to better capture dependencies, achieves a new state-of-the-art for event argument extraction.

## 4 Ablation Studies

In this section, we present comprehensive ablation studies to justify our design. To better understand the contribution of each component in the designed prompt and their effects on the different tasks, we ablate Degree(EAE) and Degree(ED) for both low-resource and high-resource situations.

Impacts of components in prompts.Table 5 lists the performance changes when removing the components in the prompts for event detection on ACE05-E. The performance decreases whenever removing any one of event type definition, event keywords, and ED template. The results suggest that three components are all necessary.

Table 6 demonstrates how different components in prompts affect the performance of event argument extr

\begin{table}
\begin{tabular}{|l|c|c c|c c|c c|} \hline Model & Type & \multicolumn{2}{c|}{ACE05-E} & \multicolumn{2}{c|}{ACE05-E\({}^{+}\)} & \multicolumn{2}{c|}{EEE-EN} \\ \cline{3-8}  & Arg- & Arg- & Arg- & Arg- & Arg- & Arg- & Arg- & Arg- \\ \hline DyGIE++ & Cls & 66.2 & 60.7 & - & - & - & - \\ BERT\_QA* & Cls & 68.2 & 65.4 & - & - & - & - \\ OneIE & Cls & 73.2 & 69.3 & 73.3 & 70.6 & 75.3 & 70.0 \\ TAIL & Gen & 65.9 & 61.0 & 66.3 & 62.3 & 75.6 & 69.0 \\ BART-Gen* & Gen & 69.9 & 66.7 & - & - & - & - \\ \hline Degree(EAE) & Gen & **76.0** & **73.5** & **75.2** & **73.0** & **80.2** & **76.3** \\ \hline \end{tabular}
\end{table}
Table 4: Results for high-resource event argument extraction. Models predict arguments based on the given gold triggers. Best scores are in bold. *We report the numbers from the original paper. Degree(EAE) achieves a new state-of-the-art performance on event argument extraction.

\begin{table}
\begin{tabular}{|l|l|c c|c c|c c|} \hline Model & Type & \multicolumn{2}{c|}{ACE05-E} & \multicolumn{2}{c|}{ACE05-E\({}^{+}\)} & \multicolumn{2}{c|}{ERE-EN} \\ \cline{3-8}  & Tri-C & Arg- & Tr-C & Arg-C & Tr-C & Arg-C \\ \hline dbRNN* & Cls & 69.6 & 50.1 & - & - & - & - \\ DyGLE++ & Cls & 70.0 & 50.0 & - & - & - & - \\ Joint3EE* & Cls & 69.8 & 52.1 & - & - & - & - \\ BERT\_QA* & Cls & 72.4 & 53.3 & - & - & - & - \\ MQAAee* & Cls & 71.7 & 53.4 & - & - & - & - \\ Onelle* & Cls & **74.7** & **56.8** & **72.8** & 54.8 & 57.0 & 46.5 \\ TANL & Gen & 68.4 & 47.6 & 68.6 & 46.0 & 54.7 & 43.2 \\ Text2Event* & Gen & 71.9 & 53.8 & 71.8 & 54.4 & **59.4** & 48.3 \\ BART-Gen & Gen & 71.1 & 53.7 & - & - & - & - \\ Defense(Pipe) & Gen & 72.2 & 55.8 & 71.7 & **56.8** & 57.8 & **59.4** \\ Degree & Gen & 73.3 & 55.8 & 70.9 & 56.3 & 57.1 & 49.6 \\ \hline \end{tabular}
\end{table}
Table 3: Results for high-resource event extraction. Highest scores are in bold and the second best scores are underlined. *We report the numbers from the original paper. Degree has a competitive performance to the SOTA model (OneIE) and outperform other baselines.

 

any one of event type definition, query trigger, and EAE template leads to performance drops, which validates their necessity. We observe that query trigger plays the most important role among the three and when less training data is given, the superiority of leveraging any of these weakly-supervised signal becomes more obvious.

Effects of different template designs.To verify the importance of using natural sentences as outputs, we study three variants of EAE templates:

* **Natural sentence.** Our proposed templates described in Section 2, e.g., "_somebody was born in somewhere._", where "_somebody"_ and "_somewhere"_ are placeholders that can be replaced by the corresponding arguments.
* **Natural sentence with special tokens.** It is similar to the natural sentence one except for using role-specific special tokens instead of "some-" words. For example, "\(<\)_Person\(>\) was born in \(<\)Place\(>\)_." We consider this to study the _label semantics_ of roles.
* **HTML-like sentence with special tokens.** To study the importance of using natural sentence, we also consider HTML-like sentence, e.g., "\(<\)_Person\(>\)\(<\)/Person\(>\)\(<\)Place\(>\)_ +/Place\(>\)_. The model aims to put argument predictions between the corresponding HTML tags.

The results of all variants of EAE templates on ACE05-E are shown in Table 7. We notice that writing templates in a natural language style get better performance, especially when only a few data is available (10% of data). This shows our design's capability to leverage pre-trained knowledge in the generation process. Additionally, there are over 1 F1 score performance drops when replacing natural language placeholders with special tokens. This confirms that leveraging label semantics for different roles is beneficial.

Sensitivity to template design.Finally, we study how sensitive our model is to the template. In addition to the original design of templates for event argument extraction, we compose other two sets of templates with different constructing rules (e.g., different word choices and different orders of roles). Table 8 shows the results of using different sets of templates. We observe a performance fluctuation when using different templates, which indicates that the quality of templates does affect the performance to a certain degree. Therefore, we need to be cautious when designing templates. However, even though our model could be sensitive to the template design, it still outperforms OneIE and BART-Gen, which are the best classification-based model and the best generation-based baseline, respectively.

## 5 Related Work

Fully supervised event extraction.Event extraction has been studied for over a decade Ahn et al. (2006); Ji and Grishman (2008) and most traditional event extraction works follow the fully supervised setting Nguyen et al. (2016); Sha et al. (2018); Nguyen and Nguyen (2019); Yang et al. (2019); Lin et al. (2020); Liu et al. (2020); Li et al. (2020). Many of them use classification-based models and use pipeline-style frameworks to extract events Nguyen et al. (2016); Yang et al. (2019); Wadden et al. (2019). To better leverage shared knowledge in event triggers and arguments, some works propose incorporating global features to jointly decide triggers and arguments Lin et al. (2020); Li et al. (2013); Yang and Mitchell (2016).

Recently, few generation-based event extraction models have been proposed Paolini et al. (2021);

\begin{table}
\begin{tabular}{|l|c c|c c|} \hline Model & \multicolumn{4}{c|}{10\% Data} & \multicolumn{2}{c|}{100\% Data} \\ \cline{2-5}  & Arg-I & Arg-C & Arg-H & Arg-C \\ \hline Full Degree(EAE) & **63.3** & **57.3** & **76.0** & **73.5** \\ - w/o Event type definition & 60.3 & 54.4 & 74.5 & 71.1 \\ - w/o EAE template & 57.0 & 51.9 & 73.8 & 70.4 \\ - w/o Query trigger & 55.2 & 49.9 & 71.4 & 69.0 \\ - only Query trigger & 51.9 & 48.1 & 71.2 & 69.4 \\ - only EAE template & 51.2 & 46.9 & 71.4 & 68.6 \\ - only Event type definition & 46.7 & 42.3 & 71.4 & 68.2 \\ \hline \end{tabular}
\end{table}
Table 6: Ablation study for the components in the prompt on event argument extraction with ACE05-E.

\begin{table}
\begin{tabular}{|l|c c|c c|} \hline Model & \multicolumn{2}{c|}{10\% Data} & \multicolumn{2}{c|}{100\% Data} \\ \cline{2-5}  & Arg-I & Arg-C & Arg-I & Arg-C \\ \hline OneIE & 48.3 & 45.4 & 73.2 & 69.3 \\ BART-Gen & - & - & 69.9 & 66.7 \\ \hline Degree(EAE) & 63.3 & **57.3** & **76.0** & **73.5** \\ Degree(EAE) + variant template 1 & 61.6 & 55.5 & 73.4 & 70.4 \\ Degree(EAE) + variant template 2 & **63.9** & 56.9 & 75.5 & 72.5 \\ \hline \end{tabular}
\end{table}
Table 7: Performances of Degree(EAE) on ACE05-E with different types of templates.

\begin{table}
\begin{tabular}{|l|c c|c c|} \hline  & \multicolumn{2}{c|}{10\% Data} & \multicolumn{2}{c|}{100\% Data} \\ \cline{2-5}  & Arg-I & Arg-C & Arg-I & Arg-C \\ \hline OneIE & 48.3 & 45.4 & 73.2 & 69.3 \\ BART-Gen & - & - & 69.9 & 66.7 \\ \hline Degree(EAE) & 63.3 & **57.3** & **76.0** & **73.5** \\ Degree(EAE) + variant template 1 & 61.6 & 55.5 & 73.4 & 70.4 \\ Degree(EAE) + variant template 2 & **63.9** & 56.9 & 75.5 & 72.5 \\ \hline \end{tabular}
\end{table}
Table 8: Study on the effect of different template constructing rules. Experiments is conducted on ACE05-E.

 

Huang et al. (2021, 2022); Li et al. (2021). TANL (Paolini et al., 2021) treats event extraction as translation tasks between augmented natural languages. Their predicted target--augmented language embed labels into the input passage via using brackets and vertical bar symbols. TempGen Huang et al. (2021) is a template-based role-filler entity extraction model, which generate outputs that fill role entities into non-natural templated sequences. The output sequence designs of TANL and Temporal hinder the models from fully leveraging label semantics, unlike Degree that generates natural sentences. BART-Gen Li et al. (2021) is also a generation-based model focusing on document-level event argument extraction. They solve event extraction with a pipeline, which prevents knowledge sharing across subtasks. All these fully supervised methods can achieve substantial performance with a large amount of annotated data. However, their designs are not specific for low-resource scenarios, hence, these models can not enjoy all the benefits that Degree obtains for low-resource event extraction at the same time, as we mentioned in Section 1.

Low-resource event extraction.It has been a growing interest in event extraction in a scenario with less data. Liu et al. (2020) uses a machine reading comprehension formulation to conduct event extraction in a low-resource regime. Text2Event Lu et al. (2021), a sequence-to-structure generation paradigm, first presents events in a linearized format, and then trains a generative model to generate the linearized event sequence. Text2Event's unnatural output format hinders the model from fully leveraging pre-trained knowledge. Hence, their model falls short on the cases with only extremely low data being available (as shown in Section 3).

Another thread of works are using meta-learning to deal with the less label challenge Deng et al. (2020); Shen et al. (2021); Cong et al. (2021). However, their methods can only be applied to event detection, which differs from our main focus on studying end-to-end event extraction.

## 6 Conclusion & Future Work

In this paper, we present Degree, a data-efficient generation-based event extraction model. Degree requires less training data because it better utilizes label semantics as well as weakly-supervised information, and captures better dependencies by jointly predicting triggers and arguments. Our experimental results and ablation studies show the superiority of Degree for low-resource event extraction.

Degree assumes that some weakly-supervised information (the description of events, similar keywords, and human-written templates) is accessible or not expensive for the users to craft. This assumption may holds for most situations. We leave the automation of template construction for future work, which can further ease the needed efforts when deploying Degree in a large-scale corpus.

## Acknowledgments

We thank anonymous reviewers for their helpful feedback. We thank the UCLA PLUSLab and UCLA-NLP group members for their initial review and feedback for an earlier version of the paper. This work is supported in part by the Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, and research awards sponsored by CISCO and Google.

## Ethics Considerations

Degree fine-tunes the pre-trained generative language model Lewis et al. (2020). Therefore, the generated output is potentially affected by the corpus for pre-training. Although with a low possibility, it is possible for our model to accidentally generate some malicious, counterfactual, and biased sentences, which may cause ethics concerns. We suggest carefully examining those potential issues before deploying the model in any real-world applications.

## References

* Ahn (2006) David Ahn. 2006. The stages of event extraction. In _Proceedings of the Workshop on Annotating and Reasoning about Time and Events_.
* Berant et al. (2014) Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Cong et al. (2021) Xin Cong, Shiyao Cui, Bowen Yu, Tingwen Liu, Yubin Wang, and Bin Wang. 2021. Few-shot event detection with prototypical amortized conditional random field. In _Findings of the Association for Computational Linguistics: ACL/IJCNLP_ 

Shumin Deng, Ningyu Zhang, Jiaojian Kang, Yichi Zhang, Wei Zhang, and Huajun Chen. 2020. Meta-learning with dynamic-memory-based prototypical network for few-shot event detection. In The Thirteenth ACM International Conference on Web Search and Data Mining (WSDM).
* tasks, data, and evaluation. In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC)_.
* Du and Cardie (2020) Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Fincke et al. (2021) Steven Fincke, Shantanu Agarwal, Scott Miller, and Elizabeth Boschee. 2021. Language model priming for cross-lingual event extraction. _arXiv preprint arXiv:2109.12383_.
* Han et al. (2021) Rujun Han, I-Hung Hsu, Jiao Sun, Julia Baylon, Qiang Ning, Dan Roth, and Nanyun Peng. 2021. ESTER: A machine reading comprehension dataset for reasoning about event semantic relations. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_.
* Han et al. (2019a) Rujun Han, I-Hung Hsu, Mu Yang, Aram Galstyan, Ralph M. Weischedel, and Nanyun Peng. 2019a. Deep structured neural network for event temporal relation extraction. In _Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019_.
* Han et al. (2019b) Rujun Han, Qiang Ning, and Nanyun Peng. 2019b. Joint event and temporal relation extraction with shared representations and structured prediction. In _2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Hogenboom et al. (2016) Frederik Hogenboom, Flavius Frasincar, Uzay Kaymak, Franciska de Jong, and Emiel Caron. 2016. A survey of event extraction methods from text for decision support systems. _Decis. Support Syst._, 85:12-22.
* Huang et al. (2022) Kuan-Hao Huang, I-Hung Hsu, Premkumar Natarajan, Kai-Wei Chang, and Nanyun Peng. 2022. Multilingual generative language models for zero-shot cross-lingual event argument extraction. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Huang and Peng (2021) Kung-Hsiang Huang and Nanyun Peng. 2021. Document-level event extraction with efficient end-to-end learning of cross-event dependencies. In _The 3rd Workshop on Narrative Understanding (NAACL 2021)_.
* Huang et al. (2021)Kung-Hsiang Huang, Sam Tang, and Nanyun Peng. 2021. Document-level entity-based extraction as template generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_.
* Huang et al. (2020) Kung-Hsiang Huang, Mu Yang, and Nanyun Peng. 2020. Biomedical event extraction with hierarchical knowledge graphs. In _the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)-Findings, short_.
* Ji and Grishman (2008) Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In _Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Li et al. (2020) Fayuan Li, Weiluha Peng, Yuguang Chen, Quan Wang, Lu Pan, Yaijuan Lyu, and Yong Zhu. 2020. Event extraction as multi-turn question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings (EMNLP)_.
* Li et al. (2013) Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Li et al. (2021) Sha Li, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-MLT)_.
* Lin et al. (2020) Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Liu et al. (2020) Jian Liu, Yuko Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020. Event extraction as machine reading comprehension. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Lohchilov and Hutter (2019) Ilya Lohchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_.
* Lu et al. (2019) Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi 

Chen. 2021. Text2event: Controllable sequence-to-structure generation for end-to-end event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP), Cited by: SS4.
* T. H. Nguyen and R. Grishman (2015)Event detection and domain adaptation with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: SS4.
* T. M. Nguyen and T. H. Nguyen (2019)One for all: Neural joint modeling of entities and events. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference (AAAI), Cited by: SS4.
* G. Paolini, B. Athiwaratkun, J. Krone, J. Ma, A. Achille, R. Anubhai, C. Nogueira dos Santos, B. Xiang, and S. Soatto (2021)Structured prediction as translation between augmented natural languages. In 9th International Conference on Learning Representations (ICLR), Cited by: SS4.
* E. Sha, F. Qian, B. Chang, and Z. Sui (2018)Jointly extracting event triggers and arguments by dependency-bridge RNN and tensor-based argument interaction. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), Cited by: SS4.
* S. Shen, T. Wu, G. Qi, Y. Li, G. Haffari, and S. Bi (2021)Adaptive knowledge-enhanced bayesian meta-learning for few-shot event detection. In Findings of the Association for Computational Linguistics: ACL/IJCNLP, Cited by: SS4.
* Z. Song, A. Bies, S. M. Strassel, T. Riese, J. Mott, J. Ellis, J. Wright, S. Kulick, Neville Ryant, and X. Ma (2015)From light to rich eke: annotation of entities, relations, and events. In Proceedings of the The 37th Workshop on EVENTS: Definition, Detection, Coreference, and Representation, (EVENTS@HLP-NAACL), Cited by: SS4.
* J. Sun and N. Peng (2021)Men are elected, women are married: Events gender bias on wikipedia. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, Virtual Event, August 1-6, 2021, Cited by: SS4.
* D. Wadden, U. Wennberg, Y. Luan, and H. Hajishirzi (2019)Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Cited by: SS4.
* X. Wang, Z. Wang, X. Han, W. Jiang, R. Han, Z. Liu, J. Li, P. Li, Y. Lin, and J. Zhou (2020)MAVEN: a massive general domain event detection dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, Cited by: SS4.
* X. Wang, Z. Wang, X. Han, Z. Liu, J. Li, P. Li, M. Sun, J. Zhou, and X. Ren (2019)HMEAE: hierarchical modular event argument extraction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Cited by: SS4.
* T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shelfer, P. von Platen, C. Ma, Y. Jernite, J. Phu, C. Xu, T. Le Scao, S. Guugger, M. Drame, Q. Lhoest, and A. Rush (2020)Huggingface's transformers: state-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Cited by: SS4.
* B. Yang and T. M. Mitchell (2016)Joint extraction of events and entities within a document context. In The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Cited by: SS4.
* S. Yang, D. Feng, L. Qiao, Zhigang Kan, and Dongsheng Li (2019)Exploring pre-trained language models for event extraction and generation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, Cited by: SS4.
* S. Yang, D. Feng, L. Qiao, Z. Kan, and D. Li (2019)Exploring pre-trained language models for event extraction and generation. In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), Cited by: SS4.
* H. Zhang, X. Liu, H. Pan, Y. Song, and C. W. Leung (2020)ASER: a large-scale eventuality knowledge graph. In The Web Conference 2020 (WWW), Cited by: SS4.

 

EAE Template Constructing

Our strategy to create an EAE template is first identifying all valid argument roles for the event type,4 such as _Attacker_, _Target_, _Instrument_, and _Place_ roles. Then, for each argument role, according to the semantics of the role type, we select natural and fluent words to form its placeholder (e.g., _some way_ for _Instrument_). This design aims to provide a simple way to help the model learn both the roles' label semantics and the _event structure_. Finally, we create a natural language sentence that connects all these placeholders. Notice that we try to keep the template as simple and short as possible. Table 9 lists all designed EAE templates for ACE05-E and ACE05-E\({}^{+}\). The EAE templates of ERE-EN can be found in Table 10.

Footnote 4: The valid roles for each event type are predefined in the event ontology for each dataset, or can be decided by the user of interest.

## Appendix B Training Details of Proposed Model

Given a passage, its annotated event types are consider as positive event types. During training, we additionally sample \(m\) event types that are not related to the passage as the negative examples, where \(m\) is a hyper-parameter. In our experiments, \(m\) is usually set to 13 or 15.

For all of Degree, Degree(ED), and Degree(EAE), we fine-tune the pre-trained BARTIarge Lewis et al. (2020) with Huggingface package Wolf et al. (2020). The number of parameters is around 406 millions. We train Degree with our machine that equips 128 AMD EPYC 7452 32-Core Processor, 4 NVIDIA A100 GPUs, and 792G RAM. We consider AdamW optimizer Loshchilov and Hutter (2019) with learning rate set to \(10^{-5}\) and the weight decay set to \(10^{-5}\). We set the batch size to 6 for Degree(EAE) and 32 for Degree(ED) and Degree. The number of training epochs is 45. It takes around 2 hours, 18 hours, 22 hours to train Degree(EAE), Degree(ED), and Degree, respectively.

We do hyper-parameter search on \(m\), the number of negative examples, from \(\{3,5,7,10,13,15,18,21\}\), and our preliminary trials shows that \(m\) less than 10 are usually less useful. For the learning rate and the weight decay, we tune it based on our preliminary experiment for event argument extraction from \(\{10^{-5},10^{-4}\}\), while they are both fixed to \(10^{-5}\) for all the experiments.

## Appendix C Datasets

We consider ACE 20055 Doddington et al. (2004) and EREG Song et al. (2015). Both consider _LDC User Agreement for Non-Members7_ as the licenses. Both datasets are created for entity, relation, and event extraction while our focus is only event extraction in this paper. In the original ACE 2005 dataset, it contains data for English, Chinese, and Arabic and we only take the English data for our experiment. In the original ERE dataset, it contains data for English, and Chinese and we only take the English data for our experiment as well.

Footnote 5: [https://catalog.ldc.upenn.edu/](https://catalog.ldc.upenn.edu/) LDC2006T06[https://catalog.ldc.upenn.edu/](https://catalog.ldc.upenn.edu/) LDC2020T19[https://catalog.ldc.upenn.edu/](https://catalog.ldc.upenn.edu/) license/ldc-non-members-agreement.pdf

Because both datasets contain event like _Justice:Execute_ and _Life:Die_, it is possible that some offensive words (e.g., killed) would appear in the passage. Also, some real names may appear in the passage as well (e.g., Palestinian president, Mahmoud Abbas). How to accurately identify these kinds of information is part of the goal of the task. Therefore, we do not take any changes on the datasets for protecting or anonymizing.

We split the training data based on documents, which is a more realistic setup compared to splitting data by instance. Table 11 lists the statistics of ACE05-E, ACE05-E\({}^{+}\), and ERE-EN. Specifically, we try to make each proportion of data contain as many event types as possible.

## Appendix D Implementation Details

This section describes the implementation details for all baselines we use. We run the experiments with three different random seeds and report the best value.

* **DyGIE++**: we use their released pre-trained model8 for evaluation. Footnote 8: [https://github.com/dwadden/dygiepp](https://github.com/dwadden/dygiepp)
* **OneIE**: we use their provided code9 to train the model with default parameters. Footnote 9: [http://lender.cs.illinois.edu/software/onei/](http://lender.cs.illinois.edu/software/onei/)
* **BERT_QA**: we use their provided code10 to train the model with default parameters. Footnote 10: [https://github.com/xinyadu/eeqa](https://github.com/xinyadu/eeqa)
* **TANL**: we use their provided code11 to train the 

model. We conduct the experiments with two variations: (1) using their default parameters, and (2) using their default parameters but with more training epochs. We observe that the second variant works better. As a result, we report the number obtained from the second setting.
* **Text2Event**: we use their official code12 to train the model with the provided parameter setting. Footnote 12: [https://github.com/luyaojie/Text2Event](https://github.com/luyaojie/Text2Event)
* **dbRNN:** we directly report the experimental results from their paper.
* **Joint3EE:** we directly report the experimental results from their paper.
* **MQAEE:** we directly report the experimental results from their paper.
* **BART-Gen**: we report the experimental results from their released appendix.13 



\begin{table}
\begin{tabular}{|l|p{142.3pt}|} \hline
**Event Type** & **EAE Template** \\ \hline Life:Be-Born & somebody was born in somewhere. \\ \hline Life:Maryr & somebody got married in somewhere. \\ \hline Life:Divorce & somebody divorced in somewhere. \\ \hline Life:Injure & somebody or some organization led to some victim injured by some way in somewhere. \\ \hline Life:Die & somebody or some organization led to some victim died by some way in somewhere. \\ Movement:Transport & something was sent to somewhere from some place by some vehicle. somebody or some organization was responsible for the transport. \\ \hline Transaction:Transfer-Ownership & someone got something from some seller in somewhere. \\ \hline Transaction:Transfer-Money & someone paid some other in somewhere. \\ \hline Business:Start-Org & somebody or some organization launched some organzation in somewhere. \\ \hline Business:Merge-Org & some organization was merged. \\ \hline Business:Declare-Bankruptcy & some organization declared bankruptcy. \\ \hline Business:End-Org & some organization dissolved. \\ \hline Conflict:Attack & some attacker attacked some facility, someone, or some organization by some way in somewhere. \\ \hline Conflict:Demonstrate & some people or some organization protest at somewhere. \\ \hline Contact:Meet & some people or some organization met at somewhere. \\ \hline Contact:Phone-Write & some people or some organization called or texted messages at somewhere. \\ \hline Personnel:Start-Position & somebody got new job and was hired by some people or some organization in somewhere. \\ \hline Personnel:End-Position & somebody stopped working for some people or some organization at somewhere. \\ \hline Personnel:Nominate & somebody was nominated by somebody or some organization to do a job. \\ \hline Personnel:Elect & somebody was elected a position, and the election was voted by some people or some organization in somewhere. \\ \hline Justice:Arrest-Jail & somebody was sent to jailed or arrested by somebody or some organization in somewhere. \\ \hline Justice:Release-Parole & somebody was released by some people or some organization from somewhere. \\ \hline Justice:Trial-Hearing & somebody, prosecuted by some other, faced a trial in somewhere. The hearing was judged by some adjudicator. \\ \hline Justice:Charge-Indict & somebody was charged by some other in somewhere. The adjudication was judged by some adjudicator. \\ \hline Justice:Sue & somebody was sued by some other in somewhere. The adjudication was judged by some adjudicator. \\ \hline Justice:Convict & somebody was convicted of a crime in somewhere. The adjudication was judged by some adjudicator. \\ \hline Justice:Sentence & somebody was sentenced to punishment in somewhere. The adjudication was judged by some adjudicator. \\ \hline Justice:Fine & some people or some organization in somewhere was ordered by some adjudicator to pay a fine. \\ \hline Justice:Execute & somebody was executed by somebody or some organization at somewhere. \\ \hline Justice:Extradite & somebody was extradicted to somewhere from some place. somebody or some organization was responsible for the extradition. \\ \hline Justice:Acquit & somebody was acquitted of the charges by some adjudicator. \\ \hline Justice:Pardon & somebody received a pardon from some adjudicator. \\ \hline Justice:Appeal & some other in somewhere appealed the adjudication from some adjudicator. \\ \hline \end{tabular}
\end{table}
Table 9: All EAE templates for ACE05-E and ACE05-E\({}^{+}\).





\begin{table}
\begin{tabular}{|l|p{142.3pt}|} \hline
**Event Type** & **EAE Template** \\ \hline Life:Be-Born & somebody was born in somewhere. \\ \hline Life:Marray & somebody got married in somewhere. \\ \hline Life:Divorce & somebody divorced in somewhere. \\ \hline Life:Injure & somebody or some organization led to some victim injured by some way in somewhere. \\ \hline Life:Die & somebody or some organization led to some victim died by some way in somewhere. \\ \hline Movement:Transport-Person & somebody was moved to somewhere from some place by some way. somebody or some organization was responsible for the movement. \\ \hline Movement:Transport-Artifact & something was sent to somewhere from some place. somebody or some organization was responsible for the transport. \\ \hline Business:Start-Org & somebody or some organization launched some organzation in somewhere. \\ \hline Business:Merge-Org & some organization was merged. \\ \hline Business:Declare-Bankruptcy & some organzation declared bankruptcy. \\ \hline Business:End-Org & some organzation dissolved. \\ \hline Conflict:Attack & some attacker attacked some facility, someone, or some organization by some way in somewhere. \\ \hline Conflict:Demonstrate & some people or some organization protest at somewhere. \\ \hline Contact:Meet & some people or some organization met at somewhere. \\ \hline Contact:Correspondence & some people or some organization contacted each other at somewhere. \\ \hline Contact:Broadcast & some people or some organization made announcement to some publicity at somewhere. \\ \hline Contact:Contact & some people or some organization talked to each other at somewhere. \\ \hline Manufacture:Artifact & something was built by somebody or some organization in somewhere. \\ \hline Personnel:Start-Position & somebody got new job and was hired by some people or some organization in somewhere. \\ \hline Personnel:End-Position & somebody stopped working for some people or some organization at somewhere. \\ \hline Personnel:Nominate & somebody was nominated by somebody or some organization to do a job. \\ \hline Personnel:Elect & somebody was elected a position, and the election was voted by somebody or some organization in somewhere. \\ \hline Transaction:Transfer-Ownership & The ownership of something from someone was transferred to some other at somewhere. \\ \hline Transaction:Transfer-Money & someone paid some other in somewhere. \\ \hline Transaction:Transaction & someone give some things to some other in somewhere. \\ \hline Justice:Armest-Jail & somebody was sent to jailed or arrested by somebody or some organization in somewhere. \\ \hline Justice:Release-Parole & somebody was released by somebody or some organization from somewhere. \\ \hline Justice:Trial-Hearing & somebody, prosecuted by some other, faced a trial in somewhere. The hearing was judged by some adjudicator. \\ \hline Justice:Charge-Indict & somebody was charged by some other in somewhere. The adjudication was judged by some adjudicator. \\ \hline Justice:Sue & somebody was sued by some other in somewhere. The adjudication was judged by some adjudicator. \\ \hline Justice:Cenvict & somebody was convicted of a crime in somewhere. The adjudication was judged by some adjudicator. \\ \hline Justice:Sentence & somebody was sentenced to punishment in somewhere. The adjudication was judged by some adjudicator. \\ \hline Justice:Fine & some people or some organization in somewhere was ordered by some adjudicator to pay a fine. \\ \hline Justice:Execute & somebody was executed by somebody or some organization at somewhere. \\ \hline Justice:Extradite & somebody was extradicted to somewhere from some place. somebody or some organization was responsible for the extradition. \\ \hline Justice:Acquit & somebody was acquitted of the charges by some adjudicator. \\ \hline Justice:Pardon & somebody received a pardon from some adjudicator. \\ \hline Justice:Appeal & somebody in somewhere appealed the adjudication from some adjudicator. \\ \hline \end{tabular}
\end{table}
Table 10: All EAE templates for ERE-EN.





\begin{table}
\begin{tabular}{|l|c|r r r r r r|} \hline Dataset & Split & \#Docs & \#Sents & \#Events & \#Event Types & \#Args & \#Arg Types \\ \hline  & Train (full) & 529 & 17172 & 4202 & 33 & 4859 & 22 \\  & Train (1%) & 5 & 103 & 47 & 14 & 65 & 16 \\  & Train (2\%) & 10 & 250 & 77 & 17 & 104 & 16 \\  & Train (3\%) & 15 & 451 & 119 & 23 & 153 & 17 \\  & Train (5\%) & 25 & 649 & 212 & 27 & 228 & 21 \\ ACE05-E & Train (10\%) & 50 & 1688 & 412 & 28 & 461 & 21 \\  & Train (20\%) & 110 & 3467 & 823 & 33 & 936 & 22 \\  & Train (30\%) & 160 & 5429 & 1368 & 33 & 1621 & 22 \\  & Train (50\%) & 260 & 8985 & 2114 & 33 & 2426 & 22 \\ \cline{2-7}  & Dev & 28 & 923 & 450 & 21 & 605 & 22 \\  & Test & 40 & 832 & 403 & 31 & 576 & 20 \\ \hline \hline  & Train (full) & 529 & 19216 & 4419 & 33 & 6607 & 22 \\  & Train (1\%) & 5 & 92 & 49 & 15 & 75 & 16 \\  & Train (2\%) & 10 & 243 & 82 & 19 & 129 & 16 \\  & Train (3\%) & 15 & 434 & 124 & 24 & 203 & 19 \\  & Train (5\%) & 25 & 628 & 219 & 27 & 297 & 21 \\ ACE05-E\({}^{+}\) & Train (10\%) & 50 & 1915 & 428 & 29 & 629 & 21 \\  & Train (20\%) & 110 & 3834 & 878 & 33 & 1284 & 22 \\  & Train (30\%) & 160 & 6159 & 1445 & 33 & 2212 & 22 \\  & Train (50\%) & 260 & 10104 & 2231 & 33 & 3293 & 22 \\ \cline{2-7}  & Dev & 28 & 901 & 468 & 22 & 759 & 22 \\  & Test & 40 & 676 & 424 & 31 & 689 & 21 \\ \hline \hline  & Train (full) & 396 & 14736 & 6208 & 38 & 8924 & 21 \\  & Train (1\%) & 4 & 109 & 61 & 14 & 78 & 16 \\  & Train (2\%) & 8 & 228 & 128 & 21 & 183 & 19 \\  & Train (3\%) & 12 & 419 & 179 & 26 & 272 & 19 \\  & Train (5\%) & 20 & 701 & 437 & 31 & 640 & 21 \\ ERE-EN & Train (10\%) & 40 & 1536 & 618 & 37 & 908 & 21 \\  & Train (20\%) & 80 & 2848 & 1231 & 38 & 1656 & 21 \\  & Train (30\%) & 120 & 4382 & 1843 & 38 & 2632 & 21 \\  & Train (50\%) & 200 & 7690 & 3138 & 38 & 4441 & 21 \\ \cline{2-7}  & Dev & 31 & 1209 & 525 & 34 & 730 & 21 \\  & Test & 31 & 1163 & 551 & 33 & 822 & 21 \\ \hline \end{tabular}
\end{table}
Table 11: Dataset statistics. Our experiments are conducted in sentences, which were split from documents. In the table, “#Docs” means the number of documents; “#Sents” means the number of sentences, “#Events” means the number of events; “#Event Types” means the number of event types in total; “#Args” means the number of argument in total; “#Arg Types” means the number of argument role types in total.





## Appendix E Few-Shot and Zero-Shot Event Extraction

In order to further test our models' generaliability, we additionally conduct zero-shot and few-shot experiments on the ACE05-E dataset with Degree(ED) and Degree(EAE).

Settings.We first select the top \(n\) common event types as "seen" types and use the rest as "unseen/rare" types, where the top common types are listed in Table 12. To simulate a zero-shot scenario, we remove all events with "unseen/rare" types from the training data. To simulate a few-shot scenario, we keep only \(k\) event examples for each "unseen/rare" type (denoted as \(k\)-shot). During the evaluation, we calculate micro F1-scores only for these "unseen/rare" types.

Compared baselines.We consider the following baselines: (1) **BERT_QA**Du and Cardie (2020) (2) **OneIE**Lin et al. (2020) (3) **Matching baseline**, a proposed baseline that makes trigger predictions by performing string matching between the input passage and the event keywords. (4) **Lemmatization baseline**, another proposed baseline that performs string matching on lemmarized input passage and the event keywords. (Note: (3) and (4) are baselines only for event detection tasks.)

Experimental results.Figure 4, Table 13, and Table 14 show the results of \(n=5\) and \(n=10\). From the two subfigures in the left column, we see that Degree(ED) achieves promising results in the zero-shot setting. In fact, it performs better than BERT_QA trained in the 10-shot setting and OneIE trained in the 5-shot setting. This demonstrates the great potential of Degree(ED) to discover new event types. Interestingly, we observe that our two proposed baselines perform surprisingly well, suggesting that the trigger annotations in ACE05-E are actually not diverse. Despite their impressive performance, Degree(ED) still outperforms the matching baseline by over 4.7% absolute trigger classification F1 in both \(n=5\) and \(n=10\) cases in zero-shot scenario. Additionally, with only one training instance for each unseen type, Degree(ED) can outperform both proposed baselines.

Next, we compare the results for the event argument extraction task. From the two middle subfigures, we observe that when given gold triggers, our model performs much better than all baselines with a large margin. Lastly, we train models for both trigger and argument extraction and report the final argument classification scores in the two right subfigures. We justify that our model has strong generalizability to unseen event types and it can outperform BERT_QA and OneIE even when they are both trained in 5-shot settings.




* [190] S. A. Krizhevsky, M. Ghahramani, and J. Malik. Learning from deep learning: a survey of machine learning models. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 257-268, 2014.
* [191] D. L. Kingma and P. Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1312.6140_ (2013).
* [192] R. C. LeCun, Y. Bengio, and T. Darrell. Generative adversarial nets. In _Advances in neural information processing systems_, volume 10. Springer, 2014.
* [193] H. Li, X. Zhang, Z. Liu, and W. Sun. Deep residual learning with generative adversarial networks. In _NIPS_, pages 1332-1339, 2014.
* [194] I. Goodfellow, I. Sutskever, and I. Vedaldi. Imagenet classification by jointly connected layers. In _NeurIPS_, pages 1332-1339, 2014.
* [195] E. Hinton, J. Friedman, and J. Friedman. Visualizing data using tiny images. _Journal of Machine Learning Research_**10** (2008), no. 1, 2008.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{|l l|c c|c c|c c|c c|c c|} \hline \multicolumn{10}{|c|}{Event Extraction} \\ \hline  & \multicolumn{6}{|c|}{Common 5} & \multicolumn{6}{|c|}{Common 10} \\ \cline{3-12} Trigger & Argument & Tri-I & Tri-C & Arg-I & Arg-C & Tri-I & Tri-C & Arg-I & Arg-C \\ \hline Matching Baseline & 42.7 & 42.1 & - & - & 46.3 & 46.3 & - & - & - \\ Lemmatization Baseline & 51.5 & 50.2 & - & - & 56.6 & 56.0 & - & - & - \\ \hline BERT\_QA 1-shot & 10.0 & 1.4 & 1.3 & 1.3 & 8.2 & 1.6 & 1.1 & 1.1 \\ BERT\_QA 5-shot & & 14.0 & 12.6 & 11.1 & 10.8 & 20.8 & 15.4 & 14.6 & 13.9 \\ BERT\_QA 10-shot & & 37.8 & 33.5 & 22.9 & 22.1 & 32.0 & 27.8 & 19.5 & 18.6 \\ OneIE 1-shot & & 4.2 & 4.2 & 1.5 & 1.5 & 4.1 & 2.7 & 2.0 & 2.0 \\ OneIE 5-shot & & 39.3 & 38.5 & 24.8 & 22.8 & 41.9 & 41.9 & 29.7 & 27.2 \\ OneIE 10-shot & & 54.8 & 53.3 & 36.0 & 34.9 & 61.5 & 57.8 & 41.4 & 39.2 \\ Degree(ED) 0-shot & Degree(EAE) 0-shot & 53.3 & 46.8 & 29.6 & 25.1 & 60.9 & 54.5 & 42.0 & 31.4 \\ Degree(ED) 1-shot & Degree(EAE) 1-shot & 60.1 & 53.3 & 38.8 & 31.6 & 61.2 & 60.9 & 41.1 & 34.7 \\ Degree(ED) 5-shot & Degree(EAE) 5-shot & 57.8 & 55.5 & 40.6 & 36.1 & 65.8 & 64.8 & 45.3 & 42.7 \\ Degree(ED) 10-shot & Degree(EAE) 10-shot & 63.8 & 61.2 & 46.0 & 42.0 & 72.1 & 68.8 & 52.5 & 48.4 \\ \hline OneIE (Full) & & 72.7 & 70.5 & 52.3 & 49.9 & 74.5 & 73.0 & 51.2 & 48.9 \\ Degree(ED) (Full) & Degree(EAE) (Full) & 68.4 & 66.0 & 51.9 & 48.7 & 72.0 & 69.8 & 52.5 & 49.2 \\ \hline \end{tabular}
\end{table}
Table 13: Full results of zero/few-shot event extraction on ACE05-E.

\begin{table}
\begin{tabular}{|l l|r r|r r r|r r r|} \hline \multicolumn{10}{|c|}{Event Argument Extraction} \\ \hline  & \multicolumn{6}{|c|}{Common 5} & \multicolumn{6}{|c|}{Common 10} \\ \cline{3-12} Trigger & Argument & Tri-I & Tri-C & Arg-I & Arg-C & Tri-I & Tri-C & Arg-I & Arg-C \\ \hline Gold Triggers & BERT\_QA 0-shot & 100.0 & 100.0 & 55.8 & 37.9 & 100.0 & 100.0 & 57.2 & 46.7 \\ Gold Triggers & BERT\_QA 1-shot & 100.0 & 100.0 & 55.8 & 44.3 & 100.0 & 100.0 & 57.8 & 47.2 \\ Gold Triggers & BERT\_QA 5-shot & 100.0 & 100.0 & 56.6 & 49.6 & 100.0 & 100.0 & 59.1 & 50.6 \\ Gold Triggers & BERT\_QA 10-shot & 100.0 & 100.0 & 58.8 & 52.9 & 100.0 & 100.0 & 60.5 & 52.8 \\ Gold Triggers & OneIE 1-shot & 100.0 & 100.0 & 40.9 & 36.5 & 100.0 & 100.0 & 48.3 & 44.2 \\ Gold Triggers & OneIE 5-shot & 100.0 & 100.0 & 55.6 & 51.4 & 100.0 & 100.0 & 58.6 & 55.0 \\ Gold Triggers & Deepic 10-shot & 100.0 & 100.0 & 59.4 & 56.7 & 100.0 & 100.0 & 62.0 & 59.5 \\ Gold Triggers & Degree(EAE) 0-shot & 100.0 & 100.0 & 56.1 & 48.0 & 100.0 & 100.0 & 66.5 & 53.3 \\ Gold Triggers & Degree(EAE) 1-shot & 100.0 & 100.0 & 65.2 & 55.2 & 100.0 & 100.0 & 65.4 & 54.7 \\ Gold Triggers & Degree(EAE) 5-shot & 100.0 & 100.0 & 70.9 & 62.2 & 100.0 & 100.0 & 68.0 & 61.7 \\ Gold Triggers & Degree(EAE) 10-shot & 100.0 & 100.0 & 71.1 & 64.2 & 100.0 & 100.0 & 71.6 & 64.3 \\ \hline Gold Triggers & BERT\_QA (Full) & 100.0 & 100.0 & 63.1 & 57.9 & 100.0 & 100.0 & 62.1 & 56.5 \\ Gold Triggers & OneIE (Full) & 100.0 & 100.0 & 70.8 & 66.4 & 100.0 & 100.0 & 67.9 & 64.1 \\ Gold Triggers & Degree(EAE) (Full) & 100.0 & 100.0 & 74.5 & 70.6 & 100.0 & 100.0 & 73.6 & 68.9 \\ \hline \end{tabular}
\end{table}
Table 14: Full results of zero/few-shot event argument extraction on ACE05-E.