SlowGuess/NAACL_2022_OCR huggingface dataset:
I-Hung Hsu*† Kuan-Hao Huang*‡ Elizabeth Boschee† Scott Miller† Premkumar Natarajan† Kai-Wei Chang‡ Nanyun Peng†‡
†Information Science Institute, University of Southern California
$^{\ddagger}$ Computer Science Department, University of California, Los Angeles
{ihunghsu, boschee, smiller, pnataraj}@isi.edu
{khhuang, kwchang, violetpeng}@cs.ucla.edu
Event extraction requires high-quality expert human annotations, which are usually expensive. Therefore, learning a data-efficient event extraction model that can be trained with only a few labeled examples has become a crucial challenge. In this paper, we focus on low-resource end-to-end event extraction and propose DEGREE, a data-efficient model that formulates event extraction as a conditional generation problem. Given a passage and a manually designed prompt, DEGREE learns to summarize the events mentioned in the passage into a natural sentence that follows a predefined pattern. The final event predictions are then extracted from the generated sentence with a deterministic algorithm. DEGREE has three advantages to learn well with less training data. First, our designed prompts provide semantic guidance for DEGREE to leverage label semantics and thus better capture the event arguments. Moreover, DEGREE is capable of using additional weakly-supervised information, such as the description of events encoded in the prompts. Finally, DEGREE learns triggers and arguments jointly in an end-to-end manner, which encourages the model to better utilize the shared knowledge and dependencies among them. Our experimental results demonstrate the strong performance of DEGREE for low-resource event extraction.
Event extraction (EE) aims to extract events, each of which consists of a trigger and several participants (arguments) with their specific roles, from a given passage. For example, in Figure 1, a Justice:Execute event is triggered by the word "execution" and this event contains three argument roles, including an Agent (Indonesia) who carries out the execution, a Person who is executed (convicts), and a Place where the event occurs (not mentioned in the passage). Previous work (Yang et al., 2019a;
Figure 1: Two examples of events (Justice:Execute and Justice:Appeal) extracted from the given passage.
Justice:Execute
Agent:	Indonesia
Person:	convicts
Place:	None
Justice:Appeal
Plaintiff :	five
Prosecutor :	None
Adjudicator :	Supreme Court
Place :	Indonesia
Fincke et al., 2021) usually divides EE into two subtasks: (1) event detection, which identifies event triggers and their types, and (2) event argument extraction, which extracts the arguments and their roles for given event triggers. EE has been shown to benefit a wide range of applications, e.g., building knowledge graphs (Zhang et al., 2020), question answering (Berant et al., 2014; Han et al., 2021), and other downstream studies (Han et al., 2019a; Hogenboom et al., 2016; Sun and Peng, 2021).
Most prior works on EE rely on a large amount of annotated data for training (Nguyen and Grishman, 2015; Nguyen et al., 2016; Han et al., 2019b; Du and Cardie, 2020; Huang et al., 2020; Huang and Peng, 2021; Paolini et al., 2021). However, high-quality event annotations are expensive to obtain. For example, the ACE 2005 corpus (Doddington et al., 2004), one of the most widely used EE datasets, requires two rounds of annotations by linguistics experts. The high annotation costs make these models hard to be extended to new domains and new event types. Therefore, how to learn a data-efficient EE model trained with only a few annotated examples is a crucial challenge.
In this paper, we focus on low-resource event extraction, where only a small amount of training examples are available for training. We propose DEGREE (Data-Efficient GeneRation-Based Event Extraction), a generation-based model that takes a passage and a manually designed prompt as the input, and learns to summarize the passage into a
* The authors contribute equally.
Human Language Technologies, pages 1890 - 1908
Passage: Earlier Monday, a 19-year-old Palestinian riding a bicycle detonated a 30-kilo (66-pound) bomb near a military jeep in the Gaza Strip, injuring three soldiers.
Figure 2: An illustration of DEGREE for predicting a Contact:Attack event. The input of DEGREE consists of the given passage and our design prompt that contains an event type description, event keywords, and a E2E template. DEGREE is trained to generate an output to fill in the placeholders (underlined words) in the E2E template with triggers and arguments. The final event prediction is then decoded from the generated output.
natural sentence following a predefined template, as illustrated in Figure 2. The event triggers and arguments can then be extracted from the generated sentence by using a deterministic algorithm.
DEGREE enjoys the following advantages to learn well with less training data. First, the framework provides label semantics via the designed template in the prompts. As the example in Figure 2 shows, the word "somewhere" in the prompt guides the model to predict words being similar to location for the role Place. In addition, the sentence structure of the template and the word "attacked" depict the semantic relation between the role Attacker and the role Target. With these kinds of guidance, DEGREE can make more accurate predictions with less training examples. Second, the prompts can incorporate additional weak-supervision signal about the task, such as the description of the event and similar keywords. These resources are usually readily available. For example, in our experiments, we take the information from the annotation guideline, which is provided along with the dataset. This information facilitates DEGREE to learn under a low-resource situation. Finally, DEGREE is designed for end-to-end event extraction and can solve event detection and event argument extraction at the same time. Leveraging the shared knowledge and dependencies between the two tasks makes our model more data-efficient.
Existing works on EE usually have only one or two of above-mentioned advantages. For exam
ple, previous classification-based models (Nguyen et al., 2016; Wang et al., 2019; Yang et al., 2019b; Wadden et al., 2019; Lin et al., 2020) can hardly encode label semantics and other weak supervision signals. Recently proposed generation-based models for event extraction solved the problem in a pipeline fashion; therefore, they cannot leverage shared knowledge between subtasks (Paolini et al., 2021; Li et al., 2021). Furthermore, their generated outputs are not natural sentences, which hinders the utilization of label semantics (Paolini et al., 2021; Lu et al., 2021). As a result, our model DEGREE can achieve significantly better performance than prior approaches on low-resource event extraction, as we will demonstrate in Section 3.
- We propose DEGREE, a generation-based event extraction model that learns well with less data by better incorporating label semantics and shared knowledge between subtasks (Section 2).
- Experiments on ACE 2005 (Doddington et al., 2004) and ERE-EN (Song et al., 2015) demonstrate the strong performance of DEGREE in the low-resource setting (Section 3).
- We present comprehensive ablation studies in both the low-resource and the high-resource setting to better understand the strengths and weaknesses of our model (Section 4).
Our code and models can be found at https://github.com/PlusLabNLP/DEGREE.
2 Data-Efficient Event Extraction
We introduce DEGREE, a generation-based model for low-resource event extraction. Unlike previous works (Yang et al., 2019a; Li et al., 2021), which separate event extraction into two pipelined tasks (event detection and event argument extraction), DEGREE is designed for the end-to-end event extraction and predict event triggers and arguments at the same time.
2.1 The DEGREE Model
We formulate event extraction as a conditional generation problem. As illustrated in Figure 2, given a passage and our designed prompt, DEGREE generates an output following a particular format. The final predictions of event triggers and argument roles can be then parsed from the generated output with a deterministic algorithm. Compared to previous classification-based models (Wang et al., 2019; Yang et al., 2019b; Wadden et al., 2019; Lin et al., 2020), the generation framework provides a flexible way to include additional information and guidance. By designing appropriate prompts, we encourage DEGREE to better capture the dependencies between entities and, therefore, to reduce the number of training examples needed.
The desired prompt not only provides information but also defines the output format. As shown in Figure 2, it contains the following components:
- Event type definition describes the definition for the given event type. For example, “The event is related to conflict and some violent physical act” describes a Conflict:Attack event.
- Event keywords presents some words that are semantically related to the given event type. For example, war, attack, and terrorism are three event keywords for the Conflict:Attack event. In practice, we collect three words that appear as the triggers in the example sentences from the annotation guidelines.
- E2E template defines the expected output format and can be separated into two parts. The first part is called ED template, which is designed as “Event trigger is <Trigger>”, where “<Trigger>” is a special token serving as a placeholder. The second part is the EAE template, which differs based on the given event type. For example, in Figure 2, the EAE template for a
Conflict:Attack event is "some attacker attacked some facility, someone, or some organization by some way in somewhere". Each underlined string starting with "some-" serves as a placeholder corresponding to an argument role for a Conflict:Attack event. For instance, "some way" corresponds to the role Instrument and "somewhere" corresponds to the role Place. Notice that every event type has its own EAE template. We list three EAE templates in Table 1. The full list of EAE templates and the construction details can be found in Appendix A.
2.2 Training
The training objective of DEGREE is to generate an output that replaces the placeholders in E2E template with the gold labels. Take Figure 2 as an example, DEGREE is expected to replace "<Trigger>" with the gold trigger (detonated), replace "some attacker" with the gold argument for role Attacker (Palestinian), and replace "some way" with the gold argument for role Instrument (bomb). If there are multiple arguments for the same role, they are concatenated with "and"; if there is no predicted argument for one role, the model should keep the corresponding placeholder (i.e., "some-" in the E2E template). For the case that there are multiple triggers for the given event type in the input passage, DEGREE is trained to generate the output text that contains multiple E2E template such that each E2E template corresponds to one trigger and its argument roles. The hyperparameter settings are detailed in Appendix B.
We enumerate all event types and generate an output for each event type. After we obtain the generated sentences, we compare the outputs with E2E template to determine the predicted triggers and arguments in string format. Finally, we apply string matching to convert the predicted string to span offsets in the passage. If the predicted string appears in the passage multiple times, we choose all span offsets that match for trigger predictions and choose the one closest to the given trigger span for argument predictions.
Notice that the E2E template plays an important role for DEGREE. First, it serves as the control signal and defines the expected output format. Second,
<sup>1</sup>The definition can be derived from the annotation guidelines, which are provided along with the datasets.
Event Type	EAE Template
Life:Divorce	somebody divorced in somewhere.
Transaction:Transfer-Ownership	someone got something from some seller in somewhere.
Justice:Sue	somebody was sued by some other in somewhere. The adjudication was judged by some adjudicator.
Table 1: Three examples of EAE templates for the ACE 2005 corpus.
it provides label semantics to help DEGREE make accurate predictions. Those placeholders (words starting with "some-") in the E2E template give DEGREE some hints about the entity types of arguments. For instance, when seeing "somewhere", DEGREE tends to generate a location rather than a person. In addition, the words other than "some- describe the relationships between roles. For example, DEGREE knows the relationship between the role Attacker and the role Target (who is attacking and who is attacked) due to E2E template. This guidance helps DEGREE learn the dependencies between entities.
Unlike previous generation-based approaches (Paolini et al., 2021; Li et al., 2020; Huang et al., 2021), we intentionally write E2E templates in natural sentences. This not only uses label semantics better but also makes the model easier to leverage the knowledge from the pre-trained decoder. In Section 4, we will provide experiments to demonstrate the advantage of using natural sentences.
Cost of template constructing. DEGREE does require human effort to design the templates; however, writing those templates is much easier and more effortless than collecting complicated event annotations. As shown in Table 1, we keep the EAE templates as simple and short as possible. Therefore, it takes only about one minute for people who are not linguistic experts to compose a template. In fact, several prior works (Liu et al., 2020; Du and Cardie, 2020; Li et al., 2020) also use constructed templates as weakly-supervised signals to improve models. In Section 4, we will study how different templates affect the performance.
Efficiency Considerations. DEGREE requires to enumerate all event types during inference, which could cause efficiency considerations when extending to applications that contain many event types. This issue is minor for our experiments on the two datasets (ACE 2005 and ERE-EN), which are relatively small scales in terms of the number of event types. Due to the high cost of annotations, there is hardly any public datasets for end-to-end event ex
traction on a large scale, $^{2}$  and we cannot provide a more thorough studies when the experiments scale up. We leave the work on benchmarking and improving the efficiency of DEGREE in the scenario considering more diverse and comprehensive types of events as future work.
2.5 DEGREE in Pipeline Framework
DEGREE is flexible and can be easily modified to DEGREE(PIPE), which first focuses event detection (ED) and then solves event argument extraction (EAE). DEGREE(PIPE) consists of two models: (1) DEGREE(ED), which aims to exact event triggers for the given event type, and (2) DEGREE(EAE), which identifies argument roles for the given event type and the corresponding trigger. DEGREE(ED) and DEGREE(EAE) are similar to DEGREE but with different prompts and output formats. We describe the difference as follows.
DEGREE(ED). The prompt of DEGREE(ED) contains the following components:
Event type definition is the same as the ones for DEGREE.
Event keywords is the same as the one for DEGREE.
- ED template is designed as “Event trigger is <Trigger>”, which is actually the first part of the E2E template.
Similar to DEGREE, the objective of DEGREE(ED) is to generate an output that replaces "<Trigger>" in the ED template with event triggers.
DEGREE(EAE). The prompt of DEGREE(EAE) contains the following components:
Event type definition is the same as the one for DEGREE.
- Query trigger is a string that indicates the trigger word for the given event type. For example, "The event trigger word is detonated" points out that "detonated" is the given trigger.
To the best of our knowledge, MAVEN (Wang et al., 2020) is the only publicly available large-scale event dataset. However, the dataset only focuses on event detection without considering event arguments.
- EAE template is an event-type-specific template mentioned previously. It is actually the second part of E2E template.
Similar to DEGREE, the goal for DEGREE(EAE) is to generate an outputs that replace the placeholders in EAE template with event arguments.
In Section 3, we will compare DEGREE with DEGREE(PIPE) to study the benefit of dealing with event extraction in an end-to-end manner under the low-resource setting.
We conduct experiments for low-resource event extraction to study how DEGREE performs.
Datasets. We consider ACE 2005 (Doddington et al., 2004) and follow the pre-processing in Wadden et al. (2019) and Lin et al. (2020), resulting in two variants: ACE05-E and ACE05-E+. Both contain 33 event types and 22 argument roles. In addition, we consider ERE-EN (Song et al., 2015) and adopt the pre-processing in Lin et al. (2020), which keeps 38 event types and 21 argument roles.
Data split for low-resource setting. We generate different proportions (1%, 2%, 3%, 5%, 10%, 20%, 30%, and 50%) of training data to study the influence of the size of the training set and use the original development set and test set for evaluation. Appendix C lists more details about the split generation process and the data statistics.
Evaluation metrics. We consider the same criteria in prior works (Wadden et al., 2019; Lin et al., 2020). (1) Trigger F1-score: an trigger is correctly identified (Tri-I) if its offset matches the gold one; it is correctly classified (Tri-C) if its event type also matches the gold one. (2) Argument F1-score: an argument is correctly identified (Arg-I) if its offset and event type match the gold ones; it is correctly classified (Arg-C) if its role matches as well.
Compared baselines. We consider the following classification-based models: (1) OneIE (Lin et al., 2020), the current state-of-the-art (SOTA) EE model trained with designed global features. (2) BERT_QA (Du and Cardie, 2020), which views EE tasks as a sequence of extractive question answering problems. Since it learns a classifier to indicate the position of the predicted span, we view it as a classification model. We also consider
the following generation-based models: (3) TANL (Paolini et al., 2021), which treats EE tasks as translation tasks between augmented natural languages. (4) Text2Event (Lu et al., 2021), a sequence-to-structure model that converts the input passage to a tree-like event structure. Note that the outputs of both generation-based baselines are not natural sentences. Therefore, it is more difficult for them to utilize the label semantics. All the implementation details can be found in Appendix D. It is worth noting that we train OneIE with named entity annotations, as the original papers suggest, while the other models are trained without entity annotations.
Table 2 shows the trigger classification F1-scores and the argument classification F1-scores in three data sets with different proportions of training data. The results are visualized in Figure 3. Since our task is end-to-end event extraction, the argument classification F1-score is the more important metric that we considered when comparing models.
From the figure and the table, we can observe that both DEGREE and DEGREE(PIPE) outperform all other baselines when using less than  $10\%$  of the training data. The performance gap becomes much more significant under the extremely low data situation. For example, when only  $1\%$  of the training data is available, DEGREE and DEGREE(PIPE) achieve more than 15 points of improvement in trigger classification F1 scores and more than 5 points in argument classification F1 scores. This demonstrates the effectiveness of our design. The generation-based model with carefully designed prompts is able to utilize the label semantics and the additional weakly supervised signals, thus helping learning under the low-resource regime.
Another interesting finding is that DEGREE and DEGREE(PIPE) seem to be more beneficial for predicting arguments than for predicting triggers. For example, OneIE, the strongest baseline, requires  $20\%$  of training data to achieve competitive performance on trigger prediction to DEGREE and DEGREE(PIPE); however, it requires about  $50\%$  of training data to achieve competitive performance in predicting arguments. The reason is that the ability to capture dependencies becomes more important for argument prediction than trigger prediction since arguments are usually strongly dependent on each other compared to triggers. Therefore, the improvements of our models for argument prediction
Trigger Classification F1-Score (%)
Model	Type	ACE05-E	ACE05-E+	ERE-EN
1%	3%	5%	10%	20%	30%	1%	3%	5%	10%	20%	30%	1%	3%	5%	10%	20%	30%
BERT_QA	Cls	20.5	40.2	42.5	50.1	61.5	61.3	-	-	-	-	-	-	-	-	-	-	-	-
OneIE	Cls	38.5	52.4	59.3	61.5	67.6	67.4	39.0	52.5	60.6	58.1	66.5	66.4	11.0	36.9	46.7	48.8	51.8	53.5
Text2Event	Gen	14.2	35.2	46.4	47.0	55.6	60.7	15.7	38.4	43.9	46.3	56.5	62.0	6.3	25.6	33.5	42.4	46.7	50.1
TANL	Gen	34.1	48.1	53.4	54.8	61.8	61.6	30.3	50.9	53.1	55.7	60.8	61.7	5.7	30.8	43.4	45.9	49.0	49.3
DEGREE(Pipe)	Gen	55.1	62.8	63.8	66.1	64.4	64.4	56.4	62.5	61.1	62.3	62.5	67.1	32.7	44.5	41.6	50.6	51.1	53.5
DEGREE	Gen	55.4	62.1	65.8	65.8	68.3	68.2	49.5	63.5	62.3	68.5	67.6	66.9	27.9	45.5	47.0	53.0	51.7	53.5
Argument Classification F1-Score (%)
Model	Type	ACE05-E	ACE05-E+	ERE-EN
1%	3%	5%	10%	20%	30%	1%	3%	5%	10%	20%	30%	1%	3%	5%	10%	20%	30%
BERT_QA	Cls	4.7	14.5	26.9	27.6	36.7	38.8	-	-	-	-	-	-	-	-	-	-	-	-
OneIE	Cls	9.4	22.0	26.8	26.8	42.7	47.8	10.4	20.6	29.7	35.5	46.7	48.0	2.6	20.3	29.7	35.1	40.7	43.0
Text2Event	Gen	3.9	12.2	19.1	24.9	32.3	39.2	5.7	16.5	21.3	26.4	35.2	42.1	2.3	15.2	23.6	28.7	35.7	38.7
TANL	Gen	8.5	17.2	24.7	29.0	34.0	39.2	8.6	22.3	30.4	29.2	34.6	39.0	1.4	20.2	29.5	30.1	35.6	36.9
DEGREE(Pipe)	Gen	13.1	26.1	27.6	42.1	40.7	44.0	16.0	26.4	29.9	39.5	41.3	48.5	12.2	29.7	31.4	39.4	41.9	42.2
DEGREE	Gen	21.7	30.1	35.5	41.6	46.2	48.7	18.7	34.0	35.7	43.6	48.9	51.2	14.5	28.9	33.4	41.7	42.9	45.5
Table 2: Trigger classification F1-scores and argument classification F1-scores for low-resource event extraction. Highest scores are in bold and the second best scores are underlined. "Cls" and "Gen" represent classification-based models and generation-based models, respectively. If the model is a pipelined model, then its argument predictions are based on its predicted triggers. DEGREE achieves a much better performance than other baselines. The performance gap becomes more significant for the extremely low-resource situation.
Figure 3: Trigger classification F1-scores and argument classification F1-scores for low-resource event extraction. DEGREE achieves a much better performance than other baselines. The performance gap becomes more significant for the extremely low-resource situation.
are more significant.
Furthermore, we observe that DEGREE is slightly better than DEGREE(PIPE) under the low-resource setting. This provides empirical evidence on the benefit of jointly predicting triggers and arguments in a low-resource setting.
Finally, we perform additional experiments on few-shot and zero-shot experiments. The results can be found in Appendix E.
3.3 High-Resource Event Extraction
Although we focus on data-efficient learning for low-resource event extraction, to better understand the advantages and disadvantages of our model, we additionally study DEGREE in the high-resource setting for controlled comparisons.
Compared baselines. In addition to the EE models mentioned above: OneIE (Lin et al., 2020), BERT_QA (Du and Cardie, 2020), TANL (Paolini et al., 2021), and Text2Event (Lu et al., 2021), we
Model	Type	ACE05-E	ACE05-E+	ERE-EN
Tri-C	Arg-C	Tri-C	Arg-C	Tri-C	Arg-C
dbRNN*	Cls	69.6	50.1	-	-	-	-
DyGIE++	Cls	70.0	50.0	-	-	-	-
Joint3EE*	Cls	69.8	52.1	-	-	-	-
BERT_QA*	Cls	72.4	53.3	-	-	-	-
MQAEE*	Cls	71.7	53.4	-	-	-	-
OneIE*	Cls	74.7	56.8	72.8	54.8	57.0	46.5
TANL	Gen	68.4	47.6	68.6	46.0	54.7	43.2
Text2Event*	Gen	71.9	53.8	71.8	54.4	59.4	48.3
BART-Gen*	Gen	71.1	53.7	-	-	-	-
DEGREE(PIPE)	Gen	72.2	55.8	71.7	56.8	57.8	50.4
DEGREE	Gen	73.3	55.8	70.9	56.3	57.1	49.6